JP6702755B2 - Image processing device, imaging device, image processing method, and program - Google Patents

Description

The present invention relates to image generation processing at an arbitrary viewpoint position by a three-dimensional reconstruction technique.

Conventionally, as a three-dimensional reconstruction technique, there is a technique of reconstructing three-dimensional data from images at a plurality of viewpoints at the time of photographing and then generating an image as if the photograph was taken from another viewpoint based on the three-dimensional data. is there. The viewpoint (hereinafter referred to as an arbitrary viewpoint) set after the image capturing can be arbitrarily selected by the user, for example. In the technique disclosed in Patent Document 1, when an image from an arbitrary viewpoint is created from an image from a plurality of viewpoints, the feature amount between the target image and the images close in time series is calculated. When the reliability of the feature amount is low, it is possible to create accurate three-dimensional data by reconstructing the three-dimensional data based on the images before and after the time series.

JP, 2009-212728, A JP, 2013-239119, A JP, 2012-221128, A

In the image generation processing at an arbitrary viewpoint position, an area in which images from multiple viewpoints cannot be correlated may be an area that is hidden by changing viewpoints (occlusion area by subject), that is, an occlusion area. .. Therefore, when reconstructing three-dimensional data from image information before and after time series, if the amount of change in the occlusion area is small between images, there is a problem in improving the mapping accuracy. Further, it is premised that all the images are once subjected to simultaneous processing for mapping to three-dimensional data, and then the images are generated from an arbitrary viewpoint. In this case, there is a problem that the amount of data to be processed at once increases.
An object of the present invention is to provide an image processing device, an imaging device, an image processing method, and a program that enable sequential processing for generating an image at an arbitrary viewpoint position from a plurality of image data.

An apparatus according to the present invention includes image acquisition means for acquiring a plurality of image data, distance information acquisition means for acquiring distance information in the depth direction of image data acquired from the image acquisition means, and acquisition from the image acquisition means. Image transformation means for transforming the image data by coordinate transformation, and distance information acquired by the distance information obtaining means is transformed by coordinate conversion corresponding to the coordinate transformation performed on the image data. Distance information transformation means, distance information synthesis means for synthesizing a plurality of distance information transformed by the distance information transformation means, distance information recording means for recording the distance information output by the distance information synthesis means, and the distance Synthesis information generating means for generating information for synthesis based on the distance information transformed by the information transforming means and the distance information recorded in the distance information recording means, and the image transforming means based on the information for synthesis. Image combining means for generating an arbitrary viewpoint image by combining a plurality of images that have been subjected to the deformation processing by.

According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, and a program that enable sequential processing of generating an image at an arbitrary viewpoint position from a plurality of image data.

It is a block diagram which shows the structural example of the imaging device which concerns on embodiment of this invention. FIG. 3 is a block diagram showing a configuration example of an image processing unit 104 according to the first embodiment. 3 is a block diagram showing a configuration example in an αMAP generation unit 208 in FIG. 2. FIG. 6 is a flowchart showing a flow of image processing in the first embodiment. 3 is a flowchart showing a processing example by an αMAP generation unit 208 in FIG. 2. It is a block diagram which shows the structural example of the image processing part 604 which concerns on 2nd Embodiment. 7 is a block diagram showing a configuration example of an αMAP generation unit 710 in FIG. 6. It is a flow chart which shows a flow of image processing in a 2nd embodiment. 7 is a flowchart showing a processing example by an αMAP generation unit 710 in FIG. 6. It is a figure explaining the relationship between a picked-up image and an arbitrary viewpoint image. It is a figure explaining the picked-up image and distance MAP in each shooting position. It is a figure which illustrates the picked-up image and arbitrary viewpoint image in each shooting position. It is a figure which illustrates the distance MAP and each arbitrary viewpoint distance MAP in each photography position. It is a figure explaining the synthetic|combination process of the photography 1 and the photography 2. FIG. 8 is a diagram illustrating a combination process of a combination result of shooting 1 and shooting 2 and shooting 3; It is a schematic diagram explaining the relationship between the two-dimensional coordinates in an image and the three-dimensional coordinates in the real space. It is explanatory drawing regarding the reliability MAP which concerns on 2nd Embodiment.

Each embodiment of the present invention will be described below in detail with reference to the accompanying drawings. In each embodiment, a process of generating an image at a position of an arbitrary viewpoint (hereinafter referred to as an arbitrary viewpoint position) from an image captured by changing the viewpoint position is performed. The above processing is executed every time the photographed images are sequentially input, and the image data at the arbitrary viewpoint position is updated. In the following description, an image at an arbitrary viewpoint position will be referred to as an “arbitrary viewpoint image”. Regarding the subject, the positional relationship will be described by defining the side closer to the imaging device as the near side.

[First Embodiment]
The image processing apparatus according to the first embodiment of the present invention will be described below. First, the outline of the image processing according to the present embodiment will be described. In order to generate an arbitrary viewpoint image, the image transformation processing for changing the viewpoint is performed on the image shot with the viewpoint position changed, and the distance is changed. The transformation process for the MAP is executed. The distance MAP is shooting information of a subject at each pixel position, that is, distance information indicating a distance distribution. Here, as the distance MAP, it suffices to know the relative relationship of the distances between the subjects in the image, which correspond to the distance distribution. For example, the distribution information of the image shift amount between the parallax images obtained from the parallax images forming a pair, the distribution information of the defocus amount obtained by converting the image shift amount into the defocus amount, and the like may be used.
The present embodiment is characterized in that the distance MAP is also deformed for changing the viewpoint. After the transformation process, a synthesis map (αMAP described later) is generated based on the distance MAP in which the viewpoint is changed. A process of updating the data of the arbitrary viewpoint image is sequentially executed by combining the plurality of images after the viewpoint change using the combining map. Hereinafter, the present embodiment will be described step by step.

FIG. 1 is a block diagram showing a configuration example when the image processing apparatus according to the present embodiment is applied to an image pickup apparatus.
The optical system 101 includes a lens group including a zoom lens and a focus lens, an aperture adjustment device, and a shutter device. The optical system 101 adjusts the magnification, the focus position, or the amount of light of the subject image formed on the light receiving surface of the image sensor 102. The image sensor 102 photoelectrically converts the light flux from the subject that has passed through the optical system 101 into an electrical signal. The image pickup device 102 is a photoelectric conversion device such as an image sensor using a CCD (charge coupled device) or a CMOS (complementary metal oxide film semiconductor). In the present embodiment, each pixel of the image sensor 102 is configured in a Bayer array having RGB color filters, and each pixel is a pupil division type sensor in which at least two photoelectric conversion elements correspond to one microlens. Has become. However, the form of the image pickup device is not limited to this. The A (Analog)/D (Digital) converter 103 acquires the output signal of the image sensor 102 and converts the video signal into a digital image signal.

The image processing unit 104 performs known signal processing on the output from the A/D conversion unit 103, and also performs processing of generating an arbitrary viewpoint image from a plurality of input images. The processing performed by the image processing unit 104 will be described in detail later. The image processing unit 104 performs image processing not only on the image data output from the A/D conversion unit 103 but also on the image data read from the recording unit 109. The drive control unit 105 controls the drive of the optical system 101 and the image sensor 102 in order to adjust the aperture value, sensitivity, focal length, and focus position, and to perform camera shake correction (image blur correction).

The system control unit 106 is a control central unit that includes a CPU (central processing unit) and controls the overall operation of the image pickup apparatus. The system control unit 106 outputs a drive control signal to the drive control unit 105 based on a brightness value obtained from the image processed by the image processing unit 104 or an instruction signal transmitted from the operation unit 107, and the optical system 101 or The image sensor 102 is controlled.

The display unit 108 includes a liquid crystal display, an organic EL (Electro Luminescence) display, and the like. The display unit 108 displays an image according to the image data acquired by the image sensor 102 and the image data read from the recording unit 109. The recording unit 109 having a function of recording image data includes an information recording medium. For example, an information recording medium using a memory card having a semiconductor memory mounted therein, a package containing a rotating recording medium such as a magneto-optical disk, or the like is used, and the information recording medium is removable from the image pickup apparatus.
The bus 110 connects the image processing unit 104, the drive control unit 105, the system control unit 106, the display unit 108, and the recording unit 109, and is used for transmitting and receiving image data, signals, and the like between the units.

Next, with reference to FIG. 2, a configuration related to the present embodiment in the image processing unit 104 will be described in detail. FIG. 2 is a block diagram showing a part of the image processing unit 104.
The image acquisition unit 201 sequentially acquires the image data output from the A/D conversion unit 103, and outputs the image data to the image development unit 202 and the distance MAP generation unit 203, respectively. The image developing unit 202 outputs the image data subjected to the developing process to the image transforming unit 204. The image recording unit 206 records the data of the arbitrary viewpoint image, the image combining unit 209 acquires the image data from the image transforming unit 204 and the image recording unit 206, and outputs the image data after the image combination.

The distance MAP generation unit 203 generates data of a distance map (also referred to as distance MAP) as distance information in the depth direction regarding the image data, and outputs the data to the distance MAP transformation unit 205. In the process of generating the distance MAP, the distance information is generated by a known method using a correlation calculation with respect to image data forming a pair corresponding to different pupil regions of the image pickup optical system acquired by the image pickup device 102. However, the present invention is not limited to this, and the distance information is obtained by a DFD method using a method of obtaining distance information from a pair of image data having parallax obtained from a plurality of image pickup means or a correlation calculation such as SAD from image data having different focus positions. It may be generated (see Patent Document 2). SAD is an abbreviation for "Sum of Absolute Difference", and DFD is an abbreviation for "Depth from Defocus". The distance MAP deforming unit 205 generates information on the distance MAP at the arbitrary viewpoint position and outputs it to the distance MAP synthesizing unit 210. The distance MAP recording unit 207 records the information subjected to the deformation processing by the distance MAP deforming unit 205. The information read from the distance MAP recording unit 207 is output to the αMAP generating unit 208 and the distance MAP combining unit 210, respectively. The αMAP generation unit 208 generates α map information for synthesis (also referred to as αMAP) as described below, and outputs it to the image synthesis unit 209 and the distance MAP synthesis unit 210.

The operation of the image processing unit 104 shown in FIG. 2 will be described with reference to the flowchart of FIG. The following processing is executed according to a control command from the system control unit 106.
When the shooting operation starts, the image processing unit 104 initializes the value of a variable (denoted as k) used in the sequential processing to 1 in step S401. In step S402, the image acquisition unit 201 performs data input processing for the kth captured image. In S403, the distance MAP generation unit 203 generates the kth distance MAP corresponding to the kth captured image input in S402. In the present embodiment, as described above, the distance MAP is generated using the parallax image data that forms a pair and corresponds to the k-th captured image obtained from the image sensor 102. At this time, the distance MAP is generated using the parallax image data in the state of the RGB color signal or the luminance signal that has been subjected to the correction processing for correcting the noise and the signal attenuation caused by the image sensor 102 and the subsequent processing circuit. It In step S404, the image developing unit 202 develops the k-th image data input in step S402, and converts the k-th image data into image data with good visibility according to the image output device. The development processing is processing including at least a part of shading correction processing, image pickup signal correction processing, white balance correction processing, noise reduction processing, edge enhancement processing, color matrix processing, gamma correction processing, and the like. Note that the k-th image handled in the processing of S405 and thereafter refers to the image subjected to the development processing in S404.

In step S405, the image transformation unit 204 uses the information of the kth distance MAP generated in step S403 to perform the transformation process on the kth image developed in step S404. Data of the k-th arbitrary viewpoint image is generated by the image transformation process. Details of the arbitrary viewpoint image generation processing will be described later. In S406, the distance MAP deforming unit 205 receives the information of the kth distance MAP generated in S403 and generates the information of the distance MAP at the kth arbitrary viewpoint position. The distance MAP at the arbitrary viewpoint position will be referred to as “arbitrary viewpoint distance MAP” below. The process of generating the arbitrary viewpoint distance MAP will be described in detail later.

In S407, the αMAP generation unit 208 reads the information of the arbitrary viewpoint distance MAP, which is processed from the first to “k−1”th and recorded in the distance MAP recording unit 207. In step S<b>408, the αMAP generation unit 208 inputs the information on the k-th arbitrary viewpoint distance MAP and the information on the arbitrary viewpoint distance MAP recorded in the distance MAP recording unit 207, and generates the αMAP information for composition. The αMAP information generated in S408 is used in S410 and S411 described below. The process of generating the αMAP for synthesis will be described in detail later.

In step S409, the image combining unit 209 reads the data of the arbitrary viewpoint image, which has been processed from the first to “k−1”th and recorded in the image recording unit 206. In step S410, the image composition unit 209 acquires the data of the k-th arbitrary viewpoint image from the image transformation unit 204, and acquires the data of the arbitrary viewpoint image recorded in the image recording unit 206. The image combining unit 209 combines the acquired image data based on the information of the α map generated in S408. Here, in the two-dimensional coordinate system set on the image, the α map value at the position coordinate (x, y) of a certain point is referred to as α(x, y). The pixel value of the k-th arbitrary viewpoint image is described as pix_k(x, y), and the pixel value of the arbitrary viewpoint image recorded in the image recording unit 206 is described as pix_m(x, y). When the pixel value of the combined arbitrary viewpoint image is described as pix(x, y), this is calculated as in (Equation 1).
In the present embodiment, the value of α(x, y) is binary (0 or 1), but multi-valued logic that takes a value of 0 or more and 1 or less may be adopted as necessary. This also applies to the following calculation of the distance MAP combination.

In step S<b>411, the distance MAP synthesizing unit 210, which is the distance information synthesizing unit, acquires the information of the k-th arbitrary viewpoint distance MAP from the distance MAP deforming unit 205, and calculates the arbitrary viewpoint distance MAP recorded in the distance MAP recording unit 207. Get information. For these arbitrary viewpoint distances MAP, the distance MAP synthesizing unit 210 performs distance information synthesizing processing based on the information of the α map generated in S408. In the two-dimensional coordinate system set for the image, the value of the k-th arbitrary viewpoint distance MAP at a certain coordinate (x, y) is described as Z_k(x, y) and is recorded in the distance MAP recording unit 207. The value of the arbitrary viewpoint distance MAP is described as Z_m (x, y). When the value of the distance MAP after combination is denoted as Z (x, y), Z (x, y) is calculated as in (Equation 2) using α (x, y) that is an α map value.

In step S412, the image combining unit 209 outputs the arbitrary viewpoint image data combined in step S410 to the image recording unit 206, and updates the arbitrary viewpoint image data. In step S413, the distance MAP combining unit 210 outputs the arbitrary viewpoint distance MAP combined in step S411 to the distance MAP recording unit 207, and updates the recorded arbitrary viewpoint distance MAP. S414 is a process of determining whether or not the image capturing by the image capturing device has been completed. That is, the image processing unit 104 determines whether or not the next “k+1”th image is input. When the imaging by the imaging device is finished and the “k+1”th image data is not input, the process proceeds to S415. If the "k+1"th image data is input without the end of the image capturing by the image capturing apparatus, the process proceeds to S416.

In step S415, the image processing unit 104 outputs the data of the arbitrary viewpoint image created from the data of the k images processed up to the present time as output image data, and ends the series of processes. The arbitrary viewpoint image data output from the image processing unit 104 is encoded and recorded in the same image file including the corresponding distance MAP. Alternatively, the output arbitrary viewpoint image data is recorded in a separate file in association with the corresponding distance MAP. On the other hand, in S416, the "k+1"th image data is input and the variable k value increment (k++) process is executed. That is, after the variable k 1 is updated as k+1, the process returns to S402.
In the present embodiment, an example has been described in which the k-th image data is processed and the data of the arbitrary viewpoint image created until the end of shooting is not output. Not limited to such an example, data of arbitrary viewpoint images that are sequentially created may be output as output image data of the image processing unit 104. This also applies to the embodiments described later.

Next, the generation processing of the arbitrary viewpoint image described in S405 of FIG. 4 will be described in detail. FIG. 10 illustrates the relationship between the captured image and the arbitrary viewpoint image. As shown in FIG. 10(A), regarding the shooting positions 1101 to 1103, it is assumed that the subject is a person, trees A to C in the back, and a building in the back. An arbitrary viewpoint position is shown at a position 1104. FIG. 10B shows a photographed image 1111 photographed at the photographing position 1101 with buildings, trees, and people as subjects. FIG. 10C shows an arbitrary viewpoint image 1114 at the arbitrary viewpoint position 1104.

When the shooting position corresponding to the shot image 1111 shown in FIG. 10B is represented in the real space, the shooting position 1101 in FIG. 10A is obtained. It is assumed that an image as if taken at the arbitrary viewpoint position 1104 is created as the arbitrary viewpoint image 1114 based on the image taken at the shooting position 1101. In this case, the subject information necessary for generating the arbitrary viewpoint image is insufficient only with the information of the image captured at the capturing position 1101. That is, it is necessary to create an arbitrary viewpoint image by inputting information of images captured from a plurality of positions including the shooting position 1102 and the shooting position 1103 as well as the shooting position 1101. In the following, shooting at the shooting position 1101 is called “shooting 1”, shooting at the shooting position 1102 is called “shooting 2”, and shooting at the shooting position 1103 is called “shooting 3”. In the present embodiment, a processing example of generating an arbitrary viewpoint image based on three images will be described.

Processing for creating an arbitrary viewpoint image from a captured image will be described with reference to FIG. 11. FIG. 11A is a diagram showing photographed images obtained at photographing positions of photographing 1 to photographing 3. FIG. 11B is a diagram showing an arbitrary viewpoint image. It is necessary to perform image transformation from the images photographed at the photographing positions of photographing 1, photographing 2, and photographing 3 so as to correspond to the position of the arbitrary viewpoint image. In order to transform the photographed image into an image at an arbitrary viewpoint position, each distance MAP in photographing 1 to photographing 3 shown in FIG. 11C is required. That is, the distance MAP representing the shooting distance of the subject formed on each pixel of the shot image is required in each of Shooting 1 to Shooting 3.

In the process of transforming an image at an arbitrary viewpoint position based on a plurality of captured images, first, a process of converting coordinates (x, y) of the captured image into real space coordinates (X, Y, Z) is performed. Coordinates (x, y) represent two-dimensional Cartesian coordinates, and real space coordinates (X, Y, Z) represent three-dimensional Cartesian coordinates. Letting the position coordinates of the captured image be (x, y) and the distance value of the distance MAP corresponding to the position coordinates (x, y) be Z, the real space coordinates (X, Y, Z) are ). "Pp" represents the pixel pitch of the captured image, and "f" represents the focal length at the time of capturing. The position coordinates (x, y) are coordinates with the center of the image as the origin (0, 0).

The calculation processing of (Equation 3) will be described with reference to FIG. FIG. 16A is a diagram showing a correspondence relationship between coordinates (x, y) on the captured image and coordinates (X, Y, Z) on the real space. A projection plane 1501 which is a captured image is shown by a line segment. (Equation 3) is calculated based on the similarity relationship of the triangles shown in FIG. Using the same variables as in (Equation 3), the similarity relationship of triangles can be expressed as in (Equation 4) below.
θ represents the apex angle between similar triangles having the same vertex. In FIG. 16A, the projection surface 1501 that is a captured image is shown on the camera side, but in FIG. 16B, the projection surface 1502 that is a captured image is on the real space side (front side of the camera). In this case, the idea is the same as above. Hereinafter, the coordinate conversion process according to the above (formula 3) is referred to as “back projection conversion process”. However, the back projection conversion process applicable to this embodiment is not limited to the above.

Next, a process of converting the real space coordinates (X, Y, Z) into the real space coordinates (X ^* , Y ^* , Z ^* ) from the arbitrary viewpoint position is executed. Specifically, coordinate conversion is performed by taking into account the rotation component from the shooting position to the arbitrary viewpoint position around the x, y, and z axes and the shift component in the x, y, and z directions. Matrix elements based on rotation components around the x, y, and z axes are denoted as r ₁₁ to r ₃₃ . It noted before rotation x, y, and z direction shift component and _{t 3} from _{t 1,} referred x after the rotation, y, and z direction shift component and ^t _{* 3} from ^t _{* 1.} In the process of converting the real space coordinates (X, Y, Z) into the real space coordinates (X ^* , Y ^* , Z ^* ) from the arbitrary viewpoint position, the calculation shown in (Formula 5) below is performed.

Hereinafter, the coordinate conversion process according to (Equation 5) is referred to as “real space movement conversion process”. However, the real space movement conversion processing applicable to this embodiment is not limited to this. The calculation processing of the rotation component and the shift component from the captured image to the image at the arbitrary viewpoint position is performed by using a known technique such as posture estimation at the time of capturing using the corresponding point search between images in the present embodiment. Be seen. By this processing, the shooting position is calculated, and the amount of movement from the shooting position to the arbitrary viewpoint position is calculated (see Patent Document 3).

Finally, the process of converting the real space coordinates (X ^* , Y ^* , Z ^* ) into the coordinates (x ^* , y ^* ) on the arbitrary viewpoint image is executed. This conversion is performed using (Equation 6) below. Here, the coordinates (x ^* , y ^* ) are two-dimensional coordinates with the center of the image as the origin position (0, 0). "Pp ^* " represents the pixel pitch of the arbitrary viewpoint image, and "f ^* " represents the focal length of the arbitrary viewpoint image.
Hereinafter, the coordinate conversion process according to (Equation 6) is referred to as “projection conversion process”. However, the projection conversion process applicable to this embodiment is not limited to this. FIG. 16C illustrates a process of transforming a captured image into an image at an arbitrary viewpoint position, which is represented by the calculations of (Formula 3) to (Formula 6).

As described above, the coordinates (x, y) of the captured image are converted into the real space coordinates (X, Y, Z) by the back projection conversion processing of (Expression 3), and then the real space movement of (Expression 5) is performed. By the conversion process, it is converted into the real space coordinates (X ^* , Y ^* , Z ^* ) of the arbitrary viewpoint position. Finally, by the projection conversion process of (Equation 6), the conversion process to the coordinates (x ^* , y ^* ) of the arbitrary viewpoint image is executed. Thus, the process of transforming the captured image into an image at the arbitrary viewpoint position is completed. In the process of generating the pixel value of the arbitrary viewpoint image from the pixel value of the captured image by the coordinate conversion, general bilinear interpolation or bicubic interpolation is used. This is the end of the description of the arbitrary viewpoint image generation processing in S405 of FIG.

Next, the process of generating the arbitrary viewpoint distance MAP in S406 of FIG. 4 will be described in detail. The relationship between the arbitrary viewpoint image and the arbitrary viewpoint distance MAP will be described with reference to FIGS. 12 and 13. FIG. 12(A) shows captured images from photography 1 to photography 3 as in FIG. 11(A). First, the distance MAP corresponding to each position of the captured image is generated as shown in FIG. FIG. 13(A) shows each distance MAP from photographing 1 to photographing 3.

FIG. 12B shows each captured image shown in FIG. 12A and each arbitrary viewpoint image generated from the distance MAP shown in FIG. 13A. A region 1301 shown in black represents a region of the arbitrary viewpoint image for which information was not obtained from the captured image. Such an area is called an “occlusion area”.

FIG. 12C illustrates an arbitrary viewpoint image without an occlusion area. In order to obtain such an arbitrary viewpoint image, it is necessary to combine a plurality of arbitrary viewpoint images generated from each captured image to generate an image. At that time, as shown in FIG. 13B, a distance MAP corresponding to each arbitrary viewpoint image, that is, an arbitrary viewpoint distance MAP is required. In FIG. 13B, a region 1302 shown by hatching is an occlusion region. FIG. 13C shows the arbitrary viewpoint distance MAP without the occlusion area. In order to obtain such an arbitrary viewpoint distance MAP, it is necessary to synthesize the arbitrary viewpoint distance MAP generated from each captured image to generate the distance MAP, as in the case of the arbitrary viewpoint image. Further, the composition of the arbitrary viewpoint distance MAP also requires the recorded arbitrary viewpoint distance MAP.

Here, a process of generating the arbitrary viewpoint distance MAP using the distance MAP generated from the captured image will be described.
The coordinates of the distance MAP are (x, y), and the coordinates of the arbitrary viewpoint distance MAP are (x ^* , y ^* ). The coordinate conversion process is performed by the same method as in the case of the above-described (formula 3), (formula 5), and (formula 6) described in the method of generating an arbitrary viewpoint image. Therefore, a detailed description thereof will be omitted, and the difference from the case of the arbitrary viewpoint image will be described. In terms of performing the process of converting the value of the arbitrary viewpoint distance MAP into the value of Z ^{* in} (Equation 5). is there. The reason for performing this processing is that it is necessary to convert the shooting distance from the arbitrary viewpoint position as the arbitrary viewpoint distance MAP.

Next, a detailed description will be given of the generation process of the αMAP for synthesis shown in S408 of FIG.
The purpose of synthesizing the arbitrary viewpoint image and the arbitrary viewpoint distance MAP generated from each captured image is to generate the arbitrary viewpoint image and the arbitrary viewpoint distance MAP without the occlusion area as described above. The overall flow of the combining process will be described with reference to FIG.

FIG. 14 is a diagram showing a flow of processing for generating an arbitrary viewpoint image and an arbitrary viewpoint distance MAP in which the occlusion area is eliminated by executing the sequential combining processing. FIG. 14 shows a process of synthesizing an arbitrary viewpoint image and an arbitrary viewpoint distance MAP between photographing 1 and photographing 2, respectively. FIG. 15 further shows a combining process of the arbitrary viewpoint image and the arbitrary viewpoint distance MAP, which are the combined result of the photographing 1 and the photographing 2, and the arbitrary viewpoint image and the arbitrary viewpoint distance MAP of the photographing 3, respectively.

As shown in FIG. 14, an arbitrary viewpoint image is combined between shooting 1 and shooting 2. As a result, the size of the occlusion area shown by the black area can be reduced. The combination is performed based on the combination αMAP generated from the arbitrary viewpoint distances MAP between the photographing 1 and the photographing 2. The white area in the combining αMAP is an area in which the αMAP value is 1, and indicates the area in which the arbitrary viewpoint image of shooting 1 is combined with the arbitrary viewpoint image of shooting 1. In the area where the αMAP value is 1, the occlusion area at the arbitrary viewpoint distance MAP of the shooting 1 is an area that is not the occlusion area at the arbitrary viewpoint distance MAP of the shooting 2. Further, in the area 1401 indicated by the dotted frame within the arbitrary viewpoint distance MAP, the value of the arbitrary viewpoint distance MAP of the photographing 2 is smaller than that of the photographing 1. That is, when viewed from the arbitrary viewpoint position, the αMAP value is set to 1 even in the area indicating that the shooting 2 is located at a shorter distance or a shorter distance than the shooting 1.

Next, as shown in FIG. 15, the result of combining shooting 1 and shooting 2 and the combining processing of shooting 3 are similarly performed. As a result, it is possible to generate the arbitrary viewpoint image and the arbitrary viewpoint distance MAP in which the size of the occlusion area is further reduced as compared with FIG.

With reference to FIG. 3, the process of generating the αMAP for synthesis will be described. FIG. 3 is a block diagram showing a configuration example of the αMAP generation unit 208. The αMAP generation unit 208 includes a distance MAP value comparison unit 301, an occlusion determination unit 302, and an αMAP value generation unit 303.
The distance MAP value comparison unit 301 acquires information on the k-th arbitrary viewpoint distance MAP and information on the arbitrary viewpoint distance MAP recorded in the distance MAP recording unit 207, compares both information, and compares the comparison result with the αMAP value. Output to the generation unit 303. The occlusion determination unit 302 acquires information on the arbitrary viewpoint distance MAP recorded in the distance MAP recording unit 207, and outputs the occlusion determination result to the αMAP value generation unit 303. The αMAP value generation unit 303 acquires each output of the distance MAP value comparison unit 301 and the occlusion determination unit 302, and outputs the combination αMAP.

With reference to the flowchart in FIG. 5, a process for generating a synthesis αMAP will be described.
In S501, the αMAP generation unit 208 initializes the coordinates (x, y) used when performing the generation process in S502 and subsequent steps to generate αMAP to (0, 0). In S502, a process of reading Z_k(x, y) that is the value of the k-th arbitrary viewpoint distance MAP is executed. The distance MAP value comparison unit 301 acquires Z_k(x, y). In S503, a process of reading Z_m(x, y) which is the value of the arbitrary viewpoint distance MAP recorded in the distance MAP recording unit 207 is executed. Z_m(x, y) is acquired by the distance MAP value comparison unit 301 and the occlusion determination unit 302.

In step S504, the occlusion determination unit 302 determines whether the value of Z_m(x, y) is the ER value. The ER value is a value indicating that it is an occlusion area. When Z_m(x, y) is the ER value, it is determined that the coordinates (x, y) of the recorded arbitrary viewpoint image belong to the occlusion area, and the process proceeds to S505. If Z_m(x, y) is not the ER value, it is determined that the coordinates (x, y) of the recorded arbitrary viewpoint image do not belong to the occlusion area, and the process proceeds to S506. In the present embodiment, the ER value is set as the maximum value within the range where numerical expression is possible.
In S505, the αMAP value generation unit 303 determines the α map value. In this example, 1 is substituted for α(x,y). The αMAP value generation unit 303 determines to output the value of Z_k(x, y) for the coordinates (x, y), and then proceeds to S508.

In step S506, the distance MAP value comparison unit 301 compares the values of Z_m(x, y) and Z_k(x, y) with each other, and the subject at the coordinate (x, y) of the k-th arbitrary viewpoint image is recorded. It is determined whether or not it is located in front of the arbitrary viewpoint image being displayed. When the value of Z_k(x, y) is smaller than the value of Z_m(x, y), that is, the subject at the coordinates (x, y) of the k-th arbitrary viewpoint image is compared with the recorded arbitrary viewpoint image. If it is determined that it is located in front, the process proceeds to S505. On the contrary, when the value of Z_k(x, y) is equal to or more than the value of Z_m(x, y), the process proceeds to S507.

In S507, the αMAP value generation unit 303 determines the α map value. In this example, 0 is substituted for α(x,y). The αMAP value generation unit 303 determines not to output the value of Z_k(x, y) for the coordinate (x, y), and then proceeds to S508. S508 is a process of determining whether or not the processes of S502 to S507 have been performed on all the coordinates of the arbitrary viewpoint image. If the αMAP generation unit 208 determines that the processing has been performed for all the coordinates, the series of processing described above ends. If there are unprocessed coordinates, the process proceeds to S509. In S509, the αMAP generation unit 208 updates the value of the coordinate (x, y) to be processed, shifts to S502, and continues the processing.

In the present embodiment, the distance MAP generation unit 203, which is a distance information acquisition unit, acquires image data from the image acquisition unit 201 and generates distance information in the depth direction, and the image transformation unit 204 performs image transformation processing by coordinate transformation. To generate an arbitrary viewpoint image. The distance MAP deforming unit 205, which is a distance information deforming unit, acquires the distance information from the distance information acquiring unit and performs a deformation process by coordinate conversion to generate an arbitrary viewpoint distance MAP. The αMAP generation unit 208, which is a combination information generation unit, acquires the modified distance information and generates the combination αMAP information. The information of the αMAP for synthesis is, for example, a binary value of 1 or 0 as shown in FIG. 5 (see S505 and S507). The image composition unit 209 composes a plurality of modified images using the information of the αMAP for composition. According to the present embodiment, the occlusion area is appropriately interpolated by the sequential processing related to the arbitrary viewpoint image and the arbitrary viewpoint distance MAP to generate the arbitrary viewpoint image and the arbitrary viewpoint distance MAP in which the size of the occlusion area is reduced. You can

In the present embodiment, an example in which the information of the αMAP for synthesis is set to a binary value has been described, but the present invention is not limited to such an example. For example, when the composite information generation unit determines that the coordinates (x, y) of the distance information recorded in the distance MAP recording unit 207, which is the distance information recording unit, belongs to the occlusion area, the value of α(x, y) is set to Set relatively large. That is, the ratio of the pixel value of the image subjected to the deformation process and the distance value indicated by the first distance information subjected to the deformation process at the coordinate becomes large. In addition, when the coordinates (x, y) do not belong to the occlusion area, the composite information generation unit may calculate the distance value indicated by the modified first distance information and the second distance recorded in the distance information recording unit. The distance values indicated by the information are compared, and the value of α(x, y) is determined from the comparison result. In this case, when the distance value indicated by the first distance information is smaller than the distance value indicated by the second distance information, the value of α(x, y) is set relatively large. Further, when the distance value indicated by the first distance information is greater than or equal to the distance value indicated by the second distance information, the value of α(x, y) is set relatively small. Such changes regarding the setting process of the ratio at the time of combination are the same in the embodiments described later.
In the present embodiment, the coordinate transformation (geometrical deformation) performed to generate the image data after changing the viewpoint from the image data is also reflected in the distance MAP corresponding to the image data. However, the present invention is not limited to this, and various image processes such as distortion correction, noise reduction process, and other coordinate conversion process performed on the image data are reflected on the distance MAP corresponding to the image data to further improve the distance MAP. It may be accurately associated with the image data. In the present embodiment, such various kinds of image processing are performed in place of or in addition to the coordinate conversion, for example, in S406 of FIG. In particular, correction processing for correcting noise, distortion, defects and the like caused by devices such as an image pickup optical system, an image pickup element, each processing circuit, and image pickup conditions should be performed not only on image data to be developed but also on the distance MAP. Is preferred.
In addition, performing geometric deformation and correction processing on both the ornamental image data and the distance MAP is not limited to generating the image data after the viewpoint change, but can be applied to various applications. For example, when geometrical deformation or correction processing is performed on ornamental image data due to the above-described device or imaging condition, and development processing is performed to display and record, similar geometrical deformation or correction processing is performed on the distance MAP. It is effective to carry out and record as the distance MAP corresponding to the image data.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the present embodiment, in addition to the configuration described in the first embodiment, a configuration unit relating to the reliability at the time of generating the distance MAP is provided. When the distance MAP is generated from the captured image, the reliability MAP is simultaneously generated as the reliability information corresponding to the distance MAP. As for the reliability MAP, the deformation process and the combining process are performed on the arbitrary viewpoint position, similarly to the distance MAP and the captured image. In the following, the same components as those in the first embodiment will be denoted by the same reference numerals as used above, and a detailed description thereof will be omitted. Differences from the first embodiment will be mainly described.

The reliability MAP will be described with reference to FIG. FIG. 17A illustrates the distance MAP generated from the captured image. As a method of generating the distance MAP, for example, a known method using a correlation calculation using SAD values of a plurality of pupil-divided images or images having different viewpoints is used. FIG. 17A illustrates a distance MAP relating to a subject area 1701 that is close to the image capturing apparatus and a subject area 1702 that is far from the image capturing apparatus. When the above method is used, the result of the correlation calculation in the shaded boundary region 1601 shown in FIG. That is, the result of the correlation calculation may not be accurate in the boundary area 1601 between the subject area 1701 having a short distance from the imaging device and the subject area 1702 having a long distance. Therefore, in this embodiment, the reliability MAP indicating whether or not the distance is reliable is used as the distance MAP value. FIG. 17C illustrates the reliability MAP. For example, a region 1602 shown in FIG. 17C corresponds to the shaded boundary region 1601 shown in FIG. 17B. In the area 1602, the value of the reliability MAP is set smaller than that in the surrounding area. The region where the result of the correlation calculation is not accurate includes not only the boundary region as described above, but also a subject region having a repeated pattern in which low contrast and the same texture are present in the image many times. Similarly, for such a region, the value of the reliability MAP is set small. The larger the reliability value, the more reliable the value of the corresponding distance map. As a method of generating the reliability MAP, there is a method of judging from the SAD value by a correlation calculation. For example, in the DFD method, the reliability is a value representing the influence of defocus. A region that is less likely to be affected by defocus means a region that has few clues for calculating the distance by the DFD method. Therefore, a region having a small reliability value is a region where the calculated distance value is not accurate. Is.

The internal configuration of the image processing unit 604 (see FIG. 1) according to the present embodiment will be described in detail with reference to the block diagram of FIG. The difference from the configuration shown in FIG. 2 is that the distance MAP/reliability MAP generation unit 703, the reliability MAP modification unit 708, the reliability MAP recording unit 709, the αMAP generation unit 710, the distance MAP combination unit 712, and the reliability MAP combination. It is a part 713. With respect to the other components, the same reference numerals as those given to the respective units shown in FIG. 2 are used, and the detailed description thereof will be omitted.

The distance MAP/reliability MAP generation unit 703 acquires the image data output by the image acquisition unit 201 and generates the distance MAP and the reliability MAP. The data of the distance MAP is output to the image transforming unit 204 and the distance MAP transforming unit 205, and the data of the reliability MAP is output to the reliability MAP transforming unit 708. The reliability MAP modification unit 708 outputs the data of the reliability MAP after the modification process to the reliability MAP synthesis unit 713. The reliability MAP combining unit 713 acquires the outputs of the reliability MAP transforming unit 708, the reliability MAP recording unit 709, and the αMAP generating unit 710, and outputs the reliability MAP data after the reliability combining process to the reliability MAP recording unit. Output to 709.

The processing of each unit will be described with reference to the flowchart in FIG.
When image capturing is started, the image processing unit 604 initializes by substituting 1 into the variable k used in the sequential processing in step S901. In step S902, the image acquisition unit 201 performs a process of acquiring the kth captured image. In S903, the distance MAP/reliability MAP generation unit 703 generates each data of the kth distance MAP corresponding to the kth image acquired in S902 and the reliability MAP corresponding to the kth distance MAP. A known method is used for the generation processing of the distance MAP and the reliability MAP (Patent Document 2).

In step S904, the image development unit 202 performs development processing on the kth image data input in step S902. In step S905, the image transformation unit 204 performs transformation processing on the kth image developed in step S904 using the kth distance MAP generated in step S903 to generate data of the kth arbitrary viewpoint image. In step S906, the distance MAP/reliability MAP generation unit 703 inputs the kth distance MAP generated in step S903 and generates the kth arbitrary viewpoint distance MAP. The development processing, the arbitrary viewpoint image data generation processing, and the arbitrary viewpoint distance MAP generation processing are the same as those in the first embodiment.

In step S907, the reliability MAP transforming unit 708 receives the kth reliability MAP generated in step S903 as input, and generates the reliability MAP at the kth arbitrary viewpoint position. The reliability MAP at the arbitrary viewpoint position is referred to as "arbitrary viewpoint reliability MAP". The method of generating the arbitrary viewpoint reliability MAP is the same as the method of generating the arbitrary viewpoint image in S905, and thus the description thereof will be omitted.

In step S<b>908, the distance MAP synthesizing unit 712 reads the data of the arbitrary viewpoint distance MAP that has already been processed up to the “k−1”th position and recorded in the distance MAP recording unit 207. In step S<b>909, the reliability MAP combining unit 713 reads the data of the arbitrary viewpoint reliability MAP recorded in the reliability MAP recording unit 709 after being processed up to the “k−1”th position.

In step S910, the αMAP generation unit 710 inputs the data of the kth arbitrary viewpoint distance MAP and the recorded arbitrary viewpoint distance MAP, and the kth arbitrary viewpoint reliability MAP and the recorded arbitrary viewpoint reliability MAP. Generate an αMAP for. The αMAP for synthesis generated by the αMAP generation unit 710 is used in S912, S913, and S914 described below. The process of generating the αMAP for synthesis will be described in detail later.

In step S911, the process of reading the already recorded arbitrary viewpoint image up to the k-1th is performed, and then in step S912, the kth arbitrary viewpoint image and the recorded arbitrary viewpoint image are generated in step S910. A process of combining is executed based on the αMAP. In S913, a process of synthesizing the k-th arbitrary viewpoint distance MAP and the recorded arbitrary viewpoint distance MAP based on the αMAP generated in S910 is performed. The processing from S911 to S913 is the same as in the case of the first embodiment.

In S914, the reliability MAP combining unit 713 combines the kth arbitrary viewpoint reliability MAP and the arbitrary viewpoint reliability MAP recorded in the reliability MAP recording unit 709 based on the αMAP generated in S910. The synthesizing method is the same as in S912 and S913. In S915, a process of updating the arbitrary viewpoint image synthesized in S912 as an arbitrary viewpoint image recorded in the image recording unit 206 is performed. In S916, a process of updating the arbitrary viewpoint distance MAP synthesized in S913 as the arbitrary viewpoint distance MAP recorded in the distance MAP recording unit 207 is performed. In S917, the reliability MAP combining unit 713 updates the arbitrary viewpoint reliability MAP combined in S914 as the arbitrary viewpoint reliability MAP recorded in the reliability MAP recording unit 709.

S918 is a process of determining whether or not the image capturing by the image capturing apparatus is completed, and the image processing unit 604 determines whether or not the "k+1"th image is input. When the imaging by the imaging device is finished and the “k+1”th image is not input, the process proceeds to S919. If the "k+1"th image is input without the end of the image capturing by the image capturing apparatus, the process proceeds to S920. In step S919, the image processing unit 604 outputs, as an output image, the arbitrary viewpoint image created by the k images so far, and ends the series of processes. The arbitrary viewpoint image data output from the image processing unit 104 is encoded and recorded in the same image file including the corresponding distance MAP and reliability MAP. The image file conforms to the EXIF file format, for example, and the distance MAP and the reliability MAP are recorded as metadata. Alternatively, the output arbitrary viewpoint image data is recorded in separate files in association with the corresponding distance MAP and reliability MAP.
In step S920, the “k+1”th image is input and updated by incrementing the variable k by adding 1 to the k value, and the process proceeds to step S902.

Next, the αMAP generation process in S910 of FIG. 8 will be described in detail. FIG. 7 is a block diagram showing a configuration example of the αMAP generation unit 710. The αMAP generation unit 710 includes a reliability MAP value comparison unit 801, a distance MAP value comparison unit 802, an occlusion determination unit 803, and an αMAP value generation unit 804. The following data is input to the αMAP generation unit 710, and the αMAP for synthesis is output.
The kth arbitrary viewpoint distance MAP and the arbitrary viewpoint distance MAP recorded in the distance MAP recording unit 207.
The k-th arbitrary viewpoint reliability MAP and the arbitrary viewpoint reliability MAP recorded in the reliability MAP recording unit 709.

The reliability MAP value comparison unit 801 compares each value of the k-th arbitrary viewpoint reliability MAP and the arbitrary viewpoint reliability MAP recorded in the reliability MAP recording unit 709, and the comparison result is the distance MAP value comparison unit. Output to 802.

With reference to the flowchart of FIG. 9, the process of generating the αMAP for synthesis will be described.
In S1001, the αMAP generation unit 710 initializes the coordinates (x, y) used in the generation processing in S1002 and subsequent steps to generate αMAP to (0, 0). In step S1002, the distance MAP value comparison unit 802 reads Z_k(x, y), which is the value of the k-th arbitrary viewpoint distance MAP. In step S1003, the distance MAP value comparison unit 802 and the occlusion determination unit 803 read Z_m(x, y), which is the value of the arbitrary viewpoint distance MAP recorded in the distance MAP recording unit 207. In step S1004, the reliability MAP value comparison unit 801 reads R_k(x, y), which is the value of the k-th arbitrary viewpoint reliability MAP. In step S1005, the reliability MAP value comparison unit 801 reads R_m(x, y), which is the value of the arbitrary viewpoint reliability MAP recorded in the reliability MAP recording unit 709.

In step S1006, the occlusion determination unit 803 determines whether the value of Z_m(x,y) is the ER value. The ER value represents the occlusion area described above. If the value of Z_m(x, y) is the ER value, it is determined that the coordinates (x, y) of the recorded arbitrary viewpoint image belong to the occlusion area, and the process proceeds to S1007. On the other hand, when the value of Z_m(x, y) is not the ER value, it is determined that the coordinates (x, y) of the recorded arbitrary viewpoint image do not belong to the occlusion area, and the process proceeds to S1008.

In S1007, the αMAP value generation unit 804 substitutes 1 into the αMAP value α(x, y), determines that the Z_k(x, y) value is output for the coordinates (x, y), and then proceeds to S1012. .. In step S1008, the distance MAP value comparison unit 802 compares each value of Z_m(x, y) and Z_k(x, y), and the subject at the coordinate (x, y) of the k-th arbitrary viewpoint image is recorded. It is determined whether or not the subject is located in front of the subject in the arbitrary viewpoint image. When the value of Z_k(x, y) is smaller than the value of Z_m(x, y), that is, the subject at the coordinates (x, y) of the k-th arbitrary viewpoint image is recorded in the arbitrary viewpoint image. When it is determined that the object is located in front of the subject, the process proceeds to S1009. If the value of Z_k(x, y) is larger than the value of Z_m(x, y), the process proceeds to S1010.

In step S1009, the reliability MAP value comparison unit 801 determines the difference value between the value R_k(x, y) of the kth arbitrary viewpoint reliability MAP and the recorded value R_m(x, y) of the arbitrary viewpoint reliability MAP. Is calculated and the difference value is compared with a first threshold value (denoted as TH1). As the first threshold TH1, a fixed value that is determined in advance or a variable value that changes according to the value of the reliability MAP is used. When the difference value between R_k(x,y) and R_m(x,y) is larger than the first threshold value TH1, the reliability MAP value comparison unit 801 compares R_k(x,y) with R_m(x,y). It is determined that the reliability is sufficiently high, and the process proceeds to S1007. When the difference value between R_k(x, y) and R_m(x, y) is less than or equal to the first threshold TH1, the process proceeds to S1011.

In S1010, the reliability MAP value comparison unit 801 compares the difference value between R_k(x, y) and R_m(x, y) with a second threshold value (denoted as TH2), and the difference value is higher than the second threshold value TH2. Judge whether it is large or not. As the second threshold TH2, a predetermined fixed value or a variable value that changes according to the value of the reliability MAP is used. In the present embodiment, the second threshold TH2 is different from the first threshold TH1, but the same threshold may be used if necessary. When the difference value between R_k(x, y) and R_m(x, y) is larger than the second threshold value TH2, R_k(x, y) has a sufficiently higher reliability than R_m(x, y). The determination is made, and the process proceeds to S1007. When the difference value between R_k(x, y) and R_m(x, y) is less than or equal to the threshold value TH2, the process proceeds to S1011. In step S1011, the αMAP value generation unit 804 substitutes 0 into the value α(x, y) of αMAP, determines that the value of Z_k(x, y) is not output for the coordinate (x, y), and then proceeds to step S1012. move on.

In S1012, the αMAP generation unit 710 determines whether or not the processes from S1002 to S1011 have been performed on all the coordinates corresponding to the arbitrary viewpoint image. When it is determined that the processing has been performed for all the coordinates, the αMAP generation unit 710 ends the processing. If the process is not completed, the process proceeds to S1013. In S1013, the αMAP generation unit 710 updates the coordinates (x, y) to be processed, proceeds to S1002, and continues the processing.

In the present embodiment, the reliability information generation unit that generates reliability information indicating the reliability of the distance information calculated from the image data is provided. The embodiment is not limited to the embodiment in which the reliability information is generated at the time of generating the distance information, but may be an embodiment in which the reliability information generation unit other than the distance information generation unit generates the reliability information. The reliability MAP synthesizing unit 713, which is a reliability information synthesizing unit, includes reliability information that has been transformed by the reliability MAP transforming unit 708, which is a reliability information transforming unit, and a reliability MAP recording unit, which is a reliability information recording unit. The reliability information recorded in 709 is acquired. The reliability MAP synthesis unit 713 synthesizes reliability information using the information of the αMAP for synthesis. According to the present embodiment, it is possible to provide an image processing apparatus that enables sequential processing of generating an arbitrary viewpoint image by appropriately interpolating an occlusion area. By using the reliability MAP as the reliability information corresponding to the distance MAP, accurate distance information can be reflected in the arbitrary viewpoint image generation process.

[Other Embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

104 Image Processing Unit 203 Distance MAP Generation Unit 204 Image Deformation Unit 205 Distance MAP Deformation Unit 208 αMAP Generation Unit 209 Image Synthesis Unit 210 Distance MAP Synthesis Unit

Claims (19)

Image acquisition means for acquiring a plurality of image data,
Distance information acquisition means for acquiring distance information in the depth direction of the image data acquired from the image acquisition means,
Image transforming means for transforming the image by coordinate transformation on the image data acquired from the image acquiring means;
Distance information transformation means for performing transformation processing on the distance information acquired by the distance information acquisition means by coordinate transformation corresponding to coordinate transformation performed on the image data,
Distance information synthesizing means for synthesizing a plurality of distance information transformed by the distance information transforming means,
Distance information recording means for recording distance information output by the distance information combining means,
Synthetic information generating means for generating synthetic information based on the distance information transformed by the distance information transforming means and the distance information recorded in the distance information recording means,
Based on the information for the synthesis, the image processing device characterized by and an image synthesizing means for generating an arbitrary viewpoint image by synthesizing a plurality of images transformation processing by the image deforming means. The image according to claim 1, wherein the coordinate conversion performed on the plurality of image data is a movement conversion process based on a movement amount from a shooting position of the plurality of images to a viewpoint position of the arbitrary viewpoint image. Processing equipment. The coordinate conversion performed on the plurality of image data is a coordinate conversion in which at least one of translation and rotation is performed based on a movement amount from a shooting position of the plurality of images to a viewpoint position of the arbitrary viewpoint image. The image processing apparatus according to claim 1 or 2. The image acquisition means sequentially acquires the plurality of image data,
Said distance information combining means, corresponding to said plurality of image data, any one of claims 1 to 3, characterized in that sequentially synthesizing a plurality of distance information modification process is performed by the distance information modification unit 1 The image processing device according to item . Before Symbol synthesis information generation unit, among the distance information distance information is transformation processing by the deformation means and the distance information recorded on the distance information recording means, the distance information indicating that the subject is present in a closer distance The image processing apparatus according to any one of claims 1 to 4, wherein the image information is selected to generate the information for composition. A distance information synthesizing unit that obtains and synthesizes a plurality of distance information corresponding to a plurality of image data, the distance information being transformed by the distance information transforming unit;
The distance information synthesizing unit synthesizes the distance information transformed by the distance information transforming unit and the distance information recorded in the distance information recording unit using the information for synthesis acquired from the synthesis information generating unit. The image processing apparatus according to claim 5 , wherein: Wherein the image deforming means is an image processing according to any one of claims 1 to 6, characterized in that the deformation processing by the coordinate transformation for generating image data for changing the viewpoint to the image data apparatus. The distance information modification means, according to any one of claims 1 to 7, characterized in that the conversion of distance information associated with the change of the viewpoint with respect to the distance information distance information acquired by the acquiring means Image processing device. Said distance information obtaining means, according to any one of claims 1 to 8, characterized in that to obtain the distance information generated from the data of the pupil divided plurality of image or view of different images Image processing device. A reliability information generating unit that generates reliability information indicating the reliability of the distance information acquired by the distance information acquiring unit;
The combined information generating means, any one of claims 1 to 9, characterized in that to generate the information for the synthesis using the distance information and the reliability information modification process by the distance information modification means The image processing device according to. Reliability information deforming means for acquiring the reliability information generated by the reliability information generating means and performing a deformation process by coordinate conversion;
The image processing apparatus according to claim 10 , further comprising a reliability information synthesizing unit that acquires a plurality of pieces of reliability information transformed by the reliability information transforming unit and sequentially synthesizes the pieces of reliability information. A reliability information recording unit for recording reliability information output by the reliability information combining unit;
The reliability information synthesizing unit records the reliability information and the reliability information recording unit that has been transformed by the reliability information transforming unit using the information for synthesis acquired from the synthesis information creating unit. The image processing apparatus according to claim 11 , wherein the reliability information is combined. The composite information generation unit represents the composition ratio by comparing the first distance information modified by the distance information modification unit with the second distance information recorded in the distance information recording unit. the image processing apparatus according to any one of claims 1 to 4, characterized in that to determine the information for the synthesis. The composite information generation unit determines an occlusion area which is an area at a viewpoint where information cannot be obtained from an image, and when the coordinates of the second distance information belong to an occlusion area, the transformation processing at the coordinates is performed. 14. The image processing apparatus according to claim 13 , wherein the ratio at the time of synthesizing the pixel value of the image and the distance value indicated by the first distance information is set relatively large. In the case where the coordinates of the second distance information do not belong to the occlusion area, the combined information generating unit makes the distance value indicated by the first distance information smaller than the distance value indicated by the second distance information. At this time, the pixel value of the transformed image and the distance value indicated by the first distance information at the coordinates are set to be relatively large, and the first distance information is also set. When the distance value indicated by is greater than or equal to the distance value indicated by the second distance information, the pixel value of the deformed image at the coordinates and the distance value indicated by the first distance information are respectively combined. 15. The image processing apparatus according to claim 14 , wherein the ratio at the time of performing is set to be relatively small. The combination information generation unit acquires the plurality of pieces of distance information transformed by the distance information transformation unit and the plurality of pieces of reliability information generated by the reliability information generation unit, and indicates the proportion of the combination. The image processing device according to claim 10 , wherein the image processing device generates information. An image pickup apparatus comprising the image processing apparatus according to any one of claims 1 to 16 . An image processing method executed by an image processing device for processing image data, comprising:
An image acquisition step of acquiring a plurality of image data,
A distance information acquisition step of acquiring distance information in the depth direction of the image data acquired in the image acquisition step,
An image transformation step of transforming the image by coordinate transformation on the image data obtained in the image obtaining step;
Distance information transformation step of performing transformation processing by coordinate transformation corresponding to coordinate transformation performed on the image data with respect to the distance information obtained in the distance information obtaining step;
A distance information synthesizing step of synthesizing a plurality of distance information transformed in the distance information transforming step,
A distance information recording step of recording the distance information combined in the distance information combining step,
A combined information generating step of generating information for combining based on the distance information modified in the distance information modifying step and the distance information recorded in the distance information recording step;
An image combining step of generating an arbitrary viewpoint image by combining a plurality of images that have been deformed in the image deforming step based on the combining information. A program that causes a computer of the image processing apparatus to execute each step according to claim 18 .