JP2013175821A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow for refocus processing capable of reproducing a smooth blurring while suppressing a calculation cost.SOLUTION: In step S401, a user operates a shutter button 111 to photograph an image. In step S402, the photographed image is input to an image input unit 301, and development processing is performed by a development processing unit 302 to generate a multi-viewpoint image. In step S403, a depth map is generated from the generated multi-viewpoint image. In step S404, the number of intermediate viewpoint images to be generated is calculated using the generated depth map. In step S405, intermediate viewpoint images are generated on the basis of the calculated number of intermediate viewpoint images using the generated multi-viewpoint image and the generated depth map. In step S408, the images are synthesized by an image synthesizing unit 308 using the multi-viewpoint images generated in step S402, the intermediate viewpoint images generated in step S405, and refocus parameters set in step S407.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more specifically to an image processing method, an image processing apparatus, and a program that generate an image using a plurality of images having different viewpoint positions.

  Conventionally, there is a technique for generating a new image by combining images taken from two or more viewpoints (hereinafter, multi-viewpoint images). For example, a refocus technique has been proposed in which a virtual focus plane and a blur amount are set for a multi-viewpoint image and are synthesized by image processing to generate a new image (for example, Patent Document 1 and Non-Patent Document 1). Patent Document 1). In the refocus technique, each image is synthesized so that the subject position matches on the set focus plane, thereby generating an image focused on the set focus plane. On the other hand, a blurred image can be reproduced on the non-focus surface by shifting the position on the non-focus surface and compositing. Further, the size of blur can be controlled by adjusting the number of sheets to be combined and the weight. In Japanese Patent Laid-Open No. 2004-228688, the size of blur is adjusted and characteristic blur is reproduced by weighting and synthesizing based on light ray information at the time of multi-viewpoint image shooting. In Non-Patent Document 1, it is possible to reproduce smooth blur by estimating an intermediate viewpoint image of a multi-viewpoint image.

JP 2009-159357 A

Natsumoto Enomoto et al., “Defocus Control by Uncalibrated Synthetic Aperture Imaging”, MIRU 2008 Image Recognition and Understanding Symposium, Japan, 2008, pp. 1057-1062

  However, in the case of the method disclosed in Patent Document 1, although the calculation cost can be reduced, since the blur is reproduced based only on the light ray information at the time of multi-viewpoint shooting, smooth blur is obtained when the subject has a depth. There is a problem that it cannot be reproduced. On the other hand, the method of Non-Patent Document 1 can reproduce smooth blur even if the subject has a depth, but generates an intermediate viewpoint image between all the multi-viewpoint images. There is a problem that the calculation cost increases due to a large number.

  The present invention is an image processing apparatus, which is determined in advance according to multi-view image input means for inputting multi-view image data acquired by an image capturing apparatus having a plurality of image capturing units, and feature quantities of the multi-view image. Intermediate viewpoint image generation number calculating means for calculating the number of intermediate viewpoint images that need to be generated in order to obtain a composite image satisfying the above characteristics, and the multi-viewpoints of the generation numbers calculated by the intermediate viewpoint image generation number calculating means An intermediate viewpoint image generation unit that generates intermediate viewpoint image data of image data, and an image synthesis unit that generates composite image data using the intermediate viewpoint image data generated by the intermediate viewpoint image generation unit .

  The present invention calculates the number of intermediate viewpoint images generated based on the feature values between multi-viewpoint images, and can switch between the density of intermediate viewpoint images according to the value to reproduce smooth blur while reducing the calculation cost. Refocus processing is possible.

1 is a diagram illustrating an example of an imaging device in Embodiment 1. FIG. 1 is a block diagram illustrating an internal configuration of an imaging apparatus according to Embodiment 1. FIG. 3 is a block diagram illustrating a configuration of image processing in Embodiment 1. FIG. 3 is a flowchart illustrating a flow of processing in the first embodiment. 3 is an example of a user interface for setting a refocus parameter in the first embodiment. It is a flowchart which shows the flow of the process in the depth map production | generation of Example 1. FIG. 6 is a diagram illustrating a relationship between an imaging position interval and a distance to a subject in Embodiment 1. FIG. 7 is a flowchart illustrating a processing flow in calculating the number of intermediate viewpoint images generated according to the first exemplary embodiment. FIG. 10 is a diagram illustrating an example of calculating a shift amount between multi-viewpoint images in the first embodiment. 6 is a diagram illustrating an example of the number of intermediate viewpoint images generated in Embodiment 1. FIG. 3 is a flowchart illustrating a processing flow of an image composition unit according to the first exemplary embodiment. FIG. 6 is a diagram illustrating an overview of a focus position and a shift amount in the first embodiment. It is a figure for demonstrating the weighting coefficient in Example 1. FIG. 10 is a flowchart illustrating a processing flow in the second embodiment. 10 is a flowchart illustrating a flow of processing in calculating similarity between images according to the second exemplary embodiment. FIG. 10 is a diagram illustrating an example of the number of intermediate viewpoint images generated in the second embodiment. 10 is a flowchart showing a flow of processing in Embodiment 3. 10 is a flowchart illustrating frequency analysis of an image in Example 3. 10 is a schematic diagram illustrating frequency characteristics of a multi-viewpoint image in Embodiment 3. FIG. It is a figure which shows the example of the production | generation number of the intermediate viewpoint image in Example 3. FIG. 2 is a diagram illustrating an example of a back surface of the imaging apparatus according to Embodiment 1. FIG.

[Example 1]
1 and 21 are diagrams illustrating an example of an imaging apparatus according to the present embodiment. As illustrated in FIG. 1, the housing 101 of the imaging apparatus includes multi-eye imaging units 102 to 110. Further, as shown in FIG. 21, an imaging button 111 for starting an operation is provided on the upper part of the imaging apparatus 101. Further, an operation button 112 for operating the image pickup apparatus 101 and a display unit 113 for displaying an image and a graphic user interface are provided on the back surface of the image pickup apparatus 101.

  FIG. 2 is a block diagram illustrating an internal configuration of the imaging apparatus 101 in the present embodiment. The multi-view imaging units 102 to 110 include a zoom lens, a focus lens, a shake correction lens, an aperture, a shutter, an optical low-pass filter, an iR cut filter, a color filter, a CMOS or CCD image sensor, and the like. With such a configuration, it is possible to detect the amount of light of the subject focused through the lens. The A / D conversion unit 201 converts the amount of light of the subject detected by the multiview imaging units 102 to 110 into a digital signal value. The image processing unit 202 performs image processing on the converted digital signal value, details of which will be described later. The D / A converter 203 converts the digital signal value into an analog signal. The encoder unit 204 converts the digital signal value into various file formats known in this technical field such as Jpeg. The media I / F 205 is an interface for connecting to a PC / media 206 (for example, a hard disk, a memory card, a CF card, an SD card, a USB memory, etc.).

  The CPU 207 is involved in all the processes of each configuration, reads the instructions stored in the ROM 208 and the RAM 209 in order, interprets them, and executes the processes according to the results. In addition, the imaging system control unit 210 performs control instructed by the CPU 207 such as adjusting the focus, opening the shutter, and adjusting the aperture for the multi-eye imaging units 102 to 110. The control unit 211 performs various controls such as start and end of the entire process according to a user instruction from the shutter button 111 or the operation button 112. The character generation unit 212 generates characters and graphics and displays them on the display unit 113. The display unit 113 typically uses a liquid crystal display or the like, and displays image data, characters, graphic interfaces, and the like received from the character generation unit 212 and the D / A conversion unit 203. In addition, the display unit 113 can be provided with a touch screen function. In this case, a user instruction via the touch screen can be input to the control unit 211. In the present embodiment, as shown in FIG. 1, nine multi-view imaging units are exemplified, but the present invention is not limited to this as long as it has two or more imaging units.

  FIG. 3 is a block diagram illustrating functions for executing image processing according to the present embodiment. An image input unit 301 serving as a multi-viewpoint image input unit receives image data captured by the multi-viewpoint imaging units 102 to 110 and generated as a digital signal value by the A / D conversion unit 201. The development processing unit 302 performs processing such as demosaic, white balance, noise reduction, gamma, and sharpness on the image data input by the image input unit 301.

  The intermediate viewpoint image generation number calculation unit 303 calculates the optimum number of intermediate viewpoint images generated, that is, the minimum number of generations necessary to reproduce smooth blur according to each multi-viewpoint image. That is, when the number of intermediate viewpoint images is small, grid-like noise is generated when multi-viewpoint image data and intermediate viewpoint image data are combined. In general, even if the number of required intermediate viewpoint images differs depending on the multi-viewpoint image, if a large amount of intermediate viewpoint image data is calculated as in the past, smooth blur can be reproduced, but a wasteful calculation cost is required. End up. Therefore, in order to solve such a problem, a feature value between multi-view images or multi-view images is calculated as an index for calculating the optimum number of intermediate viewpoint images, and a multi-view image is calculated based on the calculated feature value. The optimal number of intermediate viewpoint images to be interpolated is calculated.

  The intermediate viewpoint image generation unit 304 generates an intermediate viewpoint image from the multi-viewpoint image data that has been developed based on the number of generations calculated by the intermediate viewpoint image generation number calculation unit 303. The refocus parameter input unit 305 receives a focus position 306 and a blur size 307 designated by the user using the operation buttons 112. The image composition unit 308 performs image composition using the multi-viewpoint image, the intermediate viewpoint image, the focus position 306, and the blur size 307. The image output unit 309 converts the composite image data into an analog signal by the D / A conversion unit 203 and displays it on the display unit 113 or encodes it by the encoder unit 204 and outputs it to the PC / media 206.

  FIG. 4 is a flowchart illustrating a processing flow of the imaging apparatus according to the present exemplary embodiment. In this embodiment, the number of intermediate viewpoint images generated is calculated using a depth map as a feature amount. Here, the depth map is obtained by mapping the depth information, that is, information on the distance of the subject recorded in the image from the imaging device on the image, and in this embodiment, in order to determine the number of intermediate viewpoint images generated. This is used, but is not limited to this. That is, in this embodiment, in order to determine the number of intermediate viewpoint images generated so that the blur processing system can be optimized according to the distance between subjects, some index such as a depth map indicating the distance between subjects is used. However, other indicators known in the art can also be used.

  In step S401, the user operates the shutter button 111 to perform shooting. In step S402, the image data captured in step S401 is input to the image input unit 301, and development processing is performed in the development processing unit 302, thereby generating multi-viewpoint image data.

  In step S403, a depth map is generated from the multi-viewpoint image generated in step S402. In step S404, the number of intermediate viewpoint images generated is calculated using the depth map generated in step S403 (details of steps S403 and S404 will be described later).

  In step S405, based on the number of intermediate viewpoint images generated in step S404, intermediate viewpoint image data is generated using the multi-viewpoint image data generated in step S402 and the depth map generated in step S403. Generate. Specifically, first, the multi-viewpoint image is divided into block areas of an arbitrary size, and pattern matching for searching the corresponding area in block units is performed to obtain a pixel (corresponding pixel) corresponding to the target pixel from the target image. The search method for the corresponding pixel is not limited to these, and any method may be used as long as the corresponding pixel is searched for the multi-viewpoint image. Next, intermediate viewpoint image data at the virtual camera position is generated using the searched corresponding pixel.

  The intermediate viewpoint image data generation method at the virtual camera position may be any method as long as the viewpoint image data at the virtual camera position is generated from the corresponding pixel. For example, since the corresponding camera position is known, the depth to the corresponding pixel is calculated by triangulation, and the arbitrary image position at the virtual camera position is calculated using the calculated depth information of the corresponding corresponding pixel. To do. In step S406, an image for use when the user sets the refocus parameter is displayed on the display unit 113. Here, the image to be displayed is preferably an image obtained by photographing the entire target subject. For example, an image photographed by the multi-eye imaging device 106 is displayed. In step S407, the focus position and the blur size, which are parameters necessary for refocusing, are set.

  FIG. 5 is an example of a user interface for setting parameters necessary for refocusing. As shown in FIG. 5, in step S406, in addition to the displayed parameter setting image, a bar 501 for specifying the size of the blur and a pointer for setting the focus position 502 are simultaneously displayed on the display unit. . The user operates the operation button 112 or the bar 501 or the pointer 502 displayed on the display unit 113 to thereby adjust the focus position 306 and the size of the blur that are parameters necessary for refocusing. 307 is set.

  In step S408, the image composition unit 308 performs image composition using the multi-viewpoint image data generated in step S402, the intermediate viewpoint image data created in step S405, and the refocus parameter set in step S407. Details of step S408 will be described later. Finally, in step S409, the image data synthesized in step S408 is displayed on the display unit 113 by the image output unit 309 or transmitted to the PC / media 206. However, the present invention is not limited to this, and is known in this technical field. Output in output form.

<Generation of depth map>
Here, the generation of the depth map performed in step S403 will be described. The depth map is generated by estimating the depth of the captured scene based on the multi-viewpoint image. The existing method is used for the depth estimation method. For example, a stereo method or a multi-baseline stereo method can be applied. In this embodiment, depth estimation is performed by a stereo method, and a depth map is generated.

  FIG. 6 is a flowchart showing the flow of depth map generation processing. In step S601, two images to be used for processing are selected from the multi-viewpoint images. At this time, all pairs of multi-viewpoint images may be selected, and the processing described later may be performed on all pairs. The same subject is reflected in two images, and the set of the most distant baseline lengths. It may be done only for pairs. In general, in the stereo method, the greater the change in parallax with respect to the change in depth, the higher the measurement accuracy. Therefore, since the parallax increases as the baseline length increases, the depth map generation accuracy described later is improved. In the present embodiment, a case where a depth map is calculated from an image captured by the left center camera unit 105 of the image capturing apparatus 101 and an image captured by the right center camera unit 107 will be described as an example. Hereinafter, the former is a reference image and the latter is a target image.

  In step S602, the depth value of the image of interest to be processed thereafter is initialized. In step S603, it is determined whether the depth value calculated in S605 has been obtained for all pixels. If the depth value is obtained for all the pixels, the process proceeds to step S607 to execute the output process, and if there is a pixel for which the depth value is not obtained, the process proceeds to step S604 to obtain the depth value for the ancestor. In step S604, the multi-viewpoint image is divided into block regions of an arbitrary size, and pattern matching for searching for the corresponding region in block units is performed to obtain a pixel (corresponding pixel) corresponding to the target pixel from the target image.

  Next, in step S605, a depth value d corresponding to the target pixel is obtained based on the target pixel and the corresponding pixel obtained in S604. As can be understood with reference to FIG. 7, the depth value d is expressed by the following expression based on α, β, and b. Here, α is calculated from the horizontal angle of view of the camera unit 105, the imaging position of the reference image, and the coordinates of the target pixel. β is calculated from the horizontal angle of view of the camera unit 107, the imaging position of the target image, and the coordinates of the target pixel. b is the distance between the camera units 105 and 107, and is calculated from the imaging position of the reference image and the target image. X and z are axes indicating the horizontal direction and the depth direction, respectively.

  In step S606, the target pixel is updated, and the process returns to S603 to execute the same processing. If it is determined in step S603 that the depth value has been obtained for all the pixels, a depth map having each pixel value as the depth value of the reference image is output in step S607.

<Calculation of the number of intermediate viewpoint images generated>
The calculation of the number of intermediate viewpoint images generated in step S404 in FIG. 4 will be described. The number of intermediate viewpoint images generated in the present embodiment is calculated based on the depth range obtained from the depth map generated in step S403. Since an image with a large depth range has a large parallax, it is impossible to reproduce blur with high accuracy unless the intermediate viewpoint images are generated densely. Conversely, an image with a small depth range has a small parallax, so that there are fewer intermediate viewpoint images. Hereinafter, with reference to the flowchart shown in FIG. 8, the depth range indicating the depth of the subject is calculated, and calculation of the number of intermediate viewpoint images generated will be described.

In the present embodiment, an example of an image captured by the center camera unit 106 of the image capturing apparatus 101 and an image captured by the right center camera unit 107 will be described with reference to FIG. In the present embodiment, the depth range D calculated as follows based on the maximum distance from the imaging device to the subject and the minimum rejection door is used as the depth range, but the present invention is not limited to this. In FIG. 9, b represents a distance between multi-viewpoint images, w represents the number of horizontal pixels, θ _w represents a horizontal angle of view, and d represents a depth value. First, in step S801, a depth map is input. Next, in step S802, with respect to the depth map input in step S801, the pixel value of each pixel (i, j) of the depth map is set as the distance value d (i, j), and the minimum distance value _dmin and the maximum A distance value d _max is calculated. In step S803, in step S802, the calculating the how many pixels disparity between multi-viewpoint images the calculated minimum distance value d _min and the maximum distance value d _max based on the shift amount. The shift amount S _min of the subject that is d _min , that is, the subject that is closest to the scene can be obtained by the following equation.

Similarly, the shift amount S _max of the subject that is d _max , that is, the subject that is the deepest in the scene can be obtained by the following equation.

Therefore, the depth range D [pix] is calculated from S _min and S _max by the following equation.

In the present embodiment, an integer value obtained by rounding up the depth range D [pix] is set as the number of intermediate viewpoint images to be generated. In the present embodiment, the number of intermediate viewpoint images generated is calculated using multi-viewpoint images captured by the camera units 106 and 107 arranged in the horizontal direction as an example. However, if arranged in the vertical direction, in the formula (2) and (3), likewise depth range D by replacing the number of vertical pixels the number of horizontal pixels w h, the horizontal angle theta _w in the vertical angle theta _h The number of intermediate viewpoint images generated can be obtained by calculating [pix].

  FIG. 10 is a graph showing the relationship between the depth range and the number of intermediate viewpoint images generated. In FIG. 10, a rectangle drawn with a solid line represents a multi-viewpoint image, and a circle drawn with a dotted line represents an intermediate viewpoint image. In other words, the portion that cannot be covered by the original multi-viewpoint image is complemented by the intermediate viewpoint image drawn with a dotted line, and (C) uses more intermediate viewpoint images than (b). Complementation is possible precisely and smooth blur can be reproduced. Further, the horizontal axis of the graph represents the depth range, and the vertical axis represents the number of intermediate viewpoint images generated. It can be understood that the number of intermediate branch images generated increases as the depth range increases.

  Therefore, in this embodiment, as the depth range D is wider as shown in (b) and (c) of the graph shown in FIG. As shown in the drawing, the intermediate viewpoint image data is generated more deeply as the depth range D is wider. Here, taking FIG. 10B and FIG. 10C as examples, the number of intermediate viewpoint images is not limited to these.

  On the other hand, as shown in FIG. 10A, when the depth range D is as narrow as possible, an intermediate viewpoint image is not necessary. As described above, since the intermediate viewpoint image data has only to be generated by calculating the number of intermediate viewpoint images generated according to the depth range D, the calculation cost can be minimized. In the present embodiment, the case where the relationship between the depth range D and the number of intermediate viewpoint images generated is linear and monotonically increasing, that is, the case where the graph of FIG. Therefore, it is not always necessary to be linear, and may be nonlinear.

<Image composition>
The image composition in step S408 in FIG. 4 will be described in detail. FIG. 11 is a flowchart showing the flow of the composition process. FIG. 12 is a diagram showing an outline of the focus position and the shift amount. First, in step S1101, the multi-viewpoint image data created in step S402 and the intermediate viewpoint image data formed in step S405 are input. In step S1102, based on the focus position 306 set in step S407, the focus position 502 instructed by the user is acquired as shown in FIG. The focus position 502 specified in FIG. 5 corresponds to the focus positions 1211 and 1212 in FIG. In step S1103, based on the focus position acquired in step S1102, shift amounts Δx _i and Δy _i between the focus positions 1211 and 1212 between the multi-viewpoint images and intermediate viewpoint images as shown in FIG. Calculate with the following formula.

Here, b _m and c _m is the horizontal distance between the multi-view image and the intermediate-viewpoint images respectively, the vertical distance, d is the depth value, theta _w horizontal field angle, theta _h is the vertical field angle, w is the number of horizontal pixels , H represents the number of vertical pixels. In step S1104, a weighting factor is calculated based on the blur size control value set in step S407.

In general, the parallax of an image captured by a multi-view camera increases as the distance between the cameras increases. When the images are accurately aligned at a certain focus position, the images are shifted based on the shift amounts Δx _i and Δy _i between the multi-viewpoint image and the intermediate viewpoint image. At this time, they overlap and deviate except at the focus position. As a result, when the shifted images are superimposed, the result is “blurred”. Therefore, it is possible to control the blur depending on how much an image acquired by a camera used as a reference for synthesis (hereinafter referred to as a “composite reference camera”) and an image obtained by a camera that is far apart are synthesized. That is, the blur size can be controlled by combining the blur control values in consideration of the weighting factor corresponding to the distance between the camera that is far from the synthesis reference camera and the synthesis reference camera.

FIG. 13 is an example of the weighting factor w _k corresponding to the control value of the blur size. The weighting factor w _k of the present embodiment has a value according to a Gaussian function centered on the synthesis reference camera. As shown in FIG. 13, the weight coefficient w _k increases as the distance from the synthesis reference camera increases, and the weight coefficient w _k decreases as the distance from the synthesis reference camera increases. Here, when the multi-view image data and the intermediate-viewpoint image data are synthesized to generate synthesized image data, so that the sum of the weighting factor w _k is normalized to 1, the weighting factor w _k of each image all Divide by the sum of the image weighting factors w _k . Finally, in step S1105, the weighting coefficient w according to the control values of the focus position shift amounts Δx _i , Δy _i between the multi-viewpoint images and the intermediate images calculated in step S1103 and the shooting mode and the blur size. _{Using k} , the composite image I (x, y) is calculated by the following equation.

Here, J _i is a multi-view image and intermediate viewpoint image, D is the distance from the central image of the multi-view image and intermediate viewpoint image, and m is the total number of multi-view images and intermediate viewpoint images. In this embodiment, the number of intermediate viewpoint images generated is calculated from the depth map of the entire scene. However, the number of intermediate viewpoint images generated may be calculated for each region in the scene.

  As described above, the present invention calculates the number of intermediate viewpoint images generated according to the depth range of the depth map, and generates intermediate viewpoint image data according to the value. This provides an imaging apparatus that can reduce the calculation cost without generating extra intermediate viewpoint image data.

[Example 2]
In the first embodiment, the method of calculating the depth range from the depth map generated based on the multi-viewpoint image in the image analysis unit 303 and calculating the number of intermediate viewpoint images generated according to the calculated value has been described. In this embodiment, the image analysis unit 303 calculates the number of intermediate viewpoint images generated based on the similarity between multi-viewpoint images.

  The internal configuration of the image processing unit in the present embodiment is the same as that in the first embodiment. FIG. 14 shows the flow of processing of the image pickup apparatus of the present embodiment. Steps S1401, S1402, and S1405 to S1409 are the same processes as steps S401, S402, and S405 to S409 of FIG. 4 described in the first embodiment. That is, steps S1403 and 1404 are different from the first embodiment. In step S1403, the similarity between images is calculated from the multi-viewpoint image generated in step S1402. In step S1404, the number of intermediate viewpoint images generated is calculated based on the similarity calculated in step S1403.

<Calculation of similarity between images>
A method for calculating the similarity between images in this embodiment will be described below. In this embodiment, the similarity between images is calculated using multi-viewpoint images. Hereinafter, a method for calculating the similarity between images will be described with reference to the flowchart shown in FIG.

  First, in step S1501, two images used for processing are selected from the multi-viewpoint images. However, it is assumed that the same subject appears in these two images. In the present exemplary embodiment, an image captured by the camera unit 106 at the center of the image capturing apparatus 101 and an image captured by the camera unit 107 that is adjacent thereto in the horizontal direction will be described as an example. Hereinafter, the former is a reference image and the latter is a target image.

  In step S1502, a difference value K of pixel values of the same pixel is calculated by the following equation for all pixels of the reference image I (x, y) and the target image J (x, y).

  In step S1503, the similarity between the reference image I (x, y) and the target image J (x, y) is calculated. Any method known in this technical field can be used as a method for calculating the similarity. For example, PSNR, correlation coefficient, mean square error, etc. can be applied. In this embodiment, PSNR is set as the similarity. PSNR is calculated by the following equation.

  Here, MAX is the maximum pixel value that the reference image can take, and MSE is the mean square error. M and n are the vertical and horizontal sizes of the reference image.

<Calculation of the number of intermediate viewpoint images generated>
Calculation of the number of intermediate viewpoint images generated in the present embodiment will be described. In this embodiment, the number of intermediate viewpoint images generated is calculated based on the similarity between images.

  FIG. 16 is an image for explaining calculation of the number of intermediate viewpoint images generated. In FIG. 16, a square drawn with a solid line represents a multi-viewpoint image, and a circle drawn with a dotted line represents an intermediate viewpoint image. The horizontal axis of the graph represents similarity (PSNR), and the vertical axis represents the number of intermediate viewpoint images generated. When the degree of similarity is small, there is a large difference between the multi-viewpoint images, so it is necessary to generate intermediate viewpoint images more densely than when the degree of similarity is large. For example, when the PSNR is 20 or 30, as the similarity is lower as shown in FIGS. 16 (a) and 16 (b), more intermediate viewpoint images are required, and as shown in FIGS. 16 (A) and 16 (B), more intermediate viewpoint images are required. The intermediate viewpoint image is processed more densely.

  On the other hand, when the degree of similarity is large, the difference between the multi-viewpoint images is small, so that the intermediate viewpoint position to be created is uniform and does not need to be so dense. For example, when the PSNR is 50, the similarity is very high, so there is no intermediate viewpoint image as shown in FIG. 16C, and only the original multi-view image as shown in FIG. 16C. In the present embodiment, the case where the relationship between the similarity between the multi-viewpoint images and the number of intermediate viewpoint images generated is linear and monotonically increasing, that is, the case where the graph of FIG. Since it only needs to increase, it does not necessarily need to be linear and may be non-linear.

  In this embodiment, the number of intermediate viewpoint images generated is calculated from the similarity of the entire scene. However, the number of intermediate viewpoint images generated may be calculated for each region in the scene.

  As described above, the present invention calculates the number of intermediate viewpoint images generated according to the similarity between the images, and generates an intermediate viewpoint image according to the value. This provides an imaging apparatus that can reduce the calculation cost without generating an extra intermediate viewpoint image.

[Example 3]
In this embodiment, the number of intermediate viewpoint images generated is calculated based on frequency components obtained by frequency analysis of multi-viewpoint image data in the image analysis unit 303. The internal configuration of the image processing unit in the present embodiment is the same as in the first and second embodiments. FIG. 17 shows a processing flow of the imaging apparatus of the present embodiment.

  Steps S1701, S1702, and S1705 to S1709 are the same as steps S401, S402, and S405 to S409 of FIG. 4 described in the first embodiment. That is, steps S1703 and 1704 are different from the first embodiment. In step S1703, the multi-viewpoint image data generated in step S1702 is frequency-analyzed and converted into a spatial frequency domain. In step S1704, the number of intermediate viewpoint images generated is calculated according to the spatial frequency component of the spatial frequency domain converted in step S1703.

<Frequency analysis of images>
Here, the frequency analysis of the image in a present Example is demonstrated. Hereinafter, frequency analysis of image data will be described with reference to the flowchart shown in FIG.

  First, in step S1801, multi-viewpoint image data is sequentially converted to the spatial frequency domain. Here, the spatial frequency region is obtained by decomposing image data into sine waves and rearranging them for each frequency.

  In step S1802, the frequency characteristics of the image data are calculated. When image data is converted to the spatial frequency domain, frequency components are obtained according to the properties of the image. In the case of an image having a small overall change in shading, many low-frequency components are included (curve (a) in FIG. 19), and in the case of an image including a region with a large change, many high-frequency components are included (curve in FIG. b)). Originally, in the case of image data, there are horizontal and vertical components, but here the sum of both frequency components is simply one frequency component, with the horizontal axis representing the frequency component and the vertical axis representing each frequency component. It has energy value.

<Calculation of the number of intermediate viewpoint images generated>
Calculation of the number of intermediate viewpoint images generated in the present embodiment will be described. FIG. 20 is an image for explaining the calculation of the number of intermediate viewpoint images generated. The calculation of the number of intermediate viewpoint images generated in the present embodiment is performed based on the frequency component obtained by frequency analysis of the image. Since an image containing many high-frequency components is a complex scene, intermediate viewpoint image data must be generated densely. On the contrary, when the high frequency component is not included so much, it is a monotonous scene, so that the intermediate viewpoint image may be small. That is, the number of intermediate viewpoint images to be generated can be determined depending on how many high frequency components are included in the frequency components.

  FIG. 20 shows an example of a graph representing the relationship between the high-frequency component and the number of intermediate viewpoint images generated in this embodiment. In FIG. 20, a square drawn with a solid line represents a multi-viewpoint image, and a circle drawn with a dotted line represents an intermediate viewpoint image. The horizontal axis of the graph represents frequency components, and the vertical axis represents the number of intermediate viewpoint images generated. When there is almost no high-frequency component of each multi-viewpoint image data as shown in FIG. 20A, intermediate viewpoint image data need not be generated as shown in FIG. On the other hand, as the multi-viewpoint image data includes more high-frequency components as shown in FIGS. 20B and 20C, it is necessary to generate more intermediate viewpoint image data as shown in FIGS. 20B and 20C. is there.

  In the present embodiment, the case where the relationship between the frequency component and the number of intermediate viewpoint images generated is linear and monotonically increasing, that is, the case where the graph of FIG. However, it is not necessarily linear and may be non-linear. In this embodiment, the number of intermediate viewpoint images generated is calculated based on the high-frequency component of the frequency components of the entire scene. However, the number of intermediate viewpoint images generated may be calculated for each region in the scene.

  As described above, the present invention calculates the number of intermediate viewpoint images generated according to the frequency component of the image data, and generates an intermediate viewpoint image according to the value. This provides an imaging apparatus that can reduce the calculation cost without generating extra intermediate viewpoint image data.

[Other Examples]
In the first to third embodiments, the number of intermediate viewpoint images generated is calculated based on the multi-view images or the feature values between the multi-view images. However, these embodiments may be used in any combination. Moreover, you may process by PC etc.

  The present invention can also be realized by executing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed. The present invention can also be realized by a plurality of processors cooperating to perform processing.

Claims (10)

Multi-viewpoint image input means for inputting multi-viewpoint image data acquired by an imaging device having a plurality of imaging units;
Intermediate viewpoint image generation number calculating means for calculating the number of intermediate viewpoint images that need to be generated in order to obtain a composite image that satisfies a predetermined characteristic according to the feature amount of the multi-viewpoint image;
Intermediate viewpoint image generation means for generating intermediate viewpoint image data of the multi-viewpoint image data, the number of generations calculated by the intermediate viewpoint image generation number calculation means;
An image processing apparatus comprising: image synthesis means for generating synthesized image data using the intermediate viewpoint image data generated by the intermediate viewpoint image generation means.   The image processing apparatus according to claim 1, wherein the feature amount is a depth range indicating a depth of a subject obtained from the multi-viewpoint image.   The image processing apparatus according to claim 2, wherein the number of intermediate viewpoint images generated is greater than the number of intermediate viewpoint images generated at an intermediate viewpoint position where the depth range is narrower.   The image processing apparatus according to claim 1, wherein the feature amount is a similarity between the multi-viewpoint images.   The image processing apparatus according to claim 4, wherein the number of intermediate viewpoint images generated is greater than the number of intermediate viewpoint images generated at an intermediate viewpoint position having a higher similarity.   The image processing apparatus according to claim 1, wherein the feature amount is a frequency component obtained by frequency analysis of the multi-viewpoint image data.   The image processing apparatus according to claim 6, wherein the number of intermediate viewpoint images generated is greater than the number of intermediate viewpoint images generated at an intermediate viewpoint position at which the frequency component does not include a high frequency component.   The image processing apparatus according to claim 1, wherein the predetermined characteristic is an accuracy of blurring of a non-focus surface when performing a refocusing process. A multi-viewpoint image input step of inputting multi-viewpoint image data acquired by an imaging device having a plurality of imaging units;
An intermediate viewpoint image generation number calculating step for calculating the number of intermediate viewpoint images that need to be generated in order to obtain a composite image satisfying a predetermined characteristic according to the feature amount of the multi-viewpoint image;
An intermediate viewpoint image generation step of generating intermediate viewpoint image data of the multi-viewpoint image data, the number of generations calculated in the intermediate viewpoint image generation number calculation step;
An image composition step of generating composite image data using the intermediate viewpoint image data generated by the intermediate viewpoint image generation step.   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 8.

Images