JP5906033B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to a technique for generating image data having a shallow depth of field from captured image data captured from a plurality of viewpoints.

  Conventionally, a method for generating image data with a shallow depth of field by performing image processing on captured image data has been proposed. In Patent Literature 1, image data with a shallow depth of field is generated by applying a filter process that gives different blur to a single image data according to the distance. In Patent Literature 2, image data having a shallow depth of field is generated by performing rendering processing on a three-dimensional model of a subject generated from a plurality of image data having different viewpoints.

JP 2011-10194 Japanese Patent Application No. 2009-212254

However, in Patent Document 1, there is an image quality problem such as unnatural blurring of the subject when the front side is blurred. In Patent Document 2, although the near side is naturally blurred, there is a problem that the calculation cost is high.
Therefore, the present invention reduces the calculation cost while maintaining the image quality when generating output image data indicating an output image from a plurality of captured image data obtained by imaging a subject from a plurality of viewpoints. Objective.

In order to achieve the above object, an image processing apparatus of the present invention uses an image input means for inputting a plurality of captured image data obtained by capturing an image of a subject from a plurality of viewpoints, and the plurality of captured image data. Generating means for generating output image data having a shallower depth of field than each of the plurality of captured image data, and the generating means is present in front of a predetermined distance in the output image data. The output image data is generated using a different number of captured image data in a first image region corresponding to a subject and a second image region corresponding to a subject existing behind the predetermined distance , In the generation of the output image data, the number of captured image data used by the generation unit in the first image region is larger than the number of captured image data used by the generation unit in the second image region. The image processing apparatus characterized by.

  According to the present invention, when generating output image data indicating an output image from a plurality of captured image data obtained by imaging a subject from a plurality of viewpoints, the calculation cost can be reduced while maintaining the image quality. It becomes possible.

1 is a block diagram showing a system configuration of an image processing apparatus in the present embodiment. The figure showing the imaging device in this embodiment. The figure showing the outline functional composition in this embodiment. The flowchart which shows the flow of the process in this embodiment. The flowchart which shows the flow of the depth estimation process in this embodiment. The figure which shows the relationship between the space | interval of the viewpoint position in this embodiment, and the distance to a to-be-photographed object. The flowchart which shows the flow of the area | region division process in this embodiment. The figure which shows the example of the area | region division image in this embodiment. The flowchart which shows the flow of the depth change process in this embodiment. The flowchart which shows the flow of the depth change process using the reference | standard image and depth image in this embodiment. The figure which shows the relationship between the depth value and filter diameter in this embodiment. The flowchart which shows the flow of the depth change process using the multiview image in this embodiment. The figure which shows the example of the weighting coefficient contained in the virtual aperture parameter in this embodiment. 10 is a flowchart illustrating a flow of depth change processing according to the second embodiment. An example of block division in the present embodiment.

[Example 1]
In the present embodiment, an image with a shallow depth of field is generated based on a plurality of image data (captured image data) captured by an image capturing apparatus 107 including a plurality of camera units 201 to 209 as shown in FIG. At this time, image data with a shallow depth of field (composite) is used by using a plurality of image data in the area in front of the distance to be focused (virtual focus distance) and one image data in the area in the back. Image data). In the present embodiment, the camera units 201 to 209 are arranged at regular intervals in a lattice shape, and the vertical axis, the horizontal axis, and the optical axis of each camera unit are all in the same direction. The captured image data is preferably image data captured with pan focus. The composite image data generated in the present embodiment is obtained by reducing the depth of field of the image data captured by the camera unit 205.

  First, a system configuration example of the image processing apparatus according to the present embodiment will be described with reference to FIG. In the figure, a CPU 101 executes a program stored in a ROM 103 and a hard disk drive (HDD) 105 using a RAM 102 as a work memory, and controls each component to be described later via a system bus 112. Thereby, various processes described later are executed. The HDD interface (I / F) 104 is an interface such as serial ATA (SATA) for connecting a secondary storage device such as the HDD 105 or an optical disk drive. The CPU 101 can read data from the HDD 105 and write data to the HDD 105 via the HDD I / F 104. Further, the CPU 101 can expand the data stored in the HDD 105 in the RAM 102 and similarly store the data expanded in the RAM 102 in the HDD 105. The CPU 101 can execute the data expanded in the RAM 102 as a program. The imaging interface (I / F) 106 is a serial bus interface such as USB or IEEE 1394 for connecting the imaging device 107 including a plurality of camera units 201 to 209 as shown in FIG. The CPU 101 can control the imaging device 107 via the imaging I / F 106 to perform imaging. Further, the CPU 101 can read data captured from the image capturing apparatus 107 via the image capturing I / F. An input interface (I / F) 108 is a serial bus interface such as USB or IEEE 1394 for connecting an input device 109 such as a keyboard and a mouse. The CPU 101 can read data from the input device 109 via the input I / F 108. An output interface (I / F) 110 is a video output interface, such as DVI or HDMI, for connecting an output device 111 such as an image display device. The CPU 101 can send data to the output device 111 via the output I / F 110 to execute display.

  Next, an outline functional configuration when performing a series of processes according to the present embodiment will be described with reference to FIG. The captured image data input unit 301 as a functional unit of the CPU 101 acquires the captured image data 307 and the imaging device information 308 from the imaging device 107 or a storage device such as the ROM 103 and the HDD 105. The captured image data 307 is data indicating a plurality of images obtained by capturing an image of a subject from different viewpoint positions (imaging positions) by the imaging device 107. Further, the imaging device information 308 is information including an angle of view and a viewpoint position at the time of shooting by the imaging device 107.

  The parameter input unit 302 as a functional unit of the CPU 101 acquires the virtual focus parameter 309 and the virtual aperture parameter 310 from the input device 109 or a storage device such as the ROM 103 and the HDD 105. The virtual focus parameter 309 is information including the distance to the virtual focus surface. That is, the virtual focus parameter 309 includes information on the focus distance of the output image by a series of processes. The virtual aperture parameter 310 includes information on a filter diameter and a weight coefficient used when blurring an image according to the depth. The distance estimation processing unit 303 as a functional unit of the CPU 101 estimates the depth value of the imaging scene by the stereo matching method based on the captured image data 307 and the imaging device information 308, and generates the depth image data 311. The depth image data 311 includes information related to the distance for each pixel from the imaging device 107 to the subject. The distance information of the depth image data 311 may be set for each region (eg, for each rectangular block). Based on the virtual focus parameter 309 and the depth image data 311, the area division processing unit 304 as a functional unit of the CPU 101, based on the virtual focus parameter 309 and the depth image data 311, captures an imaging scene in front of the virtual focus plane (front blur area) and back area (back blur). Region divided image data 312 is generated. That is, the front blur area indicates an area where the subject exists before the virtual focus surface, and the rear blur area indicates an area where the subject exists behind the virtual focus surface. A depth change processing unit 305 as a function unit of the CPU 101 uses the captured image data 307 based on the imaging device information 308, the virtual focus parameter 309, the virtual aperture parameter 310, and the area division image data 312, and uses the captured image data 307. Is generated. Finally, the image output unit 306 as a functional unit of the CPU 101 outputs the depth change image data 313 (output image data 313) to the output device 111 or stores it in the HDD 105.

  FIG. 4 is a flowchart illustrating an operation procedure of a series of processes in the image processing apparatus according to the present exemplary embodiment. More specifically, after a computer-executable program describing the procedure shown in the flowchart of FIG. 4 is read from the ROM 103 or the HDD 105 onto the RAM 102, the CPU 101 executes the program to execute the process. Hereinafter, each process shown in FIG. 4 will be described.

  First, in step S <b> 401, the captured image data input unit 301 captures captured image data 307 using the imaging device 107, and captures captured image data 307 and imaging device information 308 including the angle of view and viewpoint position of the camera units 201 to 209. get. Alternatively, the captured image data 307 and the imaging device information 308 may be held in a recording device such as the ROM 103 or the HDD 105 in advance, and the captured image data input unit 301 may acquire the captured image data.

  In step S <b> 402, the parameter input unit 302 acquires from the input device 109 a virtual focus parameter 309 including a virtual focus distance and a virtual aperture parameter 310 including a weight coefficient of each image data used at the time of image synthesis. Alternatively, the virtual focus parameter 309 and the virtual aperture parameter 310 may be stored in advance in a recording device such as the ROM 103 or the HDD 105, and the parameter input unit 302 may acquire them.

  In step S403, the distance estimation processing unit 303 performs distance estimation processing using the captured image data 307 and the imaging device information 308 acquired in S401, estimates the depth of the captured scene, and generates depth image data 311 of the scene. To do. The distance estimation process performed in S403 will be described in detail later.

  Next, in step S404, based on the virtual focus parameter 309 acquired in step S402 and the depth image data 311 acquired in step S403, region divided image data 312 is generated by dividing the scene into two regions. Here, the two areas are an area where the subject is in front of the distance to be focused (front blur area), and an area where the subject is in the back of the distance to be focused (rear blur area). is there. The area dividing process performed in S404 will be described in detail later.

  In step S 405, based on the imaging device information 308, the virtual aperture parameter 310, and the segmented image data 312, the depth-changed image data 313 (output image) having a shallower depth of field than the captured image data 307 from the captured image data 307. Data 313) is generated. The image composition processing performed in S405 will be described in detail later.

  In step S <b> 406, the image output unit 306 displays the depth change image data 313 on the output device 111 or records it on a recording device such as the HDD 105.

<Distance estimation processing>
Here, the distance estimation process performed in step S403 will be described. In the distance estimation process, the depth image data 311 is generated by estimating the distance of the captured scene based on the plurality of captured image data 307 having different viewpoint positions. An existing method is used as the distance estimation method. For example, a stereo method and a multi-baseline stereo method can be used. In this embodiment, distance estimation is performed by a stereo method. Hereinafter, the details of the distance estimation process will be described using the flowchart shown in FIG.

  First, in step S501, two images to be used for processing are selected from the captured image data 307. In this embodiment, it is assumed that image data captured by the camera unit 205 at the center of the image capturing apparatus 107 and image data captured by the camera unit 206 that is adjacent to the camera unit 205 in the horizontal direction are selected. Hereinafter, the former is referred to as reference image data indicating a reference image, and the latter is referred to as target image data indicating a target image.

  In step S502, the first pixel is initialized as a target pixel.

  In step S503, it is determined whether distance values are obtained for all pixels. If the distance value is obtained for all the pixels, the process proceeds to step S507, and if there is a pixel for which the distance value is not obtained, the process proceeds to step S504.

  In step S504, first, an area composed of the target pixel of the reference image and surrounding pixels is selected. Then, pattern matching is performed with the target image using the selected region, and a pixel (corresponding pixel) corresponding to the target pixel is obtained from the target image.

  In step S505, a distance value p corresponding to the target pixel is obtained based on the imaging device information 308, the target pixel, and the corresponding pixel obtained in step S504. The distance value p is expressed by the following equation based on α, β, and s shown in FIG.

  Here, α is an angle calculated from the horizontal angle of view of the camera unit 205, the viewpoint position of the reference image data, and the coordinates of the pixel of interest. β is calculated from the horizontal angle of view of the camera unit 206, the viewpoint position of the target image data, and the coordinates of the corresponding pixel. s is the distance between the camera units, and is calculated from the viewpoint position of the reference image data and the target image data.

  Next, in step S506, the target pixel is updated to the next pixel, and the process returns to S503.

  Finally, in step S507, the depth image data 311 is output.

  In this embodiment, in step S501, image data captured by the camera units 205 and 206 is selected and distance estimation is performed. However, the combination of cameras capable of distance estimation is not limited to this. For example, image data captured by another camera unit may be selected as target image data. In this embodiment, distance information is acquired for each pixel. However, distance information may be acquired for each area such as a rectangular block.

<Area division processing>
Here, the region division processing performed in step S404 will be described. In the area division processing, based on the virtual focus parameter 309 and the depth image data 311, an area where the subject is in front of the distance to be focused (front blur area) and an area behind the area (back blur area). ) To divide the scene into region-divided image data 312. Hereinafter, the details of the region division processing will be described with reference to the flowchart shown in FIG.

  First, in step S701, the area of the front blurred area is specified. It is assumed that the virtual focus distance obtained from the virtual focus parameter 309 is dp, and the pixel value of the depth image data 311 is D (i, j). At this time, a pixel (i, j) satisfying D (i, j) ≦ dp and a set of pixels included within the surrounding r1 pixels are defined as a front blur region. Here, r1 is a predetermined maximum radius of blur.

  Next, in step S702, the area of the back blur area is specified. A set of pixels other than the front blur area in the depth image data 311 is set as a rear blur area.

  Finally, in step S703, image data in which different label values are assigned to the front blurred area and the rear blurred area are output as area divided image data 312.

  An example of the area division image data 312 obtained by the above processing is shown in FIG. It is assumed that the depth image data 311 is image data composed of distance values d0, d1, and d2, and the value of the virtual focus distance dp obtained from the virtual focus parameter 309 is d2. In this case, a set of pixels in the depth image data 311 whose distance value is d2 and pixels included within the surrounding r pixels is a front blurred area, and a set of other pixels is a back blurred area.

  In this embodiment, in step S701, the distance r for determining the front blur area is a predetermined value, but the method for setting the distance value r is not limited to this. For example, the maximum radius of blur may be calculated from the virtual focus parameter, and this value may be used as the distance r. By doing so, it is possible to set an optimal front blur area according to the virtual focus parameter.

<Depth change processing>
Here, the depth changing process performed in step S405 will be described. In the depth changing process, different processes are applied to the front blurred area and the rear blurred area while referring to the area-divided image data 312, and the depth changed image data 313 (output image data 313) having a shallow depth is generated. At this time, the pixel value of the depth-changed image data 313 is calculated by combining a plurality of pieces of image data having different viewpoint positions from the captured image data 307 in the front blur region. Further, in the back blur area, the pixel value of the depth change image data 313 is calculated by using the reference image data captured by the camera unit 205 and the depth image data 311. Hereinafter, the depth change process will be described in detail with reference to the flowchart shown in FIG.

  First, in step S901, the area division image data 312 is divided into a plurality of blocks. Steps S905 and S906 described later are performed for each block determined here. An example of block division is shown in FIG.

  In step S902, the block of interest is initialized to the first block.

  Next, in step S903, it is determined whether or not processing in step S905 or step S906 described later has been performed on all blocks. If processing has been performed for all blocks, the process proceeds to step S908, and if not, the process proceeds to S904.

  Next, in step S904, it is determined whether or not a front blur area is included in the block of interest. If not included, the process proceeds to step S905. If included, the process proceeds to step S906.

  In step S <b> 905, a process for reducing the depth is performed using one reference image data and depth image data 311, and a pixel value of a region corresponding to the target block of the depth-changed image data 313 is calculated. Details will be described later.

  In step S <b> 906, a process for reducing the depth is performed using the plurality of captured image data 307, and the pixel value of the region corresponding to the target block of the depth-changed image data 313 is calculated. Details will be described later.

  In step S907, the target block is updated to the next unprocessed block, and the process returns to step S903.

  Finally, in step S908, the depth change image data 313 is output.

  By performing the processing described above, when generating image data with a shallower depth of field from a plurality of image data, an effect of reducing the calculation cost while maintaining the blurred image quality is obtained.

<< Depth change processing using reference image data and depth image data >>
Depth change processing performed on a block that does not include the previous blurred region performed in step S905 will be described. Here, the pixel value corresponding to the target block of the depth-changed image data 313 is calculated by smoothing the reference image data while referring to the depth image data 311 and taking into account the context of the subject. Hereinafter, the pixel value corresponding to the pixel (i, j) of the depth change image data 313 is represented as H (i, j). Hereinafter, details of the processing will be described using the flowchart shown in FIG.

  First, in step S1001, the target pixel (i, j) is initialized to the first pixel.

  Next, in step S1002, it is determined whether or not processing has been performed for all pixels in the block of interest. If all the pixels have been processed, step S905 is ended, and if not, the process proceeds to step S1003.

  In step S1003, variables val and sum used in the subsequent processes are initialized to 0. Here, val is a variable for calculating the output pixel value, and sum is a variable for calculating the sum of coefficients described below.

  In step S1004, the neighboring pixel (i ′, j ′) is initialized. The neighboring pixel (i ′, j ′) is a pixel included in an area within r2 pixels from the target pixel (i, j) (hereinafter referred to as a neighboring area). Here, r2 is a predetermined constant.

  Next, in step S1005, it is determined whether all the pixels in the neighborhood area have been referred to as neighborhood pixels. If there are unreferenced pixels, the process proceeds to step S1006. If all the pixels are referenced, the process proceeds to step S1011.

  In step S1006, the filter diameters of the pixel of interest (i, j) and the neighboring pixels (i ', j') are calculated. The filter diameter is a value included in the virtual aperture parameter, and has a different value depending on the depth value. The relationship between the depth value and the filter diameter is shown in FIG. The filter diameter Rf is 0 when the virtual focus distance dp included in the virtual focus parameter 309 and the depth value of each pixel obtained from the depth image data 311 are equal, and increases as the depth value moves back and forth from the virtual focus distance. It becomes. When the depth value corresponding to the pixel (x, y) is D (x, y), the filter diameter Rf1 of the target pixel and the filter diameter Rf2 of the neighboring pixel are obtained by the following equations.

  In step S1007, it is determined whether the neighboring pixel (i ′, j ′) is a processing target pixel. If it is a processing target pixel, the process proceeds to step S1008, and if it is not a processing target pixel, the process proceeds to step S1010. For the determination, the following two determination formulas using the depth values of the target pixel and neighboring pixels and the filter diameter rf of the target pixel are used.

  Here, Expression (4) determines whether the neighboring pixel is on the near side or the far side with respect to the target pixel. When the neighboring pixel is on the near side of the target pixel, “true” is indicated, and when the neighboring pixel is on the rear side, “false” is indicated. Expression (5) determines whether or not the distance between the target pixel and neighboring pixels is larger than the filter diameter of the target pixel. If the distance between the pixel of interest and the neighboring pixel is smaller than the filter diameter of the pixel of interest, it is “true”, and if it is larger, it is “false”. The presence or absence of occlusion is determined by these equations. When the formula (4) or the formula (5) is true, the neighboring pixel (i ′, j ′) is regarded as a processing target pixel, and the process proceeds to step S1008. When both the formula (4) and the formula (5) are false, the neighboring pixel (i ′, j ′) is excluded from the processing target pixels, and the process proceeds to step S1010. Note that the method of selecting the target pixel is not limited to this. For example, only formula (4) may be used as a judgment formula, or another judgment formula for judging the presence or absence of occlusion may be used.

  In step S1008, the weighting coefficient c of the neighboring pixel that is the processing target pixel is calculated. Here, the coefficient c is calculated according to the following equation using the distance from the target pixel to the neighboring pixels.

  In the above equation, Gaussian is used to smooth the blur, but the method of determining the coefficients is not limited to this. For example, the filter coefficient may be determined based on a polynomial function.

  In step S1009, the values of variables val and sum are updated as follows.

  Here, I (x ′, y ′) is a pixel value of a neighboring pixel.

  In step S1010, the neighboring pixel (i ', j') is updated, and the process returns to step S1005.

  In step S1011, the pixel value H (i, j) of the target pixel (i, j) in the depth change image data 313 is calculated according to the following equation.

Here, the variation in the pixel value due to the presence or absence of the processing target pixel is corrected by normalizing the weighted sum of the neighboring pixels with the total value of the coefficients.
Next, in step S1012, the neighboring pixel (i, j) is updated, and the process returns to step S1012.

  Through the above processing, an area corresponding to the target block of the depth change image data 313 is generated.

  In the present embodiment, in step S1004, the distance value R when determining the neighborhood region is a predetermined value, but the method of setting the distance value R is not limited to this. For example, the maximum value of the filter diameter may be R. By doing this, it is possible to set a neighborhood area of an optimal size according to the virtual focus parameter.

<< Depth change processing using multi-viewpoint image data >>
Depth change processing performed on the block including the previous blurred region performed in step S906 will be described. Here, an example will be described in which image data having a shallower depth is generated from a plurality of captured image data 307 using the synthetic aperture method. By this processing, an area corresponding to the target block of the depth change image data 313 is generated. The details of the processing will be described below using the flowchart shown in FIG.

  First, in step S1201, a weighting factor w used when combining image data is set using the weighting factor included in the virtual aperture parameter 310 acquired in step S402. The weighting coefficient included in the virtual aperture parameter 310 is a set of coefficients for a plurality of captured image data 307. As shown in FIG. 13, the viewpoint position of the captured image data 307 is represented as Pm, and the captured image data 307 corresponding to Pm is defined as image data Im. At this time, the value of the weighting factor corresponding to Im is set to W (m). Here, W (m) is set to a normalized value according to a Gaussian function centered on P4 so that the sum is 1. By setting W (m) so as to follow a Gaussian function, the image data can be smoothly blurred during synthesis.

  In step S1202, the shift amount of each image data is calculated based on the imaging device information 308. The horizontal shift amount Δi (m, d) and the vertical shift amount Δj (m, d) of the image data Im when the focus distance is d are expressed by the following equations.

  Here, W and H are the horizontal and vertical image sizes of the image data, θw is the horizontal viewing angle of the camera unit, and θh is the vertical viewing angle of the camera unit. Further, (sm, tm) is the coordinate of Pm on the xy plane, and (s ′, t ′) is the coordinate of the viewpoint position P4 of the camera unit 205.

  Next, in step S1103, weighted addition processing is performed on the captured image data 307 using the shift amount obtained in step S1102 and the weighting coefficient obtained in step S1101, and the pixel value of the depth-changed image data 313 in the block of interest. H (i, j) is calculated. Pixel value H (i, j) is calculated by the following equation.

  Through the above processing, an area corresponding to the target block of the depth change image data 313 is generated.

  Although a method of combining captured image data using the synthetic aperture imaging method has been described here, other methods may be used. For example, the depth-change image data may be generated by generating the interpolation image data so as to incorporate the viewpoint position from the captured image data and combining the captured image data and the interpolation image data by the synthetic aperture imaging method. Alternatively, the depth-change image data may be generated by generating a three-dimensional model of the scene from the captured image data and performing rendering using the model. When these methods are used, depth-changed image data with better blurred image quality can be obtained.

<< Comparison of depth change processing >>
In general, the depth change processing for the front blur region is processing based on a plurality of captured image data, and thus the calculation cost is high. If the depth changing process for the front blur area is performed based on the reference image data and the depth image data, the image quality of the blur area is extremely bad. This is because a part of the rear subject is hidden behind the front subject that becomes the front blur. Although the hidden part of the subject behind the camera should be visible due to the front blur, the image quality is significantly reduced because there is no information. By using a plurality of captured image data, information on a hidden portion can be obtained from captured image data other than the reference image data. As a result, it is possible to prevent the deterioration of the image quality of the previous blurred area.

  On the other hand, the depth change process for the back blur area is a process based on the reference image data and the depth image data, and thus the calculation cost is low. Even if the depth change process for the back blur area is performed based on a plurality of captured image data, the improvement in the image quality of the blur area is not high. This is because a part of the rear subject is hidden behind the front subject, but this hidden portion hardly affects the image quality when blurring the rear subject.

  In this embodiment, the depth changing process is performed on the front blurred area based on the plurality of captured image data, while the depth changing process is performed on the rear blurred area based on the reference image data and the depth image data. Thereby, it is possible to reduce the calculation cost required for the depth changing process while maintaining the image quality of the blurred area.

<Modification>
In this embodiment, the depth changing process is performed on the back blur area based on the reference image data and the depth image data. However, the depth changing process for the rear blur area is not limited to this, and any depth changing process can be used as long as the calculation cost is lower than the calculation cost required for the depth changing process for the front blur area. For example, the depth changing process for the rear blurred area may be a process based on N pieces of captured image data, and the depth changing process for the front blurred area may be a process based on M (M> N) captured image data.

[Example 2]
In the first embodiment, the scene is divided into two areas, a front blur area and a rear blur area. In the second embodiment, by setting an intermediate region at these boundaries, the boundary between the front blur region and the rear blur region is smoothly connected.

  The second embodiment is different from the first embodiment in the depth change process performed in step S405. Details of the depth change process will be described below.

  In the first embodiment, different processing is applied to each block to reduce the depth, but in the second embodiment, the block boundaries are smoothly connected by blending the processing results at the block boundary portions. Details will be described below with reference to the flowchart shown in FIG.

  First, in step S1401, the image data is divided into a plurality of regions based on the region divided image data 312. First, after dividing into a plurality of blocks in the same manner as in step S902, the blocks are classified into blocks including the front blur region and regions not including the front blur region, and the boundary region is set as an intermediate region. An example of the division result is shown in FIG. Here, a set of pixels included within the distance r from the boundary between the block including the front blur area and the block not including the front blur area is used as the intermediate area.

  Next, in step S1402, a region of interest for subsequent processing is initialized.

  In step S1403, it is determined whether processing has been performed for all regions. If processing has been performed for all areas, the process proceeds to step S1409, and if there is an unprocessed area, the process proceeds to step S1404.

  In step S1404, the area is determined. When the attention area does not include the front blur area, the process proceeds to step S1405. When the attention area includes the front blur area, the process proceeds to step S1406. When the attention area includes the front blur area, the process proceeds to step S1407.

  In step S1405, the depth changing process is performed using the reference image data and the depth image data as in step S905 of the first embodiment.

  In step S1406, similarly to step S906 of the first embodiment, depth change processing is performed using a plurality of captured image data 307.

  In step S1407, two types of depth change processing are performed by the method of step S905 of the first embodiment and the method of step S906 of the first embodiment, and the results are blended. Assuming that the image data H1 obtained by the method of step S905 and the image data H2 obtained by the method of step S906 are used, the depth change image data H is calculated based on the following equation.

  Here, α is a coefficient of 0 or more and 1 or less determined by the distance r that determines the range of the intermediate region and the distance from the region that does not include the previous blurred region to the pixel (i, j). The value of α is closer to 1 as the pixel (i, j) is closer to the region not including the front blur region, and is closer to 0 as the pixel (i, j) is closer to the region including the front blur region.

  In step S1408, the attention area is updated, and the process returns to step S1403.

  Finally, in step S1309, the depth change image data 313 is output.

  By performing the processing described above, when generating an image with a shallower depth of field from a plurality of images, the boundary between the front blur region and the rear blur region is smoothly connected, and the image quality of blur is maintained. There is an effect of reducing the calculation cost.

<Other variations>
In the first and second embodiments, the depth image data 311 is generated by the distance estimation process in step S403. However, the generation method of the depth image data 311 is not limited to this. For example, the depth of the scene may be measured using an external sensor, and depth image data may be generated based on the measurement. Alternatively, a distance measuring sensor may be newly provided in the imaging apparatus 107, and depth image data may be generated based on the distance measuring sensor.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

Image input means for inputting a plurality of captured image data obtained by imaging a subject from a plurality of viewpoints;
Generation means for generating output image data having a shallower depth of field than each of the plurality of captured image data using the plurality of captured image data ;
In the output image data, the generation means includes a first image region corresponding to a subject existing before a predetermined distance, and a second image region corresponding to a subject existing behind the predetermined distance. And generating the output image data using a different number of captured image data ,
In the generation of the output image data, the number of captured image data used by the generation unit in the first image region is larger than the number of captured image data used by the generation unit in the second image region. An image processing apparatus. It further has parameter input means for inputting a focus parameter indicating the focus distance,
In the output image data, the generation means includes a first image area corresponding to a subject existing in front of the focus distance, and a second image area corresponding to a subject existing in the back of the focus distance. The image processing apparatus according to claim 1, wherein image data that is in focus at a focus distance indicated by the focus parameter is generated as the output image data using different numbers of captured image data. Obtaining means for obtaining distance information indicating a distance to the subject;
A specifying unit for specifying the first image area and the second image area based on the distance information and the focus parameter;
Said generating means, on the basis of the said and identified the first image area the second image area by the specific means, the image processing according to claim 2, characterized in that to generate the output image data apparatus.   The image processing apparatus according to claim 3, wherein the acquisition unit acquires the distance information based on the plurality of captured image data.   The generation means uses the output image data in the first image region using a larger number of captured image data than the number of captured image data used for generating the output image data in the second image region. 5. The image processing apparatus according to claim 1, wherein the image processing apparatus generates an image.   The generating means generates the output image data using at least two captured image data in the first image area and further using one captured image data in the second image area. The image processing apparatus according to claim 5.   The said generation | production means produces | generates the said output image data by performing the filter process which gives blurring to captured image data in said 2nd image area | region, The any one of Claim 1 thru | or 6 characterized by the above-mentioned. The image processing apparatus described.   8. The output unit according to claim 1, wherein the generation unit generates the output image data by shifting a plurality of captured image data and performing weighted addition in the first image region. The image processing apparatus according to item. Means for inputting an aperture parameter related to a blur amount of the output image data;
The image processing apparatus according to claim 1, wherein the generation unit generates the output image data based on the aperture parameter.   10. An imaging apparatus having a function as the image processing apparatus according to claim 1, and having a plurality of imaging units that capture the plurality of captured image data. An image input step of inputting a plurality of captured image data obtained by imaging a subject from a plurality of viewpoints;
Generating output image data having a shallower depth of field than each of the plurality of captured image data using the plurality of captured image data; and
In the generation step, in the output image data, a first image region corresponding to a subject existing before a predetermined distance and a second image region corresponding to a subject existing behind the predetermined distance The output image data is generated using a different number of captured image data, and in the generation of the output image data, the number of captured image data used in the first image region in the generation step is the generation step. Is larger than the number of captured image data used in the second image region.   The program for functioning a computer as an image processing apparatus of any one of Claims 1 thru | or 9.

Images