JP6750867B2 - Image processing apparatus, control method thereof, imaging apparatus, and program - Google Patents

Description

The present invention relates to an image processing technique for acquiring depth information of a subject and shaping the depth information.

In the shaping process of the distance map that represents the depth information of the subject, the subject contour in the distance map is adjusted to the correct contour using the captured image. Patent Document 1 discloses a method of performing geodesic joint upsampling on a low resolution depth image for the purpose of generating a high resolution depth image. By calculating the interpolated pixel position based on the geodesic distance in the high-resolution optical image, it is possible to generate a high-resolution depth image in which the contour of the subject boundary is correct. Patent Document 2 discloses a method of correcting a defocus attribute so as to correspond to a subject by referring to the feature of the subject in basic image data obtained by adding the left-eye image and the right-eye image for distance measurement. .. Further, effects such as blurring processing and edge enhancement according to the distance (hereinafter, referred to as image processing effect) are applied to the captured image by referring to the distance map after shaping.

JP, 2014-150521, A JP, 2008-15754, A JP, 2014-44408, A

However, the conventional technique disclosed in Patent Document 2 does not consider generating a distance distribution corresponding to a captured image different from the image referred to when the information related to the distance distribution of the subject is acquired.
In view of the above problem, the present invention has an object of providing an image processing apparatus that improves the accuracy of shaping processing for depth information when performing image processing by acquiring depth information related to a plurality of image data and image data. ..

An image processing apparatus according to an embodiment of the present invention includes a first acquisition unit that acquires a pair of first image data obtained from a light ray group that passes through a first pupil partial region of an imaging optical system; Second acquisition means for acquiring second image data obtained from a light ray group passing through the second pupil partial area of the imaging optical system having a size different from that of the first pupil partial area; Generating means for generating depth information corresponding to the depthwise depth of the subject in the image from the first image data, and F when the second image data or the second image data is acquired. A shaping unit that performs shaping processing for smoothing the depth information by using data of an image captured with a difference from the value within a predetermined range.

According to the image processing device of the present invention, when a plurality of image data and depth information related to the image data are acquired and image processing is performed, the accuracy of the shaping process for the depth information can be improved.

4 is a block diagram showing a functional configuration of a digital camera according to Embodiments 1 to 3. FIG. FIG. 3 is a diagram illustrating an image pickup unit of the digital camera according to the first embodiment. FIG. 6 is a diagram illustrating the relationship between the number of additions of divided pixels and the depth of focus according to the first embodiment. 3 is a block diagram illustrating a functional configuration of an image processing unit according to the first embodiment. FIG. 6 is a flowchart illustrating a process of the first embodiment. FIG. 3 is a diagram illustrating a viewpoint image according to the first embodiment. FIG. 7 is an explanatory diagram of a target image to which the image processing effect according to the first embodiment is applied. FIG. 6 is an explanatory diagram of a distance map shaping process according to the first embodiment. It is explanatory drawing of the shaping process of the distance map which concerns on a comparative example. FIG. 7 is an explanatory diagram of an image pickup unit of the digital camera according to the second embodiment. 9 is a block diagram showing a functional configuration of an image processing unit according to Examples 2 and 3. FIG. 9 is a flowchart illustrating a process of the second embodiment. 9 is a flowchart illustrating a process of the third embodiment. FIG. 9 is a block diagram showing a functional configuration of a digital camera according to a fourth embodiment. FIG. 9 is a block diagram showing a functional configuration of an image processing unit according to a fourth embodiment. 9 is a flowchart illustrating a process of the fourth embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. As an image processing apparatus of the embodiment, an application example to an image pickup apparatus such as a digital camera will be described. For example, in the configuration of a light field camera capable of acquiring light intensity information and direction information, a distance map is acquired from a viewpoint image by the image capturing unit. Further, a process of shaping the distance map is performed so that it follows the contour of the subject image in the captured image acquired in a state where the light ray group forming the image passes through the pupil region having a different size from that at the time of acquiring the distance map. ..

[Example 1]
FIG. 1 is a block diagram showing the functional arrangement of the digital camera 100 according to this embodiment. The system control unit 101 includes a CPU (central processing unit). The CPU reads an operation program of each component included in the digital camera 100 from a ROM (Read Only Memory) 102, expands it in a RAM (Random Access Memory) 103, and executes it. Thereby, the system control unit 101 controls the entire digital camera 100. The ROM 102 is a rewritable non-volatile memory, and stores parameters necessary for the operation of each unit, in addition to the operation programs of the respective units included in the digital camera 100. For example, data of the exit pupil distance is stored in the ROM 102 as lens information necessary for focus detection and the like. The RAM 103 is a rewritable volatile memory and is used as a temporary storage area for data output in the operation of each component included in the digital camera 100.

The image pickup optical system 104 forms light from a subject on the image pickup unit 105. The image capturing unit 105 includes an image capturing element such as a CCD (charge coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor. The image pickup device photoelectrically converts the optical image formed by the image pickup optical system 104, and outputs an analog image signal to the A (analog)/D (digital) conversion unit 106. The A/D conversion unit 106 performs A/D conversion on the input analog image signal and outputs digital image data to the RAM 103 for storage.

The image processing unit 107 performs various kinds of image processing such as white balance adjustment, color interpolation, reduction/enlargement processing on the image data stored in the RAM 103, further distance map generation, and distance map shaping processing. I do.

The recording medium 108 is a removable memory card or the like, and image data and various data are recorded therein. For example, image data processed by the image processing unit 107 after being read from the RAM 103, image data subjected to A/D conversion by the A/D conversion unit 106, and the like are recorded as recording image data on the recording medium 108. It The bus 109 is used to send and receive signals between each component.

The imaging unit 105 in FIG. 1 will be described with reference to FIG. FIG. 2A illustrates a pixel array of the imaging unit 105. This is the pixel array of the image sensor that acquires the light field information by the plenoptic method. FIG. 2B is a schematic diagram showing the pixel portion 200 in an enlarged manner. The direction perpendicular to the paper surface of FIG. 2 (Z direction) is defined as the optical axis direction, the horizontal direction orthogonal to the Z direction is defined as the X direction, and the vertical direction orthogonal to the Z direction is defined as the Y direction. In the plenoptic method, as shown in FIG. 2A, microlenses arranged at a ratio of one to a plurality of photoelectric conversion units (hereinafter, each unit is also referred to as divided pixels) are two-dimensionally arranged on the front surface of the image sensor. Are arranged in an array. Therefore, it is possible to acquire not only the two-dimensional intensity distribution of light but also the information of the incident direction of the light ray that is incident on the image sensor, and the three-dimensional information of the object space.

FIG. 2B shows a structural example in which one microlens 210 has divided pixels in 6 rows and 6 columns. In this example, the position coordinates in the row direction (Y direction) are represented by the symbols A to F, and the position coordinates in the column direction (X direction) are represented by the numbers 1 to 6. For example, the position of the upper left corner is expressed as "1A", and the position of the lower right corner is expressed as "6F".

A two-dimensional image composed only of divided pixels at the same pixel position (first position) of each microlens 210 becomes a two-dimensional image composed only of divided pixels at another same pixel position (second position). It has parallax. For example, there is a parallax between the first image composed only of the divided pixels corresponding to the position 1A and the second image composed only of the divided pixels corresponding to the position 2A. A total of 36 (viewpoints) different images (viewpoint images) can be obtained from the divided pixels of 6 rows and 6 columns.

Next, the relationship between the addition of divided pixels and the depth of focus will be described with reference to FIG. 3A, 3B, and 3C show the pixel portion 200 in FIG. It is assumed that an addition image (composite image) is generated by performing addition processing on the divided pixels surrounded by a thick line in the figure. FIG. 3A shows an example in which an addition image is generated by adding all the divided pixels existing under each microlens. FIG. 3B shows an example in which a divided image is generated by adding divided pixels for each of four areas. Each area is as follows.
First area (1A, 2A, 3A, 1B, 2B, 3B, 1C, 2C, 3C)
-Second area (4A, 5A, 6A, 4B, 5B, 6B, 4C, 5C, 6C)
・Third area (1D, 2D, 3D, 1E, 2E, 3E, 1F, 2F, 3F)
・Fourth area (4D, 5D, 6D, 4E, 5E, 6E, 4F, 5F, 6F)
With the first to fourth regions as a unit, a process of adding divided pixels in each region to generate an added image is performed.

FIG. 3C shows a case where addition of divided images is not performed. That is, an individual image composed of only 36 divided images is generated.
Next, the depth of focus will be described with reference to FIGS. 3D, 3E, and 3F. 3(D), 3(E), and 3(F) are diagrams corresponding to FIG. 3(A), FIG. 3(B), and FIG. 3(C), respectively.

In FIG. 3D, the addition processing of all the divided pixels corresponding to one microlens is performed. In this case, the light ray group passes through the pupil area 300 of the imaging optical system which is not divided. When the permissible circle of confusion diameter is expressed as δ and the aperture value of the imaging optical system is expressed as F, the depth of focus is ±Fδ. Since all the signals of the divided pixels are added, it is not possible to obtain a pair of viewpoint images having parallax with each other, but a bright image having a high S/N (signal to noise) ratio can be obtained. Hereinafter, an image obtained by adding the signals of all the divided pixels will be referred to as a full aperture image. Here, the permissible circle of confusion diameter δ is defined by δ=2·ΔX (ΔX: pixel period), that is, the reciprocal of the Nyquist frequency “1/(2·ΔX)”.

In FIG. 3E, the light ray group passes through the divided pupil region 301 which is divided into 2×2 and becomes narrow. Therefore, the effective aperture value is 2F, which is darker than that at the full aperture. The effective depth of focus of each viewpoint image is ±2Fδ, and it is possible to obtain a viewpoint image that is twice as deep as the depth of field compared to the full aperture image. A total of four types of viewpoint images are obtained by the 2×2 division. Note that FIG. 3E illustrates only a pupil partial region divided in the vertical direction for convenience.

In FIG. 3F, the light ray group passes through the divided pupil region 302 which is divided into 6×6 and is further narrowed. Therefore, the effective aperture value is 6F, and the effective depth of focus of each viewpoint image is ±6Fδ. A total of 36 types of viewpoint images are obtained by the 6×6 division. Note that FIG. 3F shows only the pupil partial regions divided in the vertical direction for convenience.

By performing the addition processing of the divided pixels in this way, it is possible to generate an image including a group of light rays that have passed through pupil partial regions of different sizes. An image with a shallow depth of field can be obtained as the divided pupil partial region is large, and an image with a deep depth of field can be obtained as the pupil partial region is small.

FIG. 4 is a block diagram showing a specific configuration example of the image processing unit 107 in FIG. The pixel addition unit 400 performs the pixel addition processing described in FIGS. 3A and 3B. The distance map generation unit 401 acquires a pair of viewpoint images and generates information on a distance map related to the distance distribution of the subject. The distance map shaping unit 402 performs shaping processing of the distance map generated by the distance map generation unit 401. The distance map information that has undergone the shaping process is recorded in the recording medium 108 under the recording control performed by the system control unit 101 in association with the acquired image data of the target image. The edge enhancement unit 403 refers to the shaped distance map and performs edge enhancement processing on the target image.

The processing executed by each unit will be described in detail with reference to FIG. FIG. 5 is a flowchart for explaining the processing content of the image processing unit 107. The following processing is realized by the CPU of the system control unit 101 reading the program from the memory and executing the program.

In step S500, the user shoots a subject with the digital camera 100, and information about divided pixels is acquired. Alternatively, when the information of the divided pixels is recorded in the recording medium 108 in advance, the process of reading the information of the divided pixels from the recording medium 108 is performed.

At S501, a pair of viewpoint images for generating a distance map is generated. The pixel addition unit 400 performs addition processing on the information of the divided pixels acquired in S500, and generates a pair of viewpoint images. The addition of divided pixels is as described in FIG. In the present embodiment, in FIG. 3B, the first added image obtained by adding the divided pixels in the first region (1A, 2A, 3A, 1B, 2B, 3B, 1C, 2C, 3C) is acquired. .. Further, the second added image obtained by adding the divided pixels in the second regions (4A, 5A, 6A, 4B, 5B, 6B, 4C, 5C, 6C) is acquired. An example of a pair of viewpoint images obtained from the first and second added images is shown in FIG.

The image in FIG. 6A corresponds to the first added image and shows a scene in which a person 600 is in focus and a person 601 is on the background side of the person 600. Blurring occurs in the image of the person 601 on the background side, and the outline is unclear. The image in FIG. 6B corresponds to the second added image, and has an parallax with respect to the image in FIG. 6A. That is, the image of the person 602 in FIG. 6B has a parallax with respect to the image of the person 601 in FIG. 6A.

In step S502 of FIG. 5, the distance map generation unit 401 generates a distance map (depth information) using the pair of viewpoint images acquired in step S501. Regarding the depth information, there are various embodiments as information corresponding to the depth (depth) of each subject in the captured image in the depth direction. That is, the information (depth information) indicated by the data corresponding to the depth of the subject directly represents the subject distance from the imaging device to the subject in the image, or the distance of the subject in the image (subject distance) or Any information can be used as long as it represents the relative depth relationship. For example, the image shift amount of each area is calculated by the correlation calculation between the pair of image data acquired from the image capturing unit 105, and the image shift map indicating the distribution of the image shift amount is calculated. Alternatively, the image shift amount is further converted into a defocus amount, and a defocus map representing the distribution of the defocus amount (distribution on the two-dimensional plane of the captured image) is generated. If this defocus amount is converted into a subject distance based on the conditions of the image pickup optical system and the image pickup device, distance map data representing the distribution of the subject distance can be obtained. It is possible to acquire image shift map data, defocus map data, or distance map data of a subject distance converted from the defocus amount. In this embodiment, a processing example of calculating a defocus amount at a pixel position of interest as information related to the distance distribution of a subject will be described. The method of calculating the defocus amount from the pair of viewpoint images is as follows.

Of the viewpoint images forming a pair, the signal sequences in the first viewpoint image are denoted by E(1) to E(m), and the signal sequences in the second viewpoint image are denoted by F(1) to F(m). write. m represents a natural number variable corresponding to the pixel position. While shifting the signal trains F(1) to F(m) relative to the signal trains E(1) to E(m), the shift amount (k) between the two signal trains is calculated by the following equation (1). The calculation of the correlation amount (denoted as C(k)) is performed.
C(k)=Σ|E(n)-F(n+k)| (1)

The Σ operation in equation (1) is calculated as the sum total for the variable n. In the Σ operation, the range of n and n+k is limited to the range of 1 to m. Further, the shift amount k is an integer value and is a relative shift amount in units of the detection pitch of a pair of data. Shift subtraction between the two signal sequences is performed, and the sum of absolute difference values is calculated as the correlation amount C(k) of the shift amount k.

Regarding the calculation result of the equation (1), the correlation amount C(k) becomes the minimum in the shift amount in which the pair of signal sequences has a high correlation. The value of k that minimizes the value of C(k) is expressed as kj. The shift amount x that gives the minimum value C(x) for the continuous correlation amount is calculated using the three-point interpolation method shown in the following formulas (2) to (4).
x=kj+D/SLOP (2)
D={C(kj-1)-C(kj+1)}/2 (3)
SLOP=MAX{C(kj+1)-C(kj), C(kj-1)-C(kj)}... (4)

The shift amount x obtained by the equation (2) is a real value, and the defocus amount (denoted as DEF) can be calculated from the equation (5) using this.
DEF=KX/PY/x (5)
In Expression (5), PY represents the detected pitch. KX is a conversion coefficient determined by the size of the opening angle of the center of gravity of the light flux passing through the pair of pupil partial regions. The distance map generation unit 401 outputs distance map data representing the spatial distribution of the calculated defocus amount.

In step S503, the image processing unit 107 generates a target image to which the image processing effect is applied, from the divided pixel information acquired in step S500. In the present embodiment, in FIG. 3C, an image configured only by information of divided pixels corresponding to the position 2B is set as the target image. FIG. 7 shows an example of the target image. Since the target image is an image generated from a group of rays that have passed through a small pupil partial region as compared with the case where the pair of viewpoint images was generated in S501, it is an image with a large effective F value and a deep depth of field. is there. Therefore, the position of the person 700 on the background side is within the depth of field, and the contour of the image of the person 700 is clear.

In step S504, the distance map shaping unit 402 shapes the distance map generated in step S502. In the shaping process, the bilateral filter process is performed on the distance map while referring to the shaping image. Regarding the bilateral filter processing, specifically, when the filter result at the pixel position of interest p is denoted by Jp, it is expressed by the following equation (6).
Jp=(1/Kp)ΣI1q·f(|p−q|)·g(|I2p−I2q|) (6)

The meaning of each symbol in formula (6) is as follows.
q: Peripheral pixel position Ω: Integration target area centered on the target pixel position p Σ: Integration of qεΩ range I1q: Distance map pixel value at the peripheral pixel position q f(|p−q|): Target pixel position p Gaussian function centered at I2p: Shaping image signal value at target pixel position p I2q: Shaping image signal value at peripheral pixel position q g(|I2p-I2q|): Shaping image signal value I2p Gaussian function Kp: Normalized coefficient, which is an integrated value of f·g weights.

When the difference between the signal value I2p at the target pixel position p and the signal value I2q at the peripheral pixel position q is small, that is, when the pixel value of the target pixel and the pixel value of the peripheral pixel in the shaping image are close, The f·g weight (smoothing weight) becomes large.

In the present embodiment, the target image generated in S503 and subjected to the image processing effect is used as the shaping image. Hereinafter, the effectiveness of the present embodiment will be described with reference to FIG. 8 in comparison with the case where any one of the pair of viewpoint images generated in S501 is used as the shaping image.

FIG. 8 is a diagram for explaining the result of the shaping process of this embodiment. FIG. 8A-1 shows an example of the target image 800 to which the image processing effect generated in S503 is applied. In FIGS. 8A-1 and 8B-1, the left-right direction is defined as the X direction and the right direction is defined as the +X direction on the paper surface. In addition, the vertical direction orthogonal to the X direction in the plane of the drawing is defined as the Y direction and the upward direction is defined as the +Y direction. FIG. 8A-2 shows the pixel value profile 802 in the cross section at the position indicated by the alternate long and short dash line 801 in FIG. 8A-1. The horizontal axis represents the X coordinate, and the vertical axis represents the pixel value. The shape of the pixel value profile 802 is a step shape that greatly changes (decreases) at the position of Xs in the increasing direction of the X coordinate.

FIG. 8B-1 shows an example of the distance map 803. 8B-2 is a signal value profile in a cross section at a position indicated by a dashed-dotted line 804 in FIG. 8B-1 (corresponding to a position indicated by a dashed-dotted line 801 in FIG. 8A-1). 805 is represented by a solid line. The horizontal axis represents the X coordinate, and the vertical axis represents the distance signal value. Regarding the signal value of the distance, the signal value of the background far from the position of the camera is small (black in the distance map 803), and the signal value of the subject person at the close position is large (white in the distance map 803). In the example of FIG. 8, the person outline of the distance map extends outside the person outline (dotted line) of the corrective shaping image.

The shape of the signal value profile 805 shown in FIG. 8B-2 is a step shape that greatly changes (increases) at a position of Xa smaller than Xs in the increasing direction of the X coordinate. The target pixel position p in the equation (6) is shown by black dots 807, 808, 809, and 810, respectively. Areas where the g value of Expression (6) is large, that is, smoothing ranges are shown in line segments 811, 812, 813, and 814, respectively. In the pixel value profile 802 of the shaping image shown in FIG. 8A-2, the signal value sharply changes at the position Xs corresponding to the contour of the person. For this reason, at the positions of the black dots 808 and 809 indicating the pixel position of interest in the vicinity of the contour of the person, the smoothing ranges are indicated by the line segments 812 and 813, respectively, and follow the person contour of the shaping image which is the correct answer. become. As a result, when the value of the filter result Jp shown in Expression (6) is plotted, a graph line 806 shown by a dotted line is obtained. The shape of the graph line 806 is a step shape that greatly changes (increases) at the position of Xs in the increasing direction of the X coordinate. That is, the contour map of the person in the distance map matches the correct contour (the person contour in the shaping image) by the distance map shaping process.

Next, with reference to FIG. 9, a case where either one of the pair of viewpoint images is used as the shaping image will be described as a comparative example. 9(A-1) and (A-2), (B-1) and (B-2) are shown in FIGS. 8(A-1) and (A-2), (B-1) and (B), respectively. Since it is a diagram corresponding to (2), only different points will be described below.

FIG. 9A-1 shows one image 900 of the pair of viewpoint images generated in S501 of FIG. 9A-2 shows a pixel value profile 902 in a cross section at a position indicated by a dashed-dotted line 901 in FIG. 9A-1. The outline of the person image on the back side of the image 900 is unclear because of blurring. Therefore, the pixel value changes gently in the pixel value profile 902 (see the range from positions Xb to Xc). For comparison, the pixel value profile 802 of the target image to which the image processing effect is applied is shown by a dotted line.

FIG. 9B-1 shows an example of the distance map 903. 9B-2 shows a signal value profile 905 in a cross section at a position indicated by a dashed line 904 in FIG. 9B-1 (corresponding to a position indicated by a dashed line 901 in FIG. 9A-1). It represents. Black dots 907, 908, 909, and 910 indicate the target pixel position p, respectively. Line segments 911, 912, 913, and 914 indicate the smoothing range in Expression (6), respectively. Since the pixel value of the pixel value profile 902 of the shaping image changes gently, the smoothing ranges at the target pixel positions (see black dots 908 and 909) near the human contour are as shown by line segments 912 and 913, respectively. Narrows. Therefore, when the filter result Jp is plotted, a graph line 906 indicated by a dotted line is obtained. Comparing this graph line 906 with the pixel value profile 802 (see FIG. 9A-2) of the target image to which the image processing effect is applied, it can be seen that the person contours of the distance map do not match properly. Therefore, even if the image processing effect is applied with reference to the distance map, the correct processing cannot be performed.

In this embodiment, the example in which the target image to which the image processing effect is applied is applied to the shaping image has been described. When the condition that the difference between the F value when acquiring the target image to which the image processing effect is applied and the F value when acquiring the viewpoint image is within a predetermined range is satisfied, the viewpoint image is used as the shaping image. be able to.

After the shaping process, in S505 of FIG. 5, the edge enhancement unit 403 refers to the shaped distance map generated in S504, and performs edge enhancement processing according to the distance information on the target image generated in S503. .. For example, the correction is performed so that the edge strength becomes stronger as the subject is closer to the imaging device. Since the person contour in the distance map is accurately shaped so as to follow the person contour in the captured image, it is possible to obtain an image that has been subjected to the correct edge enhancement processing according to the distance.

In the present embodiment, when a viewpoint image for generating a distance map is acquired, an image processing effect corresponding to the distance is applied to a captured image obtained from a ray group that has passed through a pupil partial region having a different size. Correctly shape the distance map so that it can be applied accurately.

[Example 2]
Next, a second embodiment of the present invention will be described. In the first embodiment, by changing the number of additions of the divided pixels, the captured image is generated from the light ray group that has passed through the pupil partial region having a different size from that when the viewpoint image for generating the distance map is acquired. .. In the present embodiment, a case will be described in which a captured image is acquired at a different F value at a different time from the viewpoint image. In the present embodiment, the same reference numerals as those used in the first embodiment are used to omit the detailed description thereof and mainly describe the differences. The method of omitting such description is the same in the embodiments described later.

In FIG. 1, the parts of the digital camera of the present embodiment different from those of the first embodiment are an image pickup unit 1001 and an image processing unit 1002. FIG. 10A shows a pixel array of the image pickup unit 1001. The image pickup unit 1001 has a structure in which the pixel units 1100 are regularly arranged in a two-dimensional manner. FIG. 10B is an enlarged schematic view of one pixel portion 1100. The pixel portion 1100 includes a microlens 1101 and a pair of photoelectric conversion portions 1102A and 1102B. In the present embodiment, a pair of viewpoint images is acquired from the signals acquired by the pair of photoelectric conversion units 1102A and 1102B in the pupil division type image sensor.

FIG. 11 is a block diagram showing a specific configuration example of the image processing unit 1002. The difference from FIG. 4 is the alignment unit 1200. Further, FIG. 12 is a flowchart for explaining the processing content of the image processing unit 1002.

In S1300 of FIG. 12, the user images a subject with a digital camera, and viewpoint image data is acquired. Alternatively, when the viewpoint image data is recorded in the recording medium 108 in advance, a process of reading the viewpoint image data from the recording medium 108 is performed. In step S1301, the distance map generation unit 401 generates distance map data.

In S1302, an image captured at a different time and different F-number from the process of S1300 is acquired as a target image to which the image processing effect is applied. In the present embodiment, a case will be described in which a viewpoint image is captured with an F value that is smaller than that when the viewpoint image is captured. In this case, it is assumed that the same image as the image described with reference to FIG. 7 of the first embodiment is obtained. By changing the F value of the imaging optical system 104, the target image can be acquired by the light ray group that has passed through the pupil partial regions having different sizes.

In S1303, the alignment unit 1200 calculates the amount of misalignment between the viewpoint image acquired in S1300 and the target image acquired in S1302. A template matching process or the like is used as a method of calculating the amount of displacement. The alignment unit 1200 shifts the distance map by the amount of misalignment to adjust the position of the target image on which the image processing effect is applied. The present embodiment is an example in which the distance map is shifted by the amount of displacement. Since it suffices to cancel the relative positional deviation between the target image to which the image processing effect is applied and the distance map, it is possible to align the position with the distance map by shifting the target image by the positional deviation amount. Good.

In step S1304, the distance map shaping unit 402 shapes the distance map using the target image to which the image processing effect is applied as the shaping image. A distance map along the contour of the subject in the target image is obtained. The viewpoint image may be used as the shaping image as long as the difference between the F value when acquiring the target image to which the image processing effect is applied and the F value when acquiring the viewpoint image is within a predetermined range. In step S1305, the edge enhancement unit 403 executes edge enhancement processing on the target image to which the image processing effect is applied with reference to the shaped distance map.

According to the present embodiment, the image processing effect according to the distance is accurately performed on the captured images acquired at different shooting times and different F-numbers from when the viewpoint image for generating the distance map is acquired. Correctly shape the distance map so that it can be applied to.

[Example 3]
Next, a third embodiment of the present invention will be described. In the present embodiment, a case will be described in which a distance map is generated from a pair of images acquired at different focus positions, and a captured image is acquired at different times and different F values.

In FIG. 1, the parts of the digital camera of this embodiment different from those of the first embodiment are an image pickup unit 1401 and an image processing unit 1402. In the image pickup unit 1401, pixels formed by one photoelectric conversion unit are regularly arranged two-dimensionally with respect to one microlens. That is, this is the case where there is one pixel in FIG. 2B (when it is not divided). Further, the difference from the second embodiment in FIG. 11 is the distance map generation unit 1500.

FIG. 13 is a flowchart for explaining the processing content of the image processing unit 1402. In step S1600, the system control unit 101 acquires a pair of images with different focus positions by controlling the drive of the focus lens. Alternatively, when a pair of image data having different focus positions is recorded in the recording medium 108 in advance, a process of reading the image data from the recording medium 108 is performed.

In step S1601, the distance map generation unit 1500 generates a distance map. As a method of generating a distance map from a pair of images having different focus positions, the DFD (Depth From Defocus) method disclosed in Patent Document 3 is applied. In the DFD method, distance information is calculated using a focus image focused on the subject and a defocus image in which the subject image has a different blurring of the subject image.

In S1602, a target image to which the image processing effect is applied is acquired at a different time and a different F value than in S1600. In the present embodiment, the image captured with the F value on the open side is acquired as compared with the case of capturing a pair of images with different focus positions. By changing the F value of the imaging optical system 104, the target image can be acquired by the light ray group that has passed through the pupil partial regions having different sizes.

In step S1603, the alignment unit 1200 calculates the amount of misalignment between one of the pair of images with different focus positions and the target image to which the image processing effect is applied, and shifts the distance map by the amount of misalignment. To do. In step S1604, the distance map is shaped using the target image as the shaping image, and the distance map along the contour of the subject of the target image is obtained. If the difference between the F value when acquiring the target image to which the image processing effect is applied and the F value when acquiring the pair of images having different focus positions is within a predetermined range, the focus position is determined as the shaping image. A pair of different images can be used. At this time, of the pair of images with different focus positions, the image in which the focused subject image is the same as the focused subject image in the target image to which the image processing effect is applied is selected as the shaping image. Processing is performed.

In step S1605, the edge enhancement unit 403 applies edge enhancement to the target image to which the image processing effect is applied with reference to the shaped distance map.

In this embodiment, a distance map is generated from a pair of images acquired at different focus positions. A distance map can be correctly shaped so that an image processing effect corresponding to a distance can be accurately applied to a captured image acquired at a different time and different F value from those of a pair of images.

[Example 4]
Next, a fourth embodiment of the present invention will be described. In this embodiment, a distance map is generated from viewpoint images acquired by an image pickup apparatus having a plurality of image pickup optical systems. Further, the distance map is shaped so as to follow the contour of the subject image in the captured image acquired via the imaging optical system different from the imaging optical system that acquired the viewpoint image.

FIG. 14 is a block diagram showing the functional arrangement of the digital camera 1700 of this embodiment. The digital camera 1700 has a configuration including three sets of the imaging optical system 104 and the imaging unit 1401 which are independent of each other. Of the three sets of the image pickup optical system and the image pickup section arranged side by side in the horizontal direction, the first set CM1 located at the center functions as a primary camera. The second set CM2 and the third set CM3 respectively arranged on both sides of the first set function as a secondary camera. Further, the processing contents executed by the image processing unit 1701 are different from those in the first to third embodiments.

FIG. 15 is a block diagram showing a specific configuration example of the image processing unit 1701. The difference from FIG. 4 is the morphing processing unit 1800. 16 is a flowchart for explaining the processing content of the image processing unit 1701.

In S1900 of FIG. 16, the system control unit 101 acquires viewpoint image data by controlling the driving of the second set CM2 and the third set CM3. Alternatively, when the viewpoint image data is recorded in the recording medium 108 in advance, a process of reading the image data from the recording medium 108 is performed. In step S1901, the distance map generation unit 401 relatively shifts the image from the right-viewpoint image to the left-viewpoint image by using the same method as in step S502 in FIG. By deviating relatively, the defocus amount is calculated respectively. Thereby, the distance map in each secondary camera (CM2, CM3) is generated.

In step S1902, the system control unit 101 acquires the target image to which the image processing effect is applied by controlling the drive of the first set CM1. That is, the image acquired by the primary camera is the target image. In the present embodiment, it is assumed that the shooting operation is performed with the F value on the open side as compared with the case of shooting with the secondary camera.

In S1903, the morphing processing unit 1800 generates a distance map in the primary camera by performing morphing processing on the distance map in each secondary camera generated in S1901. As the morphing process, a means for extracting feature points on a two-dimensional image and associating with each two-dimensional image, and a technique for estimating an object on the two-dimensional image as a set of three-dimensional models are widely known. There is. Since these morphing processes are well known in the art, the details are omitted here.

In step S1904, the distance map shaping unit 402 shapes the distance map in the primary camera using the target image to which the shaping image is subjected to the image processing effect. This makes it possible to obtain a distance map along the contour of the subject image in the target image to which the image processing effect acquired by the primary camera is applied. If the difference in F value between the primary camera and the secondary camera during shooting is within a predetermined range, an image obtained by performing morphing processing on the image captured by the secondary camera may be used as the shaping image. In step S1905, the edge enhancement unit 403 applies edge enhancement to the target image to which the image processing effect is applied by referring to the shaped distance map.

In the present embodiment, a plurality of image pickup units having an image pickup optical system and an image pickup unit are provided, and a distance map is generated from the viewpoint images acquired by the first and second image pickup units. The distance map can be correctly shaped so that the image processing effect corresponding to the distance can be accurately applied to the imaged image acquired by the third imager different from the imager that acquired the viewpoint image.
Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof.

[Other Examples]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Digital camera 101: System control unit 104: Imaging optical system 105, 1001, 1401: Imaging unit 107, 1002, 1402: Image processing unit 108: Recording medium

Claims (15)

A first acquisition unit that acquires a pair of first image data obtained from a light ray group that passes through the first pupil partial region of the imaging optical system;
Second acquisition means for acquiring second image data obtained from a light ray group passing through the second pupil partial area of the imaging optical system, the size of which is different from that of the first pupil partial area;
Generating means for generating depth information corresponding to the depth of the object in the image in the depth direction from the pair of first image data;
The depth information is smoothed using the second image data or data of an image captured within a range in which the difference between the F value when the second image data is acquired and the F value when the second image data is acquired. An image processing apparatus comprising: a shaping unit that performs shaping processing. The image pickup means includes a plurality of microlenses and an image pickup device having a plurality of photoelectric conversion units corresponding to the respective microlenses,
The first acquisition unit acquires a plurality of image data having parallax from the photoelectric conversion unit in the divided first area and the photoelectric conversion unit in the second area. 1. The image processing device according to 1. The pair of first image data is obtained by adding the image data obtained by adding the outputs of the photoelectric conversion units in the first region and the output of the photoelectric conversion units in the second region. The image processing apparatus according to claim 2, wherein the image processing apparatus acquires the image data. The second acquisition unit acquires the second image data from a photoelectric conversion unit at a predetermined position among the plurality of photoelectric conversion units corresponding to each microlens. Alternatively, the image processing apparatus according to claim 3. The second acquisition unit acquires the second image data captured at a different time and a different F value than when the first image data forming the pair is acquired. The image processing device according to. The first acquisition means respectively acquires the pair of first image data at different focus positions,
The second acquisition unit acquires the second image data imaged at a different time and different F value than when the first image data was acquired. Image processing device. The image processing device according to claim 1, wherein the first pupil partial region is larger than the second pupil partial region. An image processing apparatus that performs image processing by acquiring a plurality of image data captured by a plurality of image capturing means and depth information corresponding to a depth of a subject in the captured image in a depth direction,
A first acquisition means for acquiring a plurality of first image data respectively imaged by the first and second imaging means;
Second acquisition means for acquiring the second image data imaged by the third imaging means with an F value on the open side of the first image data ;
Generating means for generating the depth information from the plurality of first image data;
A shaping process for smoothing the depth information is performed using the second image data or image data in which a difference from the F value when the second image data is acquired is within a predetermined range. An image processing device comprising: a shaping unit. The first to third image pickup means each include an image pickup optical system and an image pickup unit,
The image processing apparatus according to claim 8, wherein a pupil region of the image pickup optical system included in the third image pickup unit is larger than a pupil region of the image pickup optical system included in the first and second image pickup units. .. The image according to any one of claims 1 to 9, further comprising a recording unit that records the depth information that has been subjected to the shaping processing by the shaping unit and the second image data in association with each other. Processing equipment. When the shaping unit generates a signal value after shaping by smoothing the signal value of the pixel of interest and its peripheral pixels with respect to the pixel of interest of the depth information, the pixel value of the pixel of interest in the image to be referred to The image processing device according to claim 1, wherein the closer the pixel value of the peripheral pixel is, the larger the weight of the smoothing of the peripheral pixel is. An image processing apparatus according to any one of claims 1 to 11,
An imaging device comprising an imaging means. A first acquisition step of acquiring a pair of first image data obtained from a light ray group passing through the first pupil partial region of the imaging optical system;
A second acquisition step of acquiring second image data obtained from a light ray group passing through the second pupil partial area of the imaging optical system having a size different from that of the first pupil partial area;
A generation step of generating depth information corresponding to the depth of the subject in the image in the depth direction from the pair of first image data;
The depth information is smoothed using the second image data or data of an image captured within a range in which the difference between the F value when the second image data is acquired and the F value when the second image data is acquired. A method of controlling an image processing device, comprising: a shaping step of performing a shaping process. A first acquisition step of acquiring a plurality of first image data respectively imaged by the first and second imaging means,
A second acquisition step of acquiring the second image data imaged by the third imaging means with an F value on the open side of the first image data ;
A generation step of generating depth information corresponding to the depth of the subject in the image in the depth direction from the plurality of first image data;
A shaping process for smoothing the depth information is performed using the second image data or image data in which a difference from the F value when the second image data is acquired is within a predetermined range. A shaping method, and a method for controlling an image processing device. A computer program that causes a computer to execute the steps according to claim 13 or 14.