JP6818734B2 - Image processing equipment, imaging equipment, image processing methods and programs - Google Patents

Description

The present invention relates to an image processing device that synthesizes a plurality of images having different focus positions.

When using a digital camera or the like to capture multiple subjects whose distances are significantly different from each other, or when capturing a subject that is long in the depth direction, the depth of field is insufficient, so only a part of the subject can be focused. It may not be. In order to solve this problem, Patent Document 1 describes the pixel values of pixels existing at overlapping positions when compositing in each of a plurality of captured images, and the compositing ratio according to the contrast value of each pixel. A technique for synthesizing using is disclosed. By giving a large composition ratio to an image having a large contrast value, the images are combined so that the ratio of the images in the in-focus state becomes large.

Japanese Unexamined Patent Publication No. 2014-17539

However, when image composition is performed using the technique described in Patent Document 1, the following problems of perspective competition and a problem that a blurred region appears at the edge portion of the composite image may occur.

FIG. 15 is a diagram for explaining perspective competition. In FIG. 15, it is assumed that a plurality of subjects having different distances from the camera exist at the same position on the image. FIG. 15A is an imaging composition, in which the subject 1501 is the front subject and the subject 1502 is the rear subject. 15 (b) is an image captured when the front subject 1501 is in focus, FIG. 15 (c) is an image captured when the rear subject 1502 is in focus, and FIG. 15 (d) is an image in which the subject 1501 and the subject 1502 overlap. It is a figure focusing on the area. When the front subject 1501 is focused as shown in FIG. 15B, the rear subject 1502 is shielded by the front subject 1501. On the other hand, when the rear subject 1502 is in focus as shown in FIG. 15C, the rear subject 1502 may be seen through due to the blur of the front subject 1501. When the contrast value of the rear subject 1502 in the focused state is higher than the contrast value of the front subject 1501 in the focused state, the composition is performed by the method described in Patent Document 1, as shown in FIG. 15 (e). The rear subject will be output. Therefore, a composite image in which a part of the subject in the foreground is missing is generated.

FIG. 16 is a diagram for explaining an example in which a blurred region appears at an edge portion of a composite image. In the case shown in FIG. 16, the digital camera 100 captures five images 1610 to 1650 at focus positions 1601 to 1605 with respect to an X-shaped subject in front of a white background, and with respect to the captured images. It is assumed that the composite image 1600 is generated by the synthesis. The image 1610 is taken at the focus position 1601, and the images 1620 to 1650 correspond to the focus positions 1602 to 1605. After the images 1610 to 1650 have been aligned, the pixels 1611 to 1651 are in the same position and the pixels 1612 to 1652 are in the same position. When a composite image is created using the technique disclosed in Patent Document 1, the pixel 1612 to 1652 having the highest contrast value is selected and used at the same position in the composite image. Therefore, it is illustrated that the digital camera uses the pixel 1632 for the composite image 1600. However, when a digital camera selects the pixel 1611 to 1651 having the highest contrast value and uses it at the same position in the composite image in the same manner, the contrast value of the pixel 1611 or the pixel 1651 is higher than the contrast value of the pixel 1631. The value will be higher. Therefore, the pixel 1611 or the pixel 1651 is selected and used for the composite image. As a result, as shown in the figure, the composite image 1600 has a blurred region around the edge portion.

The present invention has been made in view of the above problems, and an image processing apparatus capable of reducing the blurred region of an edge portion in an image synthesized by using a plurality of images having different focus positions and at the same time preventing deterioration of perspective competition. It is to provide.

In order to solve the above problems, the present invention comprises a detection means for detecting contrast values of a plurality of images, a determination means for determining the degree of perspective competition based on the contrast values of the plurality of images, and the determination. The first adjustment is made to the contrast value of the image so that the change of the contrast value of the edge portion in each of at least a part of the images of the plurality of images after the means makes the determination is smooth. , The degree of perspective competition is higher in each of the first adjusting means for generating the result of the plurality of adjustments and the image having the contrast value after the first adjusting means has made the first adjustment. It is a second adjusting means that calculates a smaller gain at a strong position, applies the gain to the results of the plurality of adjustments, and then makes a second adjustment so as to generate an integrated value. A second adjusting means for generating one integrated value for one position of the image and the integrated value of the position corresponding to each of the plurality of images. Provided is an image processing apparatus comprising: a setting means for setting a composition ratio and a generation means for generating a composite image by synthesizing the plurality of images based on the composition ratio.

According to the configuration of the present invention, there is provided an image processing device that reduces blurring of an edge portion and prevents serious transmission of a rear subject due to perspective competition in an image obtained by capturing and synthesizing a plurality of images having different focus positions. can do.

It is a block diagram which shows the structure of the digital camera which concerns on embodiment of this invention. It is a flowchart for demonstrating the generation of the composite image in 1st Embodiment of this invention. It is a flowchart for demonstrating the imaging in 1st Embodiment of this invention. It is a flowchart for demonstrating the alignment in 1st Embodiment of this invention. It is a flowchart for demonstrating composition of an image in 1st Embodiment of this invention. It is a flowchart for demonstrating the generation of the synthetic map in 1st Embodiment of this invention. It is a block diagram for demonstrating the generation of the synthetic map in 1st Embodiment of this invention. It is a figure for demonstrating the contrast value in 1st Embodiment of this invention. It is a figure for demonstrating the determination of perspective competition in the 1st Embodiment of this invention. It is a figure for demonstrating the perspective competition in the 1st Embodiment of this invention. It is a graph for demonstrating the relationship between the presence / absence of perspective competition and the change of contrast value in the 1st Embodiment of this invention. It is a graph which shows the relationship between the gain and the degree of perspective competition in the 1st Embodiment of this invention. It is a flowchart for demonstrating the generation of the synthetic map in the 2nd Embodiment of this invention. It is a block diagram for demonstrating the generation of the synthetic map in the 2nd Embodiment of this invention. It is a figure for demonstrating the perspective competition. It is a figure for demonstrating an example in which a blurred region appears at the edge part of a composite image.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First Embodiment)
FIG. 1 is a block diagram showing a structure of a digital camera as an image processing device according to the present embodiment. The digital camera 100 can capture a still image, record information on the in-focus position, calculate a contrast value, and synthesize an image. Further, the digital camera 100 can perform enlargement processing or reduction processing on an image captured and saved or an image input from the outside.

The control unit 101 is, for example, a signal processor such as a CPU or an MPU, and controls each part of the digital camera 100 while reading a program built in the ROM 105 described later in advance. For example, as will be described later, the control unit 101 issues a command to the imaging unit 104, which will be described later, regarding the start and end of imaging. Alternatively, an image processing command is issued to the image processing unit 107, which will be described later, based on the program built in the ROM 105. The command from the user is input to the digital camera 100 by the operation unit 110 described later, and reaches each part of the digital camera 100 through the control unit 101.

The drive unit 102 is composed of a motor or the like, and mechanically operates the optical system 103 described later under the command of the control unit 101. For example, based on the command of the control unit 101, the drive unit 102 moves the position of the focus lens included in the optical system 103 to adjust the focal length of the optical system 103.

The optical system 103 includes a zoom lens, a focus lens, an aperture, and the like. The diaphragm is a mechanism for adjusting the amount of transmitted light. The focusing position can be changed by changing the position of the lens.

The imaging unit 104 is a photoelectric conversion element, and performs photoelectric conversion that converts an incident optical signal into an electric signal. For example, a CCD sensor, a CMOS sensor, or the like can be applied to the image pickup unit 104. The imaging unit 104 is provided with a moving image imaging mode, and can capture a plurality of temporally continuous images as frames of the moving image.

The ROM 105 is a read-only non-volatile memory as a recording medium, and stores, in addition to the operation program of each block included in the digital camera 100, parameters and the like necessary for the operation of each block. The RAM 106 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each block included in the digital camera 100.

The image processing unit 107 performs various image processing such as white balance adjustment, color interpolation, and filtering on the image output from the image pickup unit 104 or the image signal data recorded in the built-in memory 109 described later. .. Further, the image signal data captured by the imaging unit 104 is compressed according to a standard such as JPEG.

The image processing unit 107 is composed of an integrated circuit (ASIC) that is a collection of circuits that perform specific processing. Alternatively, the control unit 101 may perform processing according to the program read from the ROM 105 so that the control unit 101 also has a part or all of the functions of the image processing unit 107. When the control unit 101 also has all the functions of the image processing unit 107, it is not necessary to have the image processing unit 107 as hardware.

The display unit 108 is a liquid crystal display, an organic EL display, or the like for displaying an image temporarily stored in the RAM 106, an image stored in the built-in memory 109 described later, a setting screen of the digital camera 100, or the like. is there.

The built-in memory 109 is a place for recording an image captured by the imaging unit 104, an image processed by the image processing unit 107, information on the focusing position at the time of image imaging, and the like. A memory card or the like may be used instead of the built-in memory.

The operation unit 110 is, for example, a button, a switch, a key, a mode dial, or the like attached to the digital camera 100, or a touch panel that is also used as the display unit 108. The command from the user reaches the control unit 101 via the operation unit 110.

FIG. 2 is a flowchart for explaining the generation of the composite image in the present embodiment. In step S201, the imaging unit 104 captures a plurality of images having different focus positions. In step S202, the control unit 101 aligns the plurality of images captured by the imaging unit 104 in step S201 so that the angles of view match. In step S203, the image processing unit 107 synthesizes the image after the alignment is performed, and generates a composite image. Each step will be described in detail below.

FIG. 3 is a flowchart for explaining the imaging in step S201 in the present embodiment.

In step S301, the control unit 101 sets the focus position. For example, the user specifies the focusing position through the touch panel that is also used by the display unit 108, and specifies a plurality of focusing positions at equal intervals before and after the focus position corresponding to the focusing position in the optical axis direction. At the same time, the control unit 101 determines the imaging order in the order of distance at the set focus position.

In step S302, the imaging unit 104 takes an image at the focus position in which the imaging order is the earliest among the focus positions set in step S301 that have not been imaged.

In step S303, the control unit 101 determines whether or not imaging has been performed at all the focus positions set in step S301. When the images are taken at all the focus positions, the process according to the flowchart shown in FIG. 3 is finished, and if there is a focus position that has not been imaged yet, the process returns to step S302.

FIG. 4 is a flowchart for explaining the alignment in step S202 in the present embodiment.

In step S401, the control unit 101 acquires a reference image for alignment from the images captured by the imaging unit 104 in step S201. The reference image for alignment shall have the earliest imaging order. Alternatively, since the angle of view changes slightly between the captured images by taking an image while changing the focus position, the angle of view may be the narrowest among the captured images.

In step S402, the control unit 101 acquires the target image for the alignment process. It is assumed that the target image is an image other than the reference image acquired in step S401 and the alignment process has not been completed. If the image having the earliest imaging order is used as the reference image, the control unit 101 may acquire the target images in the order of imaging.

In step S403, the control unit 101 calculates the amount of displacement between the reference image and the target image. An example of the calculation method will be described below. First, the control unit 101 sets a plurality of blocks on the reference image. The control unit 101 is preferably set so that the size of each block is the same. Next, the control unit 101 sets a range wider than the block of the reference image as a search range at the same position as each block of the reference image of the target image. In each search range of the target image, the control unit 101 calculates a corresponding point at which the sum of the absolute differences in brightness from the block of the reference image (Su of Absolute Difference, hereinafter referred to as SAD) is minimized. The control unit 101 calculates the deviation of the position in step S403 as a vector from the center of the block of the reference image and the corresponding point described above. In addition to SAD, the control unit 101 uses Sum of Squared Difference (hereinafter referred to as SSD), normalized cross-correlation (hereinafter referred to as NCC), and the like in the calculation of the corresponding points described above. You may.

In step S404, the control unit 101 calculates the conversion coefficient from the amount of displacement between the reference image and the target image. The control unit 101 uses, for example, a projective conversion coefficient as the conversion coefficient. However, the conversion coefficient is not limited to the projection conversion coefficient, and a simplified conversion coefficient of only the affine transformation coefficient or the horizontal / vertical shift may be used.

In step S405, the image processing unit 107 converts the target image using the conversion coefficient calculated in step S404.

For example, the control unit 101 can be transformed by using the equation shown in (Equation 1).

In (Equation 1), (x', y') indicates the coordinates after the transformation, and (x, y) indicates the coordinates before the transformation. The matrix A shows the deformation coefficient calculated by the control unit 101 in step S404.

In step S406, the control unit 101 determines whether or not all the images other than the reference image have been aligned. When the alignment is performed on all the images other than the reference image, the process in this flowchart is finished, and if there is an image that has not been processed yet, the process returns to step S402.

FIG. 5 is a flowchart for explaining the composition of images in step S203 in the present embodiment.

In step 501, the image processing unit 107 calculates the contrast value for each image (including the reference image) after the alignment is performed. As an example of the method of calculating the contrast value, for example, first, the image processing unit 107 calculates the brightness Y from the color signals Sr, Sg, and Sb of each pixel by using the following (Equation 2).
Y = 0.299Sr + 0.587Sg + 0.114Sb ... (Equation 2)

Next, as shown in the following (Equation 3) to (Equation 5), the contrast value in the pixel of interest is added to the matrix L of the brightness Y in the pixel range of 3 × 3 including the pixel of interest by using the Sobel filter. Calculate I.

Further, the above-mentioned method of calculating the contrast value is only an example, and for example, an edge detection filter such as a Laplacian filter or a bandpass filter that passes through a predetermined band can be used as the filter to be used.

In step S502, the image processing unit 107 generates a composite map. The specific method of generating the composite map will be described later.

In step S503, the image processing unit 107 generates a composite image according to the composite map calculated in step S502. When the composition ratio changes from 0% to 100% (or changes from 100% to 0%) between adjacent pixels with respect to the composition ratio calculated in this way, the unnaturalness at the composition boundary becomes conspicuous. become. Therefore, the image processing unit 107 appropriately adjusts the composition ratio between the adjacent pixels so that the composition ratio does not suddenly change between the adjacent pixels.

<Generation of composite map>
In the following, the generation of the composite map in step S502 will be described in detail.

FIG. 6 is a flow chat for explaining the generation of the composite map in step S502 in the present embodiment.

In step S601, the image processing unit 107 determines the perspective conflict. In step S602, the image processing unit 107 calculates the gain Gain [k] based on the result of the determination of perspective competition. The conventional technique may be used to determine the perspective competition, and the details will be described later.

In step S603, the image processing unit 107 performs filter processing on the contrast value corresponding to each pixel calculated in step S501. Examples of the filter type include a MAX filter and an average value filter that output the maximum value of the contrast value within the reference range defined by the number of taps. Moreover, in step S603, the image processing unit 107 separately applies filters having different numbers of taps to the contrast value to obtain a plurality of data in which the contrast value is smoothed. Here, for convenience of explanation, two types of filters, N [0] and N [1], are applied, but the number of filters is not limited to two. It is assumed that the number of taps of the filter of N [0] is smaller than that of N [1]. That is, the reference range of the filter of N [0] is narrower than that of N [1]. The filter of N [0] may be in the form of a through output (a signal having the same input and output).

FIG. 7 is a block diagram for explaining the generation of the composite map in step S502 in the present embodiment. In FIG. 7, the contrast value 701 (Cm (x, y)) means the contrast value of the coordinates (x, y). When the contrast value is calculated for each pixel, the contrast value of the block in which the coordinates (x, y) are located is taken as the contrast value of the coordinates (x, y). In order to reduce the calculation load, the contrast value corresponding to each pixel may be divided into a plurality of blocks, and the representative contrast value for each block (for example, the average value of the contrast values) may be obtained. The image processing unit 107 applies the filter 711 (N [0]) and the filter 712 (N [1]) to Cm (x, y), respectively. In such a process, the fluctuation of the contrast value can be smoothed and the peak of the contrast value can be expanded to the periphery for the calculation of the evaluation value described later.

FIG. 8 is a diagram for explaining the contrast value in the present embodiment. In the following, the contrast values before and after applying the filter will be described with reference to FIG.

Consider a case where a subject having a uniform pattern as shown in FIGS. 8 (a) and 8 (b) is imaged. FIG. 8A shows a captured image when the subject is in focus, and FIG. 8B shows a captured image when the subject is not in focus. For simplicity, the contrast values on the x-axis passing through the center of the subject shown in FIGS. 8 (a) and 8 (b) are analyzed one-dimensionally.

8 (c) shows the relationship between the contrast value of the in-focus subject shown in FIG. 8 (a) and the x-axis coordinates, and FIG. 8 (d) is shown in FIG. 8 (b). The relationship between the contrast value of the in-focus subject and the x-axis coordinates is shown. Since the subject has a uniform pattern in FIG. 8 (a), the contrast value shows a uniform value when the x-coordinate hits the subject in FIG. 8 (c). On the other hand, when the subject is blurred as shown in FIG. 8B, the contrast value when the x-coordinate hits the inside of the subject is lower than that in FIG. 8A. In FIG. 8 (b), since blurring appears around the subject, the contrast value in FIG. 8 (d) does not decrease sharply on both sides as in FIG. 8 (c), but gradually decreases. Shown.

Comparing FIG. 8 (c) and FIG. 8 (d), the contrast value shown in FIG. 8 (c) is higher than that in FIG. 8 (d) in the range 801 and FIG. 8 (d) is outside the range 801. It can be seen that the contrast value shown in c) is lower than that in FIG. 8 (d). When depth composition is performed on FIGS. 8 (a) and 8 (b) using the conventional technique described in the background, the blur of FIG. 8 (b) is used for the composite image in the portion outside the range 801. I will be broken.

8 (e) and 8 (f) are graphs showing the contrast values after filtering the 8 (c) and 8 (d), respectively. It can be seen that the shapes of the graphs of FIGS. 8 (e) and 8 (f) are further expanded to both sides as compared with FIGS. 8 (c) and 8 (d). Originally, the contrast values of the in-focus image and the subject shown in FIG. 8 (c) are higher than those of the blurred image shown in FIG. 8 (d), and even after filtering the blurred image, they are higher in a wider range than the blurred image. Comparing FIG. 8 (e) and FIG. 8 (f), the contrast value shown in FIG. 8 (e) is higher in the range 802 than in FIG. 8 (f), and outside the range 802, FIG. It can be seen that the contrast value shown in e) is lower than that in FIG. 8 (f). When depth composition is performed in the calculation of the evaluation value described later, the blur of FIG. 8B is used in the composite image in the portion outside the range 802. However, since the range 802 is wider than the range 801 the blur of the composite image created based on the evaluation value calculated here is less than the blur of the composite image obtained by the prior art in the background described above. Similarly, if a filter with a larger number of taps is applied, a composite image with even less blur can be obtained. In the present embodiment, as will be described later, the result of adjusting the contrast value by applying a filter having a different number of taps and further applying a gain is calculated as an evaluation value, and the composition ratio is calculated.

By applying the smoothing filter to the contrast value in this way, it can be expected that the contrast value of the focused image is higher than the contrast value of the blurred image in the vicinity of the edge portion. In the calculation of the evaluation value described later, the evaluation value of the part of the image that is in focus is calculated high, and it can be expected that the image that is in focus is selected from the blurred image in the composition.

In step S604, the image processing unit 107 applies a gain Gain [k] to the result obtained by filtering in step S603. For example, in FIG. 7, the image processing unit 107 applies a gain Gain [0] to the result after applying the filter 711 (N [0]), and then applies the filter 712 (N [1]). Gain [1] is applied to the result of.

In step S605, the image processing unit 107 integrates the contrast values gained in step S602 to generate the evaluation value Em (x, y) of the mth image. Examples of the integration method include addition of contrast values with gain and addition averaging. For example, as shown in FIG. 7, the image processing unit 107 adds a value of the contrast value 721 after the gain Gain [0] is applied and the contrast value 722 after the gain Gain [0] is applied. The evaluation value is Em (x, y). Alternatively, the image processing unit 107 sets the added average value of the contrast value 721 after the gain Gain [0] is applied and the contrast value 722 after the gain Gain [0] is applied as the evaluation value Em (x, y). ).

In step S606, the control unit 101 determines whether or not the processing in steps S601 to S603 has been performed on all the images, and if all the images have been processed, the process proceeds to step S605.

In step S607, the image processing unit 107 calculates the composition ratio.

Specifically, 100% of the composite ratio is given to the image having the highest evaluation value among the pixels located at the same coordinates (x, y) in the mth image. As described above, the image processing unit 107 may appropriately adjust the composition ratio so that a sudden change does not occur between adjacent pixels.

Alternatively, the image processing unit 107 calculates the composition ratio according to the following (Equation 6).

However, in (Equation 6), M indicates the number of images, and (x, y) indicates the coordinates. Further, in step S503, the calculation result of (Equation 6) is used and substituted into the following (Equation 7) to generate a composite image.

However, in (Equation 7), Ik (x, y) indicates the pixel value of the coordinates (x, y) of the kth image, and O (x, y) is the coordinates (x, y) of the composite image. Indicates the pixel value of.

Further, in the calculation of (Equation 6), it may be appropriately modified. For example, in the calculation of the denominator of (Equation 6), if the value of Em (x, y) is less than or equal to a predetermined value, it may be regarded as 0. Further, if the calculation result on the right side of (Equation 6) becomes a predetermined value or more, Am may be regarded as 100%.

<Gain calculation>
In the following, the calculation of step S602 gain Gain [k] will be described in detail.

First, in step S601, the image processing unit 107 determines from the contrast values of the plurality of images whether or not perspective competition has occurred. Conventional techniques may be used to determine perspective competition, which will be briefly described below with an example.

FIG. 9 is a diagram for explaining the determination of perspective competition in the present embodiment. FIG. 9 shows the contrast values of the corresponding pixels of a plurality of images having different focus positions. The number of the focus position on the horizontal axis indicates the distance between the focus position and the digital camera 100. When the focus position is 8, the distance between the focus position and the digital camera 100 is the farthest (farthest side), and when the focus position is 1, the distance between the focus position and the digital camera 100 is the shortest (closest side). And.

In FIG. 9A, it is shown that the contrast value 912 has reached the maximum value when the focus position is 2. The image processing unit 107 determines that the subject is in focus when the focus position is 2.

In FIG. 9B, it is shown that the contrast values 922 and 927 have reached the maximum values at the focus positions 2 and 7, respectively. The image processing unit 107 focuses on the two front and rear subjects, and determines that the cause is perspective competition. That is, as shown in FIG. 9B, the image processing unit 107 determines that perspective competition has occurred when two or more maximum values exist.

In the case shown in FIG. 9B, the contrast value 927 is higher than the contrast value 922. Therefore, if the image processing unit 107 gives a higher composition ratio to the pixel having the contrast value 927, the perspective competition as described above The problem arises. That is, in the region where two subjects overlap in the composite image, the farther subject is transmitted to the closer subject.

As shown in FIG. 9C, there may be a maximum value of 932 in which the difference in contrast value between the pixels on both sides is extremely small as compared with other maximum values. The image processing unit 107 determines that such a value is highly likely to be affected by noise or the like. That is, the image processing unit 107 does not determine that the subject is in focus at the focus position 2. The image processing unit 107 sets a threshold value in advance and uses the threshold value to distinguish between noise and a maximum value.

In step S601, the image processing unit 107 determines where the perspective conflict is occurring by performing the above processing on all the corresponding pixels.

Next, in step S602, the image processing unit 107 calculates the gain based on the degree of perspective competition. Here, it is necessary to calculate the gain in consideration of the degree of perspective competition. In areas where perspective competition occurs, if the contrast is simply expanded to the surroundings, the contrasts of subjects with different focus positions will be mixed, and if the evaluation value described later is calculated to create a composite image, perspective competition will further advance. There is a possibility that it will end up. This tendency becomes more serious as the degree of perspective competition increases. Therefore, the image processing unit 107 calculates the gain based on the degree of perspective competition by a method different from that of the region other than the perspective competition region.

First, the image processing unit 107 calculates the degree of perspective competition. As an example, the image processing unit 107 may calculate the sum of the contrast values when the two subjects in which perspective competition occurs are in focus as the degree of perspective competition. In general, the higher the contrast value when the two front and rear subjects are in focus, the stronger the degree of perspective competition is because the two subjects can be seen more clearly than the surroundings when perspective competition occurs. That is, the sum of the contrast values 922 and 927 shown in FIG. 9B is used as the degree of perspective competition.

FIG. 10 is a diagram for explaining perspective competition in the present embodiment.

FIG. 10A shows a state in which the digital camera 100 images the subject 1001 existing at the position 1011. In this case, there is no subject at position 1012.

FIG. 10B shows how the digital camera 100 images the subject 1001 existing at the position 1011 and the subject 1002 existing at the position 1012.

FIG. 10 (c) shows an image captured when the digital camera 100 focuses on each of the positions 1011 and 1012 in the case of FIG. 10 (c). When the digital camera 100 focuses on the position 1011 (when the front focus is on), the subject 1001 is clearly visible, and when the digital camera 100 focuses on the position 1012 (when the rear focus is on), the subject 1001 is out of focus. Understand.

FIG. 10D shows an image captured when the digital camera 100 focuses on the position 1011 and the position 1012 in the case of FIG. 10B. When the digital camera 100 focuses on the position 1011 (when the front focus is on), the subject 1001 is clearly visible and the subject 1002 is out of focus. Similarly, when the digital camera 100 focuses on the position 1011 (when the front focus is on), the subject 1001 is clearly visible and the subject 1002 is out of focus. Further, FIG. 10D shows a state in which perspective competition is occurring. When the digital camera 100 is in focus at the position 1011 the subject 1001 appears to be located in front of the subject 1002. However, when the digital camera 100 is in focus at the position 1012, the subject 1001 appears to be located behind the subject 1002.

The size of the blurred subject image in FIGS. 10 (c) and 10 (d) is exaggerated.

FIG. 11 is a graph for explaining the relationship between the presence / absence of perspective competition and the change in contrast value in the present embodiment. FIG. 11 (a) corresponds to the image shown in FIG. 10 (c), that is, when perspective conflict occurs, and FIG. 11 (b) shows the image shown in FIG. 10 (d), that is, perspective conflict occurs. Correspond if there is. The graphs of FIGS. 11 (a) and 11 (b) show the relationship between the contrast value and the x-axis. The x-axis corresponds to the x-axis shown in FIGS. 10 (c) and 10 (d), and the origin 0 also corresponds to each other.

FIG. 11A shows the contrast values of the front and rear pins before the curves 1101 and 1102 are filtered. Curves 1103 and 1104 show the contrast values of the front and rear pins after filtering. As seen from FIG. 11A, the contrast value of curve 1101 falls below curve 1102 in some areas before filtering, but after filtering, curve 1103 is completely above curve 1104. It looks like. That is, by applying the filter, the problem that a blurred region is generated around the edge portion is solved.

FIG. 11B shows the contrast values of the front and rear pins before the curves 1111 and 1112 are filtered. Curves 1113 and 1114 show the contrast values for the front and rear pins after filtering. As seen from FIG. 11B, the contrast value of curve 1111 falls below curve 1112 in region 1115 before filtering. However, it can be seen that after filtering, the curve 1114 falls below the curve 1113 in the region 1115. Originally, the subject 1001 is not captured in the region 1115, but when compositing according to the curve 1113 and the curve 1114, the blurred portion of the subject 1001 is used for the region 1115, resulting in a composite image in which the blur is not eliminated.

As shown above, it has been shown that due to perspective competition, filtering does not eliminate the occurrence of a blurred region around the edge portion, but conversely worsens it. For this reason, when perspective contention occurs, it is necessary to filter in a different way than usual.

FIG. 12 is a graph showing the relationship between the gain and the degree of perspective competition in the present embodiment. FIG. 12A shows the case of k = 0, and FIG. 12B shows the case of k = 1. The stronger the degree of perspective competition, the more the perspective competition may progress when the contrast value is widened. Therefore, the perspective competition is suppressed by reducing the gain.

Note that the larger the number of taps, the wider the contrast value after filtering the in-focus image and the smoother the contrast value. Therefore, the larger the number of taps, the larger the gain. That is, in the gain Gain [k], the higher k, the higher g_max in FIG. 12 needs to be.

Further, in the region where perspective competition occurs, the image processing unit 107 may appropriately adjust the gain depending on the focus position. As described above, in order to alleviate the phenomenon of perspective competition, the contrast value may be multiplied by a weighting coefficient before the composition ratio is determined. Of the plurality of pixels in focus at the same position, the composition ratio of the closer pixel may be appropriately increased so that the farther pixel is not transmitted. In the present embodiment, the image processing unit 107 further increases the weighting coefficient applied to the pixels having close focus positions in the perspective competition region, so that the effect of preventing perspective competition is further enhanced.

As described above, the image processing unit 107 determines the perspective competition and calculates the gain Gain [k]. The gain Gain [k] calculated here is used in step S604.

According to the first embodiment, the edge portion is generated by generating a composite image using the evaluation values integrated by applying a gain to the output filtered with different tap numbers in consideration of perspective competition. It is possible to reduce the degree of blurring and prevent the deterioration of perspective competition.

(Second Embodiment)
In the first embodiment, the process of expanding the contrast to the peripheral pixels is realized by filters having different taps, but in the second embodiment, the contrast is expanded to the peripheral pixels by using a multi-resolution image. Hereinafter, the second embodiment will be described focusing on the points different from those of the first embodiment. The same parts as in the first embodiment will be omitted.

The flow of the second embodiment is the same as that of FIGS. 2 to 5 of the first embodiment. However, the generation of the composite map in step S502 in the second embodiment is different from that in the first embodiment.

FIG. 13 is a flowchart for explaining the generation of the composite map in step S502 in the second embodiment. FIG. 14 is a block diagram for explaining the generation of the composite map in step S502 in the second embodiment.

In step S1301, the image processing unit 107 reduces the contrast value calculated in step S501 by reducing the reduction ratio 1411 (reduction ratio 1 / L [0]) and 1412 (reduction ratio 1 / L [0]). 1]) is performed. The output of the reduction unit 1411 is enlarged by the enlargement unit 1421 (enlargement ratio L [0]), and the output of the reduction unit 1412 is enlarged by the expansion unit 1422 (enlargement ratio L [1]). L [1] indicates that the numerical value is larger than L [0]. Further, although FIG. 14 shows an example of two reduction portions and two enlargement portions, it is assumed that the number of the reduction portions is not limited. Further, the reduction unit 1411 and the expansion unit 1421 may be set to through output (signals having the same input and output). In this way, by using the reciprocals of the reduction ratio and the enlargement ratio, the image can be returned to the original size through one reduction / enlargement process.

By the parallel reduction / enlargement processing described above, the detected contrast value can be expanded to the peripheral region thereof, as in the first embodiment. Therefore, the contrast value of the in-focus image is higher than the contrast value of the pixel in which the edge portion is blurred, and the in-focus image can be easily output.

Nearest neighbor, bilinear, bicubic, etc. can be used for the reduction and enlargement methods, but other methods may be used as long as the calculated contrast can be expanded to the peripheral pixels.

Here, as in the first embodiment, the gain is reduced as the degree of perspective competition is stronger in the region where perspective competition is occurring. This prevents the perspective competition from getting worse in the reduction / enlargement process.

The processes after step S1305 (from the process of applying the gain to the process of generating the composite image) are the same as those of the first embodiment and are omitted.

According to the first embodiment, the degree of blurring of the edge portion is obtained by generating a composite image using the evaluation values integrated by applying gain after processing using the multi-resolution image and considering the perspective competition. And prevent the deterioration of perspective competition.

(Other embodiments)
In the above embodiment, the explanation was given based on a digital camera for individuals, but if it is equipped with a depth synthesis function, it can be used for mobile devices, smartphones, network cameras connected to servers, and the like. It is also possible to apply. Alternatively, a part of the above-mentioned processing may be performed on a mobile device, a smartphone, a network camera connected to a server, or the like.

In the present invention, a program that realizes one or more functions of the above-described embodiment is supplied to a system or device via a network or a recording medium, and one or more processors in the computer of the system or device program. It can also be realized by the process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Digital camera 101 Control unit 102 Drive unit 103 Optical system 104 Imaging unit 105 ROM
106 RAM
107 Image processing unit 108 Display unit 109 Built-in memory 110 Operation unit

Claims (20)

A detection means that detects the contrast values of multiple images,
A determination means for determining the degree of perspective competition based on the contrast values of the plurality of images, and
The first adjustment with respect to the contrast value of the image so that the determination means smoothes the change in the contrast value of the edge portion in each of at least a part of the images of the plurality of images after the determination is made. And the first adjustment means to generate the result of multiple adjustments,
In each of the images having the contrast value after the first adjustment means performs the first adjustment, a gain is calculated so that the degree of perspective competition becomes smaller at a position where the degree of perspective competition is stronger, and the gain is calculated. Is a second adjusting means for making a second adjustment so as to generate an integrated value after applying the results of the plurality of adjustments, and one said integrated value with respect to one said position of one said image. A second adjustment means to generate
A setting means for setting the composition ratio of the plurality of images according to the integrated value of the position corresponding to each of the plurality of images, and
An image processing apparatus comprising a generation means for generating a composite image by synthesizing the plurality of images based on the composite ratio. The composite image generated by the generation means after the first adjusting means performs the first adjustment is a composite image generated by the generation means without the first adjusting means performing the first adjustment. The image processing apparatus according to claim 1, wherein the degree of blurring at the edge portion is weaker than that of the image processing apparatus. The image processing apparatus according to claim 1 or 2, wherein at least a part of the angles of view of the plurality of images overlap. The image according to claim 3, wherein the corresponding position is an overlapping position when the generation means aligns the plurality of images so that the angles of view match for the purpose of compositing. Processing equipment. The determination means according to any one of claims 1 to 4, wherein the determination means determines the degree of perspective competition by comparing the contrast values at the corresponding positions of the plurality of images. Image processing device. According to the determination means, when the contrast values at the corresponding positions of the plurality of images are arranged in the order of the focus position, if the maximum value of the contrast is two or more, perspective competition has occurred at the position. The image processing apparatus according to claim 5, wherein the determination is made. The image processing apparatus according to claim 6, wherein the maximum value has a difference between the maximum value and the contrast value on both sides of the maximum value being larger than a predetermined threshold value. The image processing according to claim 7, wherein the determination means determines the degree of perspective competition based on the difference between the contrast value of the maximum value and the contrast values on both sides of the maximum value. apparatus. The contrast value of the blur of the edge portion of the subject in the image in which the first adjusting means is in focus on the subject after the first adjustment is the contrast value after the first adjustment is performed. The image processing apparatus according to any one of claims 1 to 8, wherein the image processing device according to any one of claims 1 to 8 is characterized in that the contrast value of the position corresponding to the blur of the edge portion of the subject is higher than the contrast value of the image out of focus on the subject. Any of claims 1 to 9, wherein the setting means sets a larger composition ratio for an image having a higher contrast value among the contrast values of the corresponding positions in each of the plurality of images. The image processing apparatus according to item 1. The first adjustment means according to any one of claims 1 to 10, wherein the first adjustment means makes the first adjustment with respect to the contrast value using a plurality of filters having different reference ranges. Image processing device. The image processing apparatus according to claim 11, wherein the second adjusting means calculates the gain applied to the contrast value after applying the filter according to the reference range of the filter. A second reference range in which the gain applied to the contrast value after the filter by the second adjusting means has a first reference range is narrower than that of the first reference range. The image processing apparatus according to claim 12, wherein the gain is larger than the gain applied to the contrast value after applying the filter. The first adjustment means according to any one of claims 1 to 10, wherein the first adjusting means adjusts the contrast value by performing a plurality of parallel reduction / enlargement processing on the contrast value of the image. Image processing device. The image processing apparatus according to claim 14, wherein the reduction ratio and the enlargement ratio used in the plurality of parallel reduction / enlargement processes are reciprocals. 14. The second adjusting means according to claim 14 or 15, wherein the second adjusting means calculates the gain applied to the contrast value after the reduction / enlargement processing is performed according to the enlargement ratio used for the reduction / enlargement processing. The image processing apparatus described. The second adjusting means said gain the subjected to the contrast value of the image having a contrast value of the enlargement process having a first magnification, the second adjusting means is smaller than said first magnification rate the image processing apparatus according to claim 16, wherein greater than the gain applied to the contrast value of the image having a contrast value of the enlargement process with two magnification. An imaging means that captures multiple images and
A detection means for detecting the contrast values of the plurality of images, and
A determination means for determining the degree of perspective competition based on the contrast values of the plurality of images, and
The first adjustment with respect to the contrast value of the image so that the determination means smoothes the change in the contrast value of the edge portion in each of at least a part of the images of the plurality of images after the determination is made. And the first adjustment means to generate the result of multiple adjustments,
In each of the images having the contrast value after the first adjustment means performs the first adjustment, a gain is calculated so that the degree of perspective competition becomes smaller at a position where the degree of perspective competition is stronger, and the gain is calculated. Is a second adjusting means for making a second adjustment so as to generate an integrated value after applying the results of the plurality of adjustments, and one said integrated value with respect to one said position of one said image. A second adjustment means to generate
A setting means for setting the composition ratio of the plurality of images according to the integrated value of the position corresponding to each of the plurality of images, and
An imaging device having a generation means for generating a composite image by synthesizing the plurality of images based on the composite ratio. A detection step that detects the contrast values of multiple images,
A determination step for determining the degree of perspective competition based on the contrast values of the plurality of images, and
The first adjustment with respect to the contrast value of the image so that the change of the contrast value of the edge portion in each of at least a part of the images of the plurality of images after the determination is performed in the determination step is smooth. And the first adjustment step, which produces the results of multiple adjustments,
In each of the images having the contrast value after the first adjustment is performed in the first adjustment step, a gain is calculated so that the degree of perspective competition becomes smaller at a position where the degree of perspective competition is stronger, and the gain is calculated. Is a second adjustment step in which the second adjustment is performed so as to generate an integrated value after applying the results of the plurality of adjustments, and one said integrated value with respect to one said position of one said image. The second adjustment step to generate
A setting step of setting the composition ratio of the plurality of images according to the integrated value of the position corresponding to each of the plurality of images, and
An image processing method comprising a generation step of performing composition on a plurality of images based on the composition ratio to generate a composite image. A computer program that runs on the computer of an image processing device.
A detection step that detects the contrast values of multiple images,
A determination step for determining the degree of perspective competition based on the contrast values of the plurality of images, and
The first adjustment with respect to the contrast value of the image so that the change of the contrast value of the edge portion in each of at least a part of the images of the plurality of images after the determination is performed in the determination step is smooth. And the first adjustment step, which produces the results of multiple adjustments,
In each of the images having the contrast value after the first adjustment is performed in the first adjustment step, a gain is calculated so that the degree of perspective competition becomes smaller at a position where the degree of perspective competition is stronger, and the gain is calculated. Is a second adjustment step in which the second adjustment is performed so as to generate an integrated value after applying the results of the plurality of adjustments, and one said integrated value with respect to one said position of one said image. The second adjustment step to generate
A setting step of setting the composition ratio of the plurality of images according to the integrated value of the position corresponding to each of the plurality of images, and
A computer program that performs a generation step of synthesizing a plurality of images based on the composition ratio and generating a composite image.