JP2017073613A - Image processing apparatus, imaging device, image processing method, program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve image processing which includes filtering for improving the impact of bit rounding relying upon the feature of a subject.SOLUTION: An imaging device, to which an inventive image processing apparatus is applied, acquires the image data of a subject by an imaging unit 105, and performs various imaging processing, including the filtering, for the image data. A reference image determination unit 203 determines a reference image, when performing the filtering, by using the image of the subject. A filtering unit 204 acquires the reference pixel determination results 214, and performs filtering of an input image 211 by using a filter range assigned with a predetermined filter coefficient group. A synthesis unit 205 synthesizes multiple filtering results of bit shift operation or division carried out with multiple set values, with a synthesis ratio according to the number of reference pixels.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus that performs filter processing of image data, and more particularly to a technique for acquiring information related to a distance distribution of a subject as map data.

  There is a technique for obtaining a distance map by correlation calculation of pupil-divided images obtained by receiving light beams that have passed through different pupil regions of the imaging optical system and stereo pair images obtained by photographing with two eyes. Japanese Patent Application Laid-Open No. 2004-228561 discloses a process for matching a distance value corresponding to the contour of a subject to a correct contour using a captured image.

JP 2014-150521 A

In the prior art disclosed in Patent Document 1, the filtering process is performed using only regions having similar subject characteristics during the distance map shaping process as reference pixels. That is, the number of pixels to be referred to in the filter process varies depending on the characteristics of the subject. In order to suppress the shaped distance map signal to a predetermined number of bits, it is necessary to perform bit shift or division. Since the number of signal values to be added differs depending on the characteristics of the subject, areas with large bit rounding and areas with small bit rounding may be scattered, and image quality after filtering may be degraded.
An object of the present invention is to realize image processing including filter processing for improving the influence of bit rounding depending on the characteristics of a subject.

  An image processing apparatus according to an embodiment of the present invention performs filtering using an acquisition unit that acquires an image of a subject, and a filter range in which a filter coefficient group is assigned to the image acquired by the acquisition unit. Filter means for determining, a determination means for determining a reference pixel when performing the filter processing, an arithmetic means for performing bit shift calculation or division on the output of the filter means, and the arithmetic means at different set amounts, respectively Combining means for synthesizing the results of the plurality of filtering processes that have undergone the calculation at a combining ratio corresponding to the number of reference pixels determined by the determining means is provided.

  According to the present invention, it is possible to realize image processing including filter processing that improves the influence of bit rounding depending on the characteristics of a subject.

It is a block diagram which shows the function structure of the digital camera which concerns on embodiment of this invention. It is a block diagram which shows the function structure of the image process part which concerns on 1st Embodiment of this invention. It is a flowchart of the process which concerns on 1st Embodiment. It is a figure for demonstrating operation | movement of the distance map shaping part which concerns on embodiment of this invention. It is a figure for demonstrating the distance map synthetic | combination process which concerns on embodiment of this invention. It is a block diagram which shows the function structure of the image process part which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the process which concerns on 2nd Embodiment. It is a figure for demonstrating the division | segmentation of the filter coefficient which concerns on 2nd Embodiment. It is explanatory drawing of the correspondence at the time of applying the division | segmentation filter which concerns on 2nd Embodiment to a distance image. It is a block diagram which shows the division | segmentation filter processing function which concerns on 2nd Embodiment.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the drawings. In each embodiment, a digital camera will be described as an application example of the image processing apparatus of the present invention.

[First embodiment]
FIG. 1 is a block diagram illustrating the functional configuration of a digital camera 100 according to an embodiment of the invention. 1 can transmit and receive signals to and from each other via a bus 111.
The system control unit 101 includes a CPU (Central Processing Unit). The system control unit 101 reads a program for controlling the operation and processing of each component of the digital camera 100 from a ROM (Read Only Memory) 102, expands it in a RAM (Random Access Memory) 103, and executes it. The ROM 102 is a non-volatile memory in which data can be rewritten, and stores parameters necessary for the operation of each component in addition to a program for controlling the operation and processing of the digital camera 100. For example, information such as an exit pupil distance necessary for focus detection and the like is stored in the ROM 102. The RAM 103 is a volatile memory capable of rewriting data, and is used as a temporary storage area for data output in the processing of the digital camera 100.

  The imaging optical system 104 forms an image of light from the subject on the imaging unit 105. The imaging optical system 104 includes a lens and a diaphragm, and the diaphragm adjusts the aperture diameter of the imaging optical system 104. The imaging unit 105 includes an imaging element such as a CCD (charge coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor. The imaging element photoelectrically converts the optical image formed by the imaging optical system 104 and outputs the obtained analog image signal to the A (analog) / D (digital) conversion unit 106. The A / D converter 106 performs A / D conversion processing on the input analog image signal, and outputs the digital image data to the RAM 103 for storage.

  The image processing unit 107 performs processes such as white balance adjustment, color interpolation, reduction / enlargement, and filtering on the image data stored in the RAM 103. The recording medium 108 is a removable memory card or the like, and data such as an image processed by the image processing unit 107 and an image A / D converted by the A / D conversion unit 106 stored in the RAM 103 are recorded images. Recorded as data. The display unit 109 includes a display device such as an LCD (Liquid Crystal Display). The display unit 109 displays various information such as through display of the subject image based on the imaging data acquired by the imaging unit 105.

  The distance map acquisition unit 110 is a distance sensor module that acquires object distance information by a predetermined detection method. For example, the distance map acquisition unit 110 acquires information related to the distance distribution of the subject by the “Time of Flight (TOF)” method or the like. In the TOF method, the light emitted from the light source is reflected by the object, and the distance information to the subject is acquired based on the time and light speed until the detection unit receives the light. The distance map acquisition unit 110 outputs distance map data having a predetermined resolution to the RAM 103 as digital data and stores it.

FIG. 2 is a diagram for explaining the configuration of the image processing unit 107 in FIG.
The developing unit 201 illustrated in FIG. 2A performs development processing including white balance adjustment and color interpolation on the digital image data after A / D conversion input as the captured image input 210, and develops the image 212. Is generated. The developed image 212 is input to the distance map shaping unit 200 and used as a distance map shaping image.

As shown in FIG. 2B, the distance map shaping unit 200 includes a pixel value comparison unit 202, a reference pixel determination unit 203, a filter processing unit 204, and a synthesis unit 205.
The pixel value comparison unit 202 compares the pixel values of the developed image 212 (hereinafter referred to as a shaping image) corresponding to the target pixel position of the distance map image 211 and the peripheral pixel position, and outputs a comparison result 213. The reference pixel determination unit 203 determines the comparison result 213 with a predetermined threshold value, and outputs a reference pixel determination result 214 indicating whether or not the reference pixel position is set. For example, the reference pixel determination result 214 is binary data, and takes a value of “1” or “0” depending on whether the comparison result 213 is equal to or less than a threshold value. The determination value “1” corresponding to the pixel position means that it has been determined as a reference pixel, and the determination value “0” means that it has not been determined as a reference pixel.

The filter processing unit 204 performs bilateral filter processing on the distance map image 211 based on the reference pixel determination result 214. The filter processing unit 204 performs a bit shift operation at the time of filter processing, and outputs a plurality of filter processing results 215 having different bit shift amounts. The combining unit 205 combines the filter processing results 215 having different bit shift amounts, and outputs the combined output as a combined distance map image 216.
In the basic shaping process, the filter processing result (denoted as Jp) at the target pixel position p is expressed by the following equation (1).
Jp = (1 / Kp) ΣI1q · f (| p−q |) · g (| I2p−I2q |) (1)

The meaning of each symbol in Formula (1) is as follows.
q: peripheral pixel position Ω: integration target area centered on the target pixel position p Σ: integration in the q∈Ω range I1q: distance map signal value at the peripheral pixel position q f (| p−q |): target pixel position p Gaussian function centered at I2p: pixel value of the shaping image at the target pixel position p I2q: pixel value of the shaping image at the peripheral pixel position q g (| I2p−I2q |): pixel value I2p of the shaping image Gaussian function centered on Kp: normalization coefficient, integrated value of f · g weights.

When the difference between I2p at the target pixel position p and I2q at the peripheral pixel position q is small, that is, when the pixel values of the target pixel and the peripheral pixel are close in the shaping image, the f · g weight of the peripheral pixel increases. In the present embodiment, taking into account the circuit scale, the functions f and g are not Gaussian functions but functions that make the weighting coefficients in a predetermined pixel range uniform. Accordingly, a function that simply adds pixel values (pixel information) when | p−q | and | I2p−I2q | For example, the functions f and g are expressed by the following equations.
f = α (| pq | <th1), f = 0 (| pq |> = th1) (2)
g = β (| I2p−I2q | <th2), g = 0 (| I2p−I2q |> = th2) (3)
Α in Equation (2) and β in Equation (3) each represent a constant, and th1 in Equation (2) and th2 in Equation (3) each represent a threshold value. In this embodiment, α = 1 and β = 1.

  Next, the overall processing of the digital camera 100 will be described with reference to the flowchart of FIG. The processing in FIG. 3 is realized by the CPU of the system control unit 101 reading out the control program from the memory and executing it. For example, the processing is started when an instruction to shoot is given by the digital camera 100.

  In S301 of FIG. 3, the imaging unit 105 performs imaging and acquires a distance map shaping image. Next, in S302, the distance map acquisition unit 110 acquires a distance map image corresponding to the shaping image. In S303, the filter processing unit 204 performs the filtering process of Expression (1) on the distance map image acquired in S302 using the pixel information of the shaping image, and the shaping process of the distance map image is performed. Regarding the output result after the filter processing, the number of bits for storing the output value after the filter processing becomes enormous as the number of filter taps (reference pixel range) increases. Therefore, a bit shift operation or division by a predetermined set amount is performed on the numerator of the filter processing result of Expression (1) so that the output signal fits in the storable bit number. For example, in the case of bit shift calculation, the first set amount is described as “bit shift amount A”.

In S304, a distance map image shaping process is performed as in S303. However, in the shaping process of S304, the shaping process is performed by setting a bit shift amount B smaller than the bit shift amount A. The bit shift amount B is a second set amount in the bit shift calculation. In next step S305, the synthesizing unit 205 performs a synthesizing process of the distance map image after the shaping process, which is each output of S303 and S304. In the combining process, a ratio (compositing ratio) for combining two distance map images after the shaping process is determined using a map (reference pixel number map) that records the number of reference pixels at the time of the shaping process in S304.
As described above, the process of synthesizing the distance map that has been subjected to the shaping process by setting a plurality of bit shift amounts is executed.

  Next, each process important in the present embodiment will be described. First, the shaping process will be described. FIG. 4 is a diagram for explaining the operation of the distance map shaping unit 200 of FIG. 4A-1 and 4B-1, the left-right direction is defined as the x direction and the right direction is defined as the + x direction in the drawing. In the drawing, the vertical direction perpendicular to the x direction is defined as the y direction, and the upward direction is defined as the + y direction.

  FIG. 4A-1 illustrates the shaping image 401, and the subject is a person on the right side of the screen and a background. FIG. 4A-2 illustrates a pixel value profile 403 in a cross section at the position indicated by the alternate long and short dash line 402 in FIG. The horizontal axis represents the x coordinate, and the vertical axis represents the pixel value. The shape of the pixel value profile 403 is a step shape that greatly changes (decreases) at the position of xs in the increasing direction of the x coordinate. The pixel value profile 403 maintains a sharp edge at the contour of the person.

  FIG. 4B-1 illustrates the distance map 404. A portion indicated by a dotted line in FIG. 4B-1 represents an outline of a person in the shaping image 401 illustrated in FIG. 4B-2 is a distance value profile in a cross section at the position indicated by the alternate long and short dash line 405 in FIG. 4A-2 (corresponding to the position indicated by the alternate long and short dash line 402 in FIG. 4A-1). 406 is represented by a solid line. The horizontal axis represents the x coordinate, and the vertical axis represents the distance value. As for the distance value, it is assumed that the distance value of the background far from the camera position is small and the distance value of the person located nearby is large. In the case of FIG. 4 (B-1), the outline of the person in the distance map protrudes outside the outline of the person in the shaping image that is the correct answer (see the dotted line). The shape of the distance value profile 406 indicated by a solid line in FIG. 4B-2 is a step shape that greatly changes (increases) at a position xa smaller than xs in the increasing direction of the x coordinate.

  The target pixel position p in Expression (1) is indicated by black points 408, 410, 412, and 414, respectively. Line segments 409, 411, 413, and 415 indicate regions where the g value of Equation (1) is large, that is, the smoothing range. The pixel value profile 403 of the shaping image changes sharply at a position xs corresponding to the contour of the person. For this reason, at the positions of the black dots 410 and 412 indicating the pixel positions of interest in the vicinity of the contour of the person, the smoothing ranges become line segments 411 and 413, respectively, so as to follow the person contour of the correct shaping image. Become. As a result, when the value of the filter result Jp shown in Expression (1) is plotted, a graph line 407 indicated by a dotted line is obtained. The shape of the graph line 407 is a step shape that greatly changes (increases) at the position of xs in the increasing direction of the x coordinate. That is, by the distance map shaping process, the person contour of the distance map matches the correct contour (the person contour of the shaping image).

  Next, the bit shift amounts A and B used for the shaping process shown in S303 and S304 of FIG. 3 will be described. In the present embodiment, equations (2) and (3) are set for equation (1). Further, distance weights at the target pixel position p and the reference pixel position of the distance map image and weights based on the pixel values of the distance map image are set to constant values α and β (for example, α = β = 1). In the distance map image that holds the distance information as image data, if the resolution of the distance information decreases due to the contrast of the subject, the brightness of the subject, the occlusion of the subject, or the settings when shooting with the camera, the correct distance information is displayed. You may not be able to get it. Therefore, the reliability (hereinafter referred to as h (q)) as to whether the distance information is correctly output is newly set as a parameter to be added to the equation (1). Thereby, the shaping process using the distance information of the reference pixel position holding the correct distance information can be performed.

The reliability in the DFD (Depth From Defocus) method for acquiring distance information using the first and second images will be described. The first image is a focus image that is focused on the subject, and the second image is a defocus image that differs from the focus image in how the subject image is blurred. In the DFD method, distance information is calculated using the blur amount of the focus image and the blur amount of the defocus image. For this reason, correct distance information cannot be calculated in a region where the difference in blur between the subject images cannot be detected between the focus image and the defocus image. That is, such a region is a region with low reliability of distance information. The reliability is an index representing the ease of detecting a difference in blur, and can be determined based on the luminance value, saturation value, hue, frequency characteristic of the target region, and the like in the target region. Specifically, the following areas are determined to be areas with low reliability.
Example 1: When the average value of the luminance values of the pixels in the area is out of the predetermined range.
Example 2: When the amount of change in luminance value of pixels in the area is greater than a predetermined threshold.
Example 3: When the spatial frequency in the region is lower than a predetermined threshold.

  A plurality of criteria can be adopted as the evaluation criteria for determining the reliability h (q). Evaluation criteria other than those exemplified may be used. In any case, the value of the reliability h (q) can be calculated by comparing the evaluation value with a predetermined value (threshold value). The reliability h (q) of the present embodiment is expressed as a binary value of “1” or “0” with respect to the evaluation criterion in reliability considering the circuit scale. “1” means reliable distance information, and “0” means unreliable distance information.

In consideration of the above, the formulas (1) to (3) can be replaced with the following formulas (4) to (6).
J ^* p = ΣI1q · f (| pq |) · g (| I2p−I2q |) · h (q)
/ Σf (| p−q |) · g (| I2p−I2q |) (4)
f = 1 (| pq | <th1), f = 0 (| pq |> = th1) (5)
g = 1 (| I2p−I2q | <th2), g = 0 (| I2p−I2q |> = th2) (6)

Thus, molecules of J ^* p is represented by the formula obtained by multiplying the h (g) in the molecule of Jp, the denominator of J ^* p corresponds to the sum of the number of reference pixels. As the number of bits of the numerator of J ^* p, a sufficient number of bits is required to store the signal value of the product of the maximum value of input pixels and the maximum number of reference pixels. However, in the case of a small device, the number of bits of the numerator of J ^* p is limited. In other words, a bit shift operation is performed in order to realize the J ^* p operation processing with a limited number of bits, and processing for limiting the number of output bits is performed. Taking the bit shift operation into account, Equation (4) becomes Equation (7) below.
J ^** p = Σ {I1q · f (| pq |) · g (| I2p−I2q |) >> bit shift amount}
/ {Σf {(| p−q |) · g (| I2p−I2q |)} >> bit shift amount} ·· (7)

The calculation of the bit shift amount will be specifically described. For example, the number of taps of the filter is 63 × 63, and the input signal value of the distance map image is 8 bits. For the numerator of J ^** p, the maximum number of bits is 14 bits. That is, a right bit shift for 6 bits is performed, and a process of suppressing the output pixel value to 14 bits is performed. “Bit shift amount A = number of input signal bits (8 bits) + bit number of maximum filter reference pixel number (63 × 63≈12 bits) −number of output signal bits (14 bits) = 6 bits” Is decided. The bit shift amount A in S303 of FIG. 3 is set.

On the contrary, when the shaping process of Expression (7) is performed using the bit shift amount A, the influence of bit rounding may be increased depending on the number of reference pixels. For example, when there are only two reference pixels, the numerator of J ^** p is a signal that fits in 9 bits (input signal bit number (8 bits) + reference pixel bit number (2 pixels = 1 bit)). Value. 9 bits = number of input signal bits (8 bits) + number of bits of reference pixels (2 pixels = 1 bit), but when the bit shift amount A is 6 bits, the signal value of the numerator is converted to 3 bits. It may be rounded and affect the calculation accuracy. Therefore, when the number of reference pixels is less than a predetermined threshold, a bit shift amount B different from the bit shift amount A is set in order to reduce the influence of bit rounding. In order to reduce the influence of bit rounding, for example, the bit shift amount B in S304 of FIG. 3 is set to 2 bits.

Next, the composition processing in S305 of FIG. 3 will be described with reference to FIG. In S303 and S304 in FIG. 3, a process of combining distance maps shaped using different bit shift amounts A and B is executed. FIG. 5 shows the composition ratio of the shaping processing result with the bit shift amount B (see FIG. 3: S304) at this time. The horizontal axis indicates the number of reference pixels at the time of the shaping process with the bit shift amount B when the filter process of Expression (7) is performed on a certain target pixel. In addition, the vertical axis indicates the composition ratio of the distance map after the shaping process in S304 of FIG. The combined distance map is expressed by the following formula.
Post-combination distance map = (1-c) · Post-shaping distance map (bit shift amount A) + c · Post-shaping distance map (bit shift amount B) (8)

  In the equation (8), “c” represents a composition ratio of the distance map after the shaping process using the bit shift amount B. The c value is calculated from the reference pixel map at the time of filter processing with the bit shift amount B.

  As shown in FIG. 5, in the region 501 where the number of reference pixels is small, the value of the synthesis ratio is set to be relatively large, and a distance map after shaping processing with a bit shift amount A that is less affected by bit rounding is output. On the other hand, in the region 503 where the number of reference pixels is large, the value of the combination ratio is set to be relatively small, and a distance map after shaping processing with the bit shift amount B is output to suppress the number of output bits. An area 502 is an area where two distance maps after shaping processing are blended, and has a characteristic that the value of the composition ratio gradually decreases as the number of reference pixels increases.

The region 502 needs to be set with a reference pixel number (saturation pixel number 511) that does not exceed at least the number of output bits with the bit shift amount B. In other words, when the number of saturated pixels 511 is exceeded in the region 502, the number of output bits is reached, so that a correct distance map may not be output. The number of saturated pixels 511 can be calculated by the following equation (9).
Number of saturated pixels = 2 ^ (bit shift amount A−bit shift amount B) −1 (9)
“^” Is a symbol representing a power.
According to the present embodiment, even when the number of reference pixels is small at the time of filter processing, it is possible to improve the influence of bit rounding and obtain a good high-resolution distance map.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the present embodiment, the filtering process used for the shaping process of the low-resolution distance map image is divided. When the number of filter taps in the shaping process (the range in which the filter is applied at a time) increases, it becomes necessary to store pixel data corresponding to the reference area for performing the filter process in the memory. As the number of filter taps increases, the circuit scale of the image processing apparatus increases. Therefore, in this embodiment, the shaping process is performed by suppressing the increase in circuit scale by dividing the number of filter taps and integrating the filter processing results.
Regarding the system control unit 101, the ROM 102, the RAM 103, the imaging optical system 104, the imaging unit 105, the A / D conversion unit 106, the recording medium 108, and the display unit 109 of FIG. 1, which are the same components as in the first embodiment. Description is omitted.

  The image processing unit 601 in FIG. 1 applies white balance adjustment, color interpolation, and various image processing to the image data stored in the RAM 103. The configuration of the image processing unit 601 will be described with reference to FIG. The developing unit 201, the pixel value comparing unit 202, the reference pixel determining unit 203, the synthesizing unit 205, the captured image input 210, the distance map image 211, the developed image 212, the comparison result 213, and the reference pixel determining result 214 are the first implementation. The form is as described in FIG. Below, it demonstrates centering around difference.

  The distance map shaping unit 700 includes a divided filter processing unit 701 and a filter result adding unit 702. The division filter processing unit 701 divides the number of filter taps used in the shaping process, and performs a convolution process on the distance map image 211 with the divided filter taps. The division filter processing unit 701 outputs a plurality of division filter processing results 711. At this time, the division filter processing is executed with different bit shift amounts. The filter result adding unit 702 acquires and synthesizes a plurality of divided filter processing results 711 and outputs a divided filter synthesis result 712. At this time, a process of combining the divided filter processing results 711 having the same bit shift amount is performed, and divided filter combined results having different bit shift amounts are obtained.

  With reference to the flowchart of FIG. 7, the overall processing of the digital camera of this embodiment will be described. The processing shown in FIG. 7 is realized by the control of the system control unit 101. Note that the processing of S301 and S302 in FIG. 7 is the same as each step in FIG.

  In S801, the distance map image acquired in S302 is subjected to the filtering process of Expression (4) using the image data of the shaping image, and the shaping process of the distance map image is performed. The filter coefficient used in the shaping process is divided so as to have a predetermined number of taps. Processing is performed using each filter coefficient after division. When the convolution processing of the filter coefficient and the distance map image is performed, a bit shift operation is also performed at the same time (the bit shift amount is set as the bit shift amount A). At the same time, each reference pixel number map is generated. The reference pixel number map is map data including data obtained by counting the number of pixels referred to when Expression (4) is performed by the division filter.

  In step S <b> 802, the divided filter processing unit 701 divides and executes the number of filter taps regarding the filter processing of Expression (4). The bit shift amount at this time is defined as a bit shift amount B. It is assumed that the relationship of “bit shift amount A> bit shift amount B” is satisfied. In step S803, the filter result addition unit 702 combines the results obtained by performing the division filter processing of the distance map image in steps S801 and S802. The composite result is a distance map image after shaping with different bit shift amounts. In addition, the filter result adding unit 702 similarly synthesizes the reference pixel number maps of the bit shift amount B, and calculates as an integrated reference pixel number map.

  In step S804, the synthesis unit 205 determines a saturation reference pixel number according to the integrated reference pixel number map generated in step S803, and performs threshold processing for calculating a synthesis ratio in a range that is equal to or less than the saturation reference pixel number. Then, the combining unit 205 combines the shaped distance map images with different bit shift amounts according to the integrated reference pixel number map, and outputs the final shaped distance map image.

  Hereinafter, each process important in the present embodiment will be described. First, the filter dividing process performed in S801 and S802 of FIG. 7 will be described. FIG. 8 is an explanatory diagram of filter coefficients related to the two-dimensional filter processing applied to the distance map image. The shape of the filter is a square, and the number of filter taps is (54 taps) × (54 taps). Filters obtained by dividing the entire filter coefficient are denoted by f1, f2,..., F36. Each filter is (9 taps) × (9 taps). By arranging 6 blocks in the vertical direction and 6 blocks in the horizontal direction, an overall filter coefficient of (54 taps) × (54 taps) is realized. The

  FIG. 9 is a diagram for explaining a state in which the two-dimensional filter of FIG. 8 is applied to the distance map image. In the acquired distance map image 1001, a region 1002 is a region of an output image after applying a (54 tap) × (54 tap) two-dimensional filter.

  FIG. 10 is a diagram for explaining the configuration of the image processing unit 601. A two-dimensional filter circuit 1101 in FIG. 10A is a circuit that performs a two-dimensional filter process of (9 taps) × (9 taps). The input to the two-dimensional filter circuit 1101 is a two-dimensional filter coefficient input 1103 and an image input 1104, and the output is a two-dimensional filter output 1105. The two-dimensional filter circuit 1101 includes a position shifting unit 1101a and a two-dimensional convolution processing unit 1101b. The two-dimensional filter circuit 1101 reads a desired area of the input image acquired from the image input 1104 and performs a two-dimensional convolution process. That is, the position shifting unit 1101a executes position shifting processing on the image data from the image input 1104, and outputs the processed image data to the two-dimensional convolution processing unit 1101b. The two-dimensional convolution processing unit 1101b acquires coefficient value data from the two-dimensional filter coefficient input 1103, and performs two-dimensional convolution processing on the data in the specific area related to the input image.

  First, the two-dimensional filter circuit 1101 applies the filter coefficient f1 of FIG. 8 to the input image. At this time, the intermediate image 1 is generated by partially reading out the region 1003 (FIG. 9) of the input image and applying the filtering process. Similarly, the intermediate image 2 is generated by filtering the region 1004 with the filter coefficient f2 of FIG. In the filter processing at this time, Expression (7) is applied to each intermediate image. At the same time, a reference pixel number map for each intermediate image for normalization processing is also output. Thereafter, the filter processing is similarly performed for the filter coefficients f3 to f36 in FIG.

  The same bit shift amount is applied during the division filter processing of the filter coefficients f1 to f36. As a method for determining the bit shift amount, for example, a bit shift amount A that does not exceed the maximum number of output bits when the intermediate images are added in the subsequent processing is set. Specifically, “bit shift amount A = number of input signal bits (8 bits) + bit number of maximum filter reference pixel number (9 × 9≈7 bits) + number of intermediate images (6 × 6≈6 bits) − The number of output signal bits (14 bits) = 7 bits ”. Further, a bit shift amount B smaller than the bit shift amount A is set to prevent the influence of bit rounding.

  With reference to FIG. 10B, the processing of S803 in FIG. 7 will be described. The image addition circuit 1106 receives the intermediate image 1 and the intermediate image 2 from the image inputs 1107 and 1108, respectively. The image addition circuit 1106 generates the accumulated image 2 and outputs it as an image output 1109. Similarly, the image addition circuit 1106 accumulates the intermediate images 3 to 36 and accumulates all the intermediate images. At this time, integration is also performed for each intermediate image of the reference pixel number map output in S802 of FIG. 7, and an integrated reference pixel number map is calculated.

  According to the present embodiment, the filter range to which the predetermined filter coefficient group is assigned is divided into smaller ranges (sub-filter ranges), and the results of the divided filter processing are integrated, thereby suppressing an increase in circuit scale. A shaping process can be realized.

As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.
[Modified Embodiment]
In the first embodiment and the second embodiment, the filter has a square shape, but may be applied to a cylindrical shape. Regarding the bit shift amount A at that time, the number of reference pixels in the case of a square is set to the maximum number of filter reference pixels. Even when the weighting coefficient is changed for each distance without uniformly setting the filter coefficient to 1, when the bit shift amount A is obtained, it is set as a uniform coefficient.

  In the first embodiment and the second embodiment, the synthesis process is performed according to the number of reference pixels subjected to the shaping filter processing with the bit shift amount B for the distance map images after shaping with the bit shift amounts A and B. Was executed. However, the synthesis process may be executed according to the number of reference pixels of the bit shift amount A.

In the first embodiment and the second embodiment, the shaping process of the distance map image has been described as an example, but it can be implemented as a filter process for a general image. In that case, the filtering process is executed by an expression excluding h (q) which is the reliability of the distance information. That is, as shown in the following equation (10), a process of determining a pixel position to be referred to in the filter process is performed based on a difference in signal value between a target pixel and a peripheral pixel in the subject image.
J ^* p = ΣI1q · f (| pq |) · g (| I2p−I2q |)
/ Σf (| p−q |) · g (| I2p−I2q |) (10)

In the first and second embodiments, the bit shift amount is set as a method of changing the bit depth used for the shaping process, and the process of suppressing the number of output bits by performing the bit shift operation is performed. It is also possible to perform division processing instead of bit shift. By using the division process, the number of saturated pixels can be calculated more accurately, and the accuracy can be increased when calculating the composite ratio of the distance map after shaping. Using the division amount A and the division amount B instead of the bit shift amount A and the bit shift amount B, the number of saturated pixels can be calculated by the following equation, for example.
Division amount A = maximum input signal value × number of maximum filter reference pixels × number of filter divisions (number of intermediate images) −maximum output signal value (11)
Number of saturated pixels = divided amount A / divided amount B (12)
During each filter process, a division process using the division amount A and the division amount B is executed. The division amount B is set to a value that is less affected when the number of reference pixels is small.

  The present invention uses an embodiment in which defocus map data is calculated and used as information related to the distance distribution of the subject, and an embodiment in which distance map data in which the defocus amount is converted into the subject distance (distance in the depth direction of the subject) is used. Can be applied. That is, information (distance information) indicated by data related to the distance distribution of the subject directly represents the distance value of the subject in the image or indirectly represents information corresponding to the distance value.

  Further, in the application example to the parallax ranging method, a process of calculating distance information based on parallax images acquired under different imaging conditions is executed. Specifically, the imaging device of the imaging unit 105 is a pupil division type imaging device, and has a plurality of pixel units arranged in a regular array in a two-dimensional array. One pixel unit includes a microlens and a plurality of photoelectric conversion units. A pair of light beams passing through different pupil regions of the imaging optical system 104 can be formed as optical images, respectively, and a pair of image data can be output from a plurality of photoelectric conversion units. The defocus amount is calculated by the correlation calculation between the paired images, and a defocus map representing the distribution of the defocus amount (distribution on the two-dimensional plane of the captured image) is generated. In this case, the distance map acquisition unit 110 acquires defocus map data or distance map data of the subject distance converted from the defocus amount.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100: Digital camera 101: System control part 104: Imaging optical system 105: Imaging part 107,601: Image processing part 108: Recording medium 110: Distance map acquisition part 200,700: Distance map shaping part

Claims (14)

An acquisition means for acquiring an image of a subject;
Filter means for performing filter processing on the image acquired by the acquisition means using a filter range to which a filter coefficient group is assigned;
Determining means for determining a reference pixel when performing the filtering process;
Arithmetic means for performing bit shift operation or division on the output of the filter means;
The image processing apparatus includes a combining unit that combines a plurality of the filter processing results calculated by the calculating unit with different set amounts at a combining ratio corresponding to the number of reference pixels determined by the determining unit. Image processing device.   The image processing apparatus according to claim 1, wherein the filter unit divides the filter range into a plurality of parts and executes a filter process.   The image processing apparatus according to claim 1, wherein the determination unit determines the reference pixel by acquiring and comparing pixel information at a target pixel position and a peripheral pixel position of the image. A shaping means for shaping data related to the distance distribution of the subject is provided,
The image processing apparatus according to any one of claims 1 to 3, wherein the filter unit included in the shaping unit performs a filter process on the data. Calculating means for calculating reliability of data related to the distance distribution of the subject;
The image processing apparatus according to claim 4, wherein the determination unit determines the reference pixel using the image of the subject and the reliability calculated by the calculation unit.   6. The calculation unit according to claim 1, wherein the arithmetic unit performs a bit shift operation using a first set amount and a second set amount smaller than the first set amount. An image processing apparatus according to 1.   The image processing apparatus according to claim 6, wherein the synthesizing unit determines the synthesis ratio with respect to a result of the filter process calculated using the second set amount.   The image processing apparatus according to claim 7, wherein the combination ratio is relatively large when the number of the reference pixels is small and relatively small when the number of the reference pixels is large. A generator for generating map data representing the number of the reference pixels;
The image processing apparatus according to claim 7 or 8, wherein the synthesizing unit acquires the map data and determines a saturation reference pixel number and the synthesis ratio.   The combination ratio is such that the number of reference pixels exceeds the number of saturated reference pixels, compared to the case where the number of reference pixels referenced in the filtering process with the second set amount exceeds the number of saturated reference pixels. The image processing apparatus according to claim 9, wherein the image processing apparatus is larger when there is no image. The image processing apparatus according to any one of claims 1 to 10,
An image pickup apparatus comprising image pickup means for picking up an image of a subject.   The imaging apparatus according to claim 11, wherein the acquisition unit acquires a plurality of images having different ways of blurring from the imaging unit. An image processing method executed by an image processing apparatus that acquires an image of a subject and performs image processing,
Determining a reference pixel when performing filtering by the filter means;
The filter means performing a filtering process on the image using a filter range to which a filter coefficient group is assigned;
A step of performing a bit shift operation or division on the output of the filter means;
An image processing method comprising: combining a plurality of the filter processing results respectively calculated by the calculation means with different set amounts at a combination ratio corresponding to the determined number of reference pixels. A program for causing a computer of an image processing apparatus to execute each step according to claim 13.

Images