JP7009253B2 - Image processing equipment, image processing methods and programs - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method and a program.

In order to correct camera shake for images taken with an image pickup device such as a digital still camera or digital video camera, the amount of movement between frame images is detected and alignment is performed for multiple images. There is a need. As a method of detecting the amount of movement between frame images, there are a method of using information of an external device such as a gyro sensor, a method of estimating the amount of movement from a captured frame image, and the like.

Various methods for estimating the amount of motion using a frame image have been proposed, and a typical example thereof is motion vector detection by template matching. In template matching, first, one of two frame images in the video is used as a reference image, and the other is used as a reference image. Then, a rectangular area of a predetermined size arranged on the reference image is used as a template block, and the correlation with the distribution of pixel values in the template block is obtained at each position of the reference image. At this time, the position where the correlation is highest in the reference image is the movement destination of the template block, and the direction and movement amount to the movement destination when the position of the template block on the reference image is used as a reference becomes the motion vector. .. At this time, a method of improving the accuracy of template matching by performing preprocessing on the input image is known.

In Patent Document 1, as a preprocessing, a plurality of hierarchical images having different resolutions are generated from the input image, a motion vector is detected from the lowest resolution hierarchical image, and a high resolution is obtained while referring to the motion vector. A technique for detecting a motion vector into a hierarchical image is disclosed. At this time, the reliability is determined for each motion vector in each layer, it is determined whether or not the motion vector is detected in the high-resolution layered image, and the processing time is improved.

Japanese Unexamined Patent Publication No. 2010-288110

However, as in the technique disclosed in Patent Document 1, for a motion vector having low reliability in a low-resolution hierarchical image, simply using a high-resolution hierarchical image and detecting the motion vector is not enough to detect the motion vector. Reliability may not improve. For example, when a motion vector with low reliability is detected due to the texture of a repeating pattern such as a grid, a motion vector with lower reliability can be easily detected by making a high-resolution hierarchical image. In addition, when the reliability is low due to noise or when the reliability is low due to the texture of the edge pattern, the reliability may not be improved only by increasing the resolution of the image. As described above, depending on the cause of the low reliability, it may not be possible to improve the reliability simply by increasing the resolution of the image.

An object of the present invention is to be able to improve the detection accuracy of a motion vector.

The image processing apparatus of the present invention includes a first image processing means for performing a first image processing on a first image and a second image, and a first image on which the first image processing is performed. A first motion vector detecting means for detecting a first motion vector, a first reliability calculating means for calculating the reliability of the first motion vector, and the first Based on the reliability of the motion vector of 1, the second image processing means for performing the second image processing on the first image and the second image, and the second image processing are performed. A second motion vector detecting means for detecting a second motion vector and a second reliability calculation for calculating the reliability of the second motion vector based on the first image and the second image. A third motion vector is obtained based on the means, the first motion vector, the second motion vector, the reliability of the first motion vector, and the reliability of the second motion vector. It has a motion vector determining means for determining.

According to the present invention, the accuracy of motion vector detection can be improved.

It is a block diagram which shows the structural example of the image processing apparatus by 1st Embodiment. It is a flowchart for demonstrating image processing. It is a figure for demonstrating template matching. It is a figure for demonstrating a correlation value map. It is a figure for demonstrating the expression method of the correlation value map. It is a figure for demonstrating the correlation value index which shows the reliability. It is a figure for demonstrating edge pattern determination using a correlation value map. It is a figure for demonstrating the calculation method of the reliability. It is a block diagram which shows the structural example of the image processing apparatus by 2nd Embodiment. It is a flowchart for demonstrating image processing. It is a figure for demonstrating a luminance value and an image characteristic index. It is a figure for demonstrating edge pattern determination using a luminance value. It is a block diagram which shows the structural example of the image processing apparatus according to 3rd Embodiment. It is a flowchart for demonstrating image processing. It is a figure which shows the combination of reliability and image processing, and image characteristic and image processing.

(First Embodiment)
FIG. 1 is a block diagram showing a configuration example of the image processing apparatus 100 according to the first embodiment of the present invention. The image processing device 100 includes an image input unit 101, an image memory 102, a first image processing unit 103, a first motion vector detection unit 104, and a first reliability determination unit 105. Further, the image processing apparatus 100 includes a second image processing unit 106, a second motion vector detection unit 107, a second reliability determination unit 108, and a motion vector determination unit 109.

The image input unit 101 inputs images of a plurality of frames in order. The image memory 102 temporarily stores an image of one frame or a plurality of frames input by the image input unit 101. The first motion vector detection unit 104 and the second motion vector detection unit 107 use the image input by the image input unit 101 as a reference image, the image stored in the image memory 102 as a reference image, and the current frame image. Detects motion vectors from to past frame images. The reference image and the reference image may be interchanged. In that case, the first motion vector detection unit 104 and the second motion vector detection unit 107 move vectors from the past frame image to the current frame image. Is detected.

The first image processing unit 103 performs image processing on the reference image input by the image input unit 101 and the reference image stored in the image memory 102. The first motion vector detection unit 104 uses, for example, template matching to obtain a motion vector based on the reference image processed by the first image processing unit 103 and the reference image processed by the first image processing unit 103. To detect. The first reliability determination unit 105 is the first reliability calculation unit, and calculates the reliability of the motion vector detected by the first motion vector detection unit 104.

The second image processing unit 106 is based on the reliability calculated by the first reliability determination unit 105 with respect to the reference image input by the image input unit 101 and the reference image stored in the image memory 102. And perform image processing. The second motion vector detection unit 107 obtains a motion vector by, for example, template matching, based on the reference image processed by the second image processing unit 106 and the reference image processed by the second image processing unit 106. To detect. The second reliability determination unit 108 is a second reliability calculation unit, and calculates the reliability of the motion vector detected by the second motion vector detection unit 107.

The motion vector determination unit 109 includes a motion vector detected by the first motion vector detection unit 104 and the second motion vector detection unit 107, and the first reliability determination unit 105 and the second reliability determination unit 108. The motion vector of the image is determined based on the calculated reliability.

FIG. 2 is a flowchart for explaining an image processing method of the image processing apparatus 100. In step S201, the image input unit 101 sequentially inputs captured images of a plurality of frames. The image memory 102 sequentially stores images of a plurality of frames input by the image input unit 101. In the present embodiment, the image stored in the image memory 102 is used as a reference image, and the image before being stored in the image memory 102 is used as a reference image.

In step S202, the first image processing unit 103 performs the first image processing on the reference image input by the image input unit 101 and the reference image stored in the image memory 102. In template matching, noise and DC components become disturbances, so the first image processing unit 103 applies a bandpass filter to the reference image and the reference image to simultaneously perform noise removal and image enhancement.

In step S203, the first motion vector detection unit 104 calculates the motion vector between the reference image and the reference image processed by the first image processing unit 103 by template matching.

3A and 3B are diagrams for explaining template matching. FIG. 3 (a) shows a reference image, and FIG. 3 (b) shows a reference image. The first motion vector detection unit 104 arranges the template area 301 in the reference image of FIG. 3 (a), arranges the search area 302 in the reference image of FIG. 3 (b), and arranges the template area 301 and the search area 302. Calculate the correlation value of. Here, the template area 301 is arranged at arbitrary coordinates of the reference image of FIG. 3A. The search area 302 is arranged in a predetermined size so as to evenly include the template area 301 in the vertical and horizontal directions. The template area 301 may be arranged with predetermined coordinates as the center, or the feature points may be calculated and arranged with the feature points as the center. In this embodiment, a sum of absolute differences (Sum of Absolute Difference: hereinafter abbreviated as SAD) is used as a method for calculating the correlation value. The calculation formula of SAD is shown in the formula (1).

In the formula (1), f (i, j) indicates the luminance value at the coordinates (i, j) in the template area 301. Further, g (i, j) indicates the luminance value at each coordinate in the correlation value calculation area 303, which is the target for calculating the correlation value, in the search area 302. The first motion vector detection unit 104 calculates the absolute value of the difference between the luminance values f (i, j) and g (i, j), and obtains the sum of them to obtain the correlation value S_SAD. The smaller the value of the correlation value S_SAD, the higher the degree of similarity between the textures of the template area 301 and the correlation value calculation area 303. As a method for calculating the correlation value, a method other than SAD may be used, and for example, a difference squared sum (SSD) or a normalized cross-correlation (NCC) may be used.

The first motion vector detection unit 104 moves the correlation value calculation area 303 in the entire area of the search area 302 to calculate the correlation value S_SAD. Then, the first motion vector detection unit 104 generates a correlation value map as shown in FIGS. 4A and 4B for the search area 302.

FIG. 4A shows a correlation value map calculated in the coordinate system of the search area 302, the X-axis and the Y-axis represent the correlation value map coordinates, and the Z-axis shows the magnitude of the correlation value at each coordinate. Represents. FIG. 4B shows the contour lines of FIG. 4A. In FIGS. 4A and 4B, the minimum correlation value is the minimum value 401. The first motion vector detection unit 104 can determine that a texture very similar to the template area 301 exists in the area in which the minimum value 401 is calculated in the search area 302. The local minimum value 402 represents the second smallest local minimum value, and the local minimum value 403 represents the third smallest local minimum value. The local minimum values 402 and 403 mean that a texture similar to the local minimum value 401 is present.

In this way, the first motion vector detection unit 104 calculates the correlation value S_SAD between the template area 301 and the search area 302, and determines the position of the correlation value calculation area 303 where the value is the smallest. As a result, the first motion vector detection unit 104 can specify the destination of the template area 301 on the reference image on the reference image. Then, the first motion vector detection unit 104 detects the motion vector having the direction and the amount of movement to the movement destination on the reference image with reference to the position of the template area 301 on the reference image. can do.

In step S204, the first reliability determination unit 105 uses the correlation value calculated by the first motion vector detection unit 104 to determine the reliability of the motion vector detected by the first motion vector detection unit 104. calculate.

FIG. 5 is a diagram showing correlation values represented by one-dimensional pixel addresses by arranging the correlation values in raster order 404 in the two-dimensional correlation value map of FIG. 4 (b). The vertical axis of FIG. 5 shows the correlation value, and the horizontal axis is the pixel address uniquely determined by the X coordinate and the Y coordinate of the correlation value map. Hereinafter, the expression shown in FIG. 5 is used to calculate the reliability. The correlation value 501 corresponds to the minimum value in FIGS. 4A and 4B. The motion vector detection units 104 and 107 calculate the correlation value for each pixel represented by the pixel address, and detect the motion vector based on the correlation value.

6 (a) to 6 (d) show an example of the index of the correlation value showing the reliability. The horizontal axis of FIG. 6 indicates the pixel address, and the vertical axis indicates the correlation value. In FIG. 6A, the minimum value Da of the correlation value is used as an index of reliability. When the minimum value Da is large, it is considered that the similarity between the template area 301 and the search area 302 is low, which means that the degree of matching is low and the reliability is low. Since the minimum value Da and the reliability are in an inversely proportional relationship, the reciprocal of the minimum value Da (1 / Da) is used as an index of the reliability.

In FIG. 6B, the ratio Db (= B / A) of the difference B between the average value and the minimum value of the correlation value is used as the index of reliability with respect to the difference A between the maximum value and the minimum value of the correlation value. .. The ratio Db represents the steepness of the maximum correlation value. When the ratio Db is small, the similarity between the template area 301 and the search area 302 is considered to be low, indicating that the reliability is low.

In FIG. 6C, the difference Dc between the maximum value and the minimum value of the correlation value is used as an index of reliability. The difference Dc represents the range of the correlation value map. When the difference Dc is small, it is considered that the contrast of the texture is low, indicating that the reliability is low.

In FIG. 6D, the difference Dd between the minimum minimum value 601 of the correlation value and the second smallest minimum value 602 is used as an index of reliability. Here, the minimum minimum value 601 is the minimum value and corresponds to the minimum minimum value 401 in FIGS. 4A and 4B. The local minimum value 602 corresponds to the second smallest local minimum value 402 in FIGS. 4 (a) and 4 (b). The local minimum value 603 corresponds to the third smallest local minimum value 403 in FIGS. 4 (a) and 4 (b). Therefore, FIG. 6 (d) means confirming whether or not there is a minimum value similar to the minimum value of the correlation value in the contour line of FIG. 4 (b). The difference Dd represents the periodicity of the correlation value map. When the difference Dd is smaller than the threshold value, it is considered that the texture is a repeating pattern, an edge pattern, or the like, indicating that the reliability is low. Here, the minimum minimum value 601 and the second smallest minimum value 602 are selected, but other minimum values may be selected as long as the periodicity of the correlation value map can be determined. If the difference Dd is smaller than the threshold, and if the minimum value 702 continues in the horizontal, vertical, or diagonal direction with reference to the minimum value 701 of the correlation value, as shown in the contour lines in FIG. 7, it is determined that the edge pattern of the texture exists. Then, the difference Dd is used as the reliability index Dd2. If not, it is determined that the texture repetition pattern exists, and the difference Dd is used as the reliability index Dd1.

The index of the correlation value described above can be used as it is as the reliability, but for example, as shown in FIG. 8, the index of the correlation value and the reliability may be associated with each other. The horizontal axis of FIG. 8 indicates the correlation value index D (one of the above 1 / Da, Db, Dc, Dd1 and Dd2), and the vertical axis indicates the reliability R. In this example, two thresholds T1 and T2 are provided. When the correlation value index D is equal to or less than the threshold value T1, the reliability R is set to 0. When the correlation value index D is equal to or higher than the threshold value T2, the reliability R is set to 1. The threshold values T1 and T2 may be changed for each correlation value index. In the interval between the threshold values T1 and T2, the correlation value index D and the reliability R may be associated linearly or non-linearly. In the following description, the reliability obtained from each correlation value index 1 / Da, Db, Dc, Dd1 and Dd2 is expressed as Ra, Rb, Rc, Rd1 and Rd2. Here, there is a relationship of Ra = f (1 / Da), Rb = f (Db), Rc = f (Dc), Rd1 = f (Dd1), and Rd2 = f (Dd2).

The final reliability R may be calculated by combining these Ra, Rb, Rc, Rd1 and Rd2. Here, a combination method based on weighted addition will be described. In the combination by weighting addition, if the weights of Ra, Rb, Rc, Rd1 and Rd2 are Wa, Wb, Wc, Wd1 and Wd2, respectively, the reliability R is calculated as in the equation (2).

For example, the weights Wa = 0.4, Wb = 0.3, Wc = 0.2, Wd1 = 0.05, Wd2 = 0.05. When all the reliabilitys are sufficiently high and Ra = Rb = Rc = Rd1 = Rd2 = 1, R = 1.0 is obtained from the equation (2). Further, in the case of Ra = 0.6, Rb = 0.5, Rc = 0.7, Rd1 = 0.7, Rd2 = 0.7, R = 0.6 is obtained from the equation (2). Become.

In step S205, the second image processing unit 106 is calculated by the first reliability determination unit 105 with respect to the reference image input by the image input unit 101 and the reference image stored in the image memory 102. Image processing is performed adaptively based on the reliability.

FIG. 15A is a diagram showing an example of a combination of reliability and image processing. When the second image processing unit 106 determines that the reliability Ra is lower than the threshold value and the reliability Rb is lower than the threshold value, the bandpass filter is used to perform image processing for emphasizing the texture. The reason why the similarity is low may be that the texture of the image is not clear or the influence of noise or the like. When the reliability Ra and Rb are both lower than the threshold value, it is considered that the texture of the image is unclear. Therefore, the characteristics of the bandpass filter of the second image processing unit 106 are set higher than those of the first image processing unit 103. Try to leave high frequencies. As a result, the second image processing unit 106 can perform processing for emphasizing the texture of the reference image and the reference image. If the reliability Ra is higher than the threshold value, but the reliability Rb is lower than the threshold value, there is a possibility that the wrong correlation value is high due to noise. It has a characteristic that lowers a high frequency than that of the first image processing unit 103. In this case, the second image processing unit 106 can reduce noise with respect to the reference image and the reference image. As a result, the degree of similarity between the image template area 301 and the search area 302 can be increased.

When the second image processing unit 106 determines that the reliability Rc is lower than the threshold value, the second image processing unit 106 performs a process of making the bright portion of the luminance value in the image region brighter and the dark portion darker, thereby contrasting the image. Perform the process of adding. Further, the second image processing unit 106 also performs processing for leaving the texture information of the image by using the high-pass filter, and performs contrast enhancement and texture enhancement.

When the second image processing unit 106 determines that the reliability Rd1 is lower than the threshold value, the second image processing unit 106 uses a notch filter to remove the repeating pattern. Further, when the second image processing unit 106 determines that the reliability Rd2 is lower than the threshold value, the second image processing unit 106 uses a differential filter, extracts an edge by the differential filter, and extracts an edge according to the length of the extracted edge. Edges are removed (suppressed) by lowering the response value of.

In step S206, the second motion vector detection unit 107 detects the motion vector between the reference image and the reference image processed by the second image processing unit 106 by template matching. Since this process is the same as the process of step S203, the description thereof will be omitted.

In step S207, the second reliability determination unit 108 uses the correlation value calculated by the second motion vector detection unit 107 to determine the reliability of the motion vector detected by the second motion vector detection unit 107. calculate. Since this process is the same as the process of step S204, the description thereof will be omitted.

In step S208, the motion vector determination unit 109 has the motion vector detected by the first motion vector detection unit 104 and the second motion vector detection unit 107, and the first reliability determination unit 105 and the second reliability. The reliability calculated by the determination unit 108 is input. Then, the motion vector determination unit 109 determines the motion vector by using the two input motion vectors and the two reliabilitys. The motion vector determination unit 109 may simply select the motion vector having the higher reliability among the two reliabilitys, or moves with the reliabilitys R1 and R2 as weights as in the equation (3). The vector V may be determined. In equation (3), V is a motion vector that is finally determined. V1 is a motion vector detected in step S203. R1 is the reliability calculated in step S204. V2 is a motion vector detected in step S206. R2 is the reliability calculated in step S207.

As described above, in the present embodiment, the image processing apparatus 100 applies the image processing of the second image processing unit 106 according to the reliability of the motion vector detected by the first motion vector detection unit 104. Do it. As a result, the second motion vector detection unit 107 can detect the motion vector with high accuracy. The motion vector determination unit 109 has the first motion vector detection unit 104 and the second motion based on the reliability R1 and R2 calculated by the first reliability determination unit 105 and the second reliability determination unit 108. The motion vector weighting detected by the vector detection unit 107 is performed. As a result, the reliability of the motion vector can be improved even for a motion vector having a low reliability that cannot be dealt with by the image processing of the first image processing unit 103 and the second image processing unit 106.

(Second embodiment)
FIG. 9 is a block diagram showing a configuration example of the image processing apparatus 900 according to the second embodiment of the present invention. In FIG. 9, the same reference numerals are given to the components common to those shown in FIG. The image processing device 900 includes an image input unit 101, an image memory 102, an image analysis unit 901, a first image processing unit 902, a first motion vector detection unit 104, and a first reliability determination unit 105. And have. Further, the image processing apparatus 900 has a second image processing unit 903, a second motion vector detection unit 107, a second reliability determination unit 108, and a motion vector determination unit 109.

The image analysis unit 901 analyzes the characteristics of at least one of the reference image input by the image input unit 101 and the reference image stored in the image memory 102. The analysis of the image characteristics is performed for each template area 301 or search area 302 used by the motion vector detection unit 104 or 107.

The first image processing unit 902 performs image processing on the reference image input by the image input unit 101 and the reference image stored in the image memory 102 based on the image characteristics analyzed by the image analysis unit 901. .. The first motion vector detection unit 104 detects a motion vector by, for example, template matching with respect to the reference image processed by the first image processing unit 902 and the reference image processed by the first image processing unit 902. do.

The second image processing unit 903 performs image processing on the reference image input by the image input unit 101 and the reference image stored in the image memory 102 based on the image characteristics analyzed by the image analysis unit 901. .. The second motion vector detection unit 107 detects a motion vector for the reference image processed by the second image processing unit 903 and the reference image processed by the second image processing unit 903, for example, by template matching. do.

The first image processing unit 902 and the second image processing unit 903 perform different image processing from each other. For example, when the image characteristics analyzed by the image analysis unit 901 have low contrast, the image processing units 902 and 903 apply a high-pass filter, but if the peak band of the filter is too high, not only the texture but also the noise is emphasized. .. Therefore, the image processing units 902 and 903 adaptively apply filters having two types of characteristics, a band that prioritizes texture enhancement and a band that suppresses noise while emphasizing the texture. In this way, the image processing units 902 and 903 perform image processing with filters having two types of characteristics, so that the correlation value changes depending on the filter characteristics. Therefore, even if the reliability of one filter characteristic is low, the other one. The filter characteristics of can be highly reliable.

FIG. 10 is a flowchart for explaining an image processing method of the image processing apparatus 900. In FIG. 10, the steps common to those shown in FIG. 2 are designated by the same reference numerals as those in FIG. 2, and the description thereof will be omitted.

After step S201, in step S1001, the image analysis unit 901 performs image analysis on the reference image input by the image input unit 101 and the reference image stored in the image memory 102, and based on the brightness value. , Determine the characteristics of the image.

11 (a) to 11 (c) and FIG. 12 show an example of an index of a luminance value representing an image characteristic. The horizontal axis of FIGS. 11A to 11C is a pixel address uniquely determined by the X-coordinate and the Y-coordinate of the image, and the vertical axis of FIGS. 11A to 11C is a luminance value. The image analysis unit 901 analyzes the brightness value for each pixel of the image. In FIG. 12, one square represents a pixel, an arrow represents a gradient of luminance value with an adjacent pixel, the size of the arrow represents the magnitude of change in luminance value, and the direction of the arrow represents the change in luminance value. Represents the direction of.

In FIG. 11A, the difference Da between the maximum value and the minimum value of the luminance value is used as an index. When the difference Da is smaller than the threshold value, the image analysis unit 901 determines that the contrast of the texture is low.

In FIG. 11B, the number of repetitions Db of the luminance value is used as an index. The image analysis unit 901 detects the number of repetitions Db by using a filter and extracting a frequency having a response higher than the threshold value. When the number of repetitions Db is larger than the threshold value, the image analysis unit 901 determines that there is a texture of the repetition pattern.

In FIG. 11C, the high frequency Dc of the luminance value is used as an index. The image analysis unit 901 extracts a high frequency component of the brightness value of the image by using a high-pass filter, detects the frequency Dc of the extracted high frequency component, and when the frequency Dc is higher than the threshold value, noise is added to the image. Is determined.

In FIG. 12, with respect to the gradient 1201 having the maximum gradient of the luminance value, the gradient 1202 of the luminance value in which the direction of the gradient of the luminance value is the same and the change in magnitude is within a certain range is horizontal, vertical, or When continuing in the diagonal direction, the number Dd of the gradient 1202 of the luminance value is used as an index. When the number Dd is larger than the threshold value, the image analysis unit 901 determines that an edge pattern of the brightness value of the image exists.

In step S1002, the first image processing unit 902 changes the image characteristics analyzed by the image analysis unit 901 with respect to the reference image input by the image input unit 101 and the reference image stored in the image memory 102. Image processing is performed adaptively accordingly.

FIG. 15B is a diagram showing an example of a combination of image characteristics and image processing. When the first image processing unit 902 determines that the difference (low contrast) Da is smaller than the threshold value, the first image processing unit 902 performs a process of making the bright portion of the luminance value in the image region brighter and the dark portion darker. Performs processing to add contrast to the image. Further, the first image processing unit 902 uses a high-pass filter to perform processing such as leaving the texture information of the image, and performs contrast enhancement and texture enhancement.

Further, when the first image processing unit 902 determines that the number of repetitions Db is larger than the threshold value, the notch filter is used to remove the repetition pattern of the texture. Further, when the first image processing unit 902 determines that the frequency (noise) Dc is higher than the threshold value, the first image processing unit 902 removes (reduces) the noise component which is a high frequency component by using a low-pass filter.

Further, when the first image processing unit 902 determines that the number (edge pattern) Dd is larger than the threshold value, the edge pattern of the luminance value of the image exists, so that the differential filter is used. The first image processing unit 902 extracts edges by a differential filter, and removes (reduces) the edges by lowering the response value of the differential filter according to the length of the extracted edges.

The image processing apparatus 900 performs the processing of steps S203 and S204 after step S1002.

In step S1003, the second image processing unit 903 has the image characteristics analyzed by the image analysis unit 901 with respect to the reference image input by the image input unit 101 and the reference image stored in the image memory 102. Image processing is performed adaptively accordingly. Similar to step S1002, the second image processing unit 903 performs image processing according to the image characteristics as shown in FIG. 15B. However, the second image processing unit 903 performs image processing different from that of the first image processing unit 902.

When the second image processing unit 903 determines that the low contrast Da is smaller than the threshold value, the second image processing unit 903 further brightens the bright portion of the brightness value in the image region and further brightens the dark portion as compared with the first image processing unit 902. By performing the darkening process, the contrast of the image is extremely increased. Further, the second image processing unit 903 performs a process of further emphasizing the texture of the image by setting the peak band of the high-pass filter to be used to a higher frequency than that of the first image processing unit 902.

Further, when the second image processing unit 903 determines that the number of repetitions Db is larger than the threshold value, the second image processing unit 903 narrows the band to be removed by the notch filter narrower than that of the first image processing unit 902, thereby performing specific repetition. Try to remove only the pattern.

Further, when the second image processing unit 903 determines that the frequency (noise) Dc is higher than the threshold value, the peak band of the low-pass filter is set to a lower frequency than that of the first image processing unit 902. Perform processing to reduce higher frequency components.

Further, when it is determined that the number (edge pattern) Dd is large, the second image processing unit 903 increases the response value of the differential filter as compared with the first image processing unit 902, thereby increasing the luminance value. Edges with a low slope are also extracted. Then, the second image processing unit 903 performs processing so that the edge having a lower gradient of the luminance value can be removed by lowering the response value of the differential filter according to the length of the extracted edge.

The image processing apparatus 900 performs the processing of steps S206 to S208 after step S1003. As described above, in the present embodiment, the image processing units 902 and 903 perform adaptive image processing based on the image characteristics, and unlike the first embodiment, both perform adaptive image processing. It is possible to do. Further, the first image processing unit 902 and the second image processing unit 903 perform different image processing from each other, so that the reliability calculated by the first reliability determination unit 105 and the second reliability determination unit 108 is performed. There is a difference in degree. Therefore, in the first reliability determination unit 105 and the second reliability determination unit 108, even if the reliability of one is lower than that of the first embodiment, the reliability of the other may be higher. Yes, the reliability of the motion vector can be improved.

(Third embodiment)
FIG. 13 is a block diagram showing a configuration example of the image processing apparatus 1300 according to the third embodiment of the present invention. In FIG. 13, the same reference numerals are given to the components common to those shown in FIGS. 1 or 9. The image processing device 1300 includes an image input unit 101, an image memory 102, an image analysis unit 901, a first image processing unit 902, a first motion vector detection unit 104, and a first reliability determination unit 105. And have. Further, the image processing apparatus 1300 has a second image processing unit 106, a second motion vector detection unit 107, a second reliability determination unit 108, and a motion vector determination unit 109.

FIG. 14 is a flowchart for explaining an image processing method of the image processing apparatus 1300. In FIG. 14, the steps common to those shown in FIGS. 2 or 10 are designated by the same reference numerals, and the description thereof will be omitted. The third embodiment is an embodiment in which the first embodiment and the second embodiment are combined.

After step S201, in step S1001, the image analysis unit 901 displays an image with respect to the reference image input by the image input unit 101 and the reference image stored in the image memory 102, as in the second embodiment. The analysis is performed and the characteristics of the image are determined based on the brightness value.

In step S1002, the first image processing unit 902 performs image analysis on the reference image input by the image input unit 101 and the reference image stored in the image memory 102, as in the second embodiment. Image processing is adaptively performed according to the image characteristics analyzed by the unit 901.

After that, the image processing apparatus 1300 performs the processing of steps S203 to S208 as in the first embodiment. That is, in step S205, the second image processing unit 106 uses the first reliability determination unit 105 for the reference image input by the image input unit 101 and the reference image stored in the image memory 102. Image processing is performed adaptively based on the calculated reliability.

As described above, in the present embodiment, the first image processing unit 902 performs image processing adaptively based on the image characteristics, and the second image processing unit 106 is performed by the first reliability determination unit 105. Image processing is performed adaptively based on the calculated reliability. Therefore, in this embodiment, the reliability of the first reliability determination unit 105 is improved as compared with the first embodiment. Further, in the present embodiment, as compared with the second embodiment, the second image processing unit 106 uses the reliability of the first reliability determination unit 105 including the result of the image analysis, so that the second reliability is used. The reliability of the degree determination unit 108 is improved.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

It should be noted that the above embodiments are merely examples of embodiment of the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

100 image processing device, 101 image input unit, 102 image memory, 103 first image processing unit, 104 first motion vector detection unit, 105 first reliability determination unit, 106 second image processing unit, 107th 2 motion vector detection unit, 108 second reliability determination unit, 109 motion vector determination unit

Claims (19)

A first image processing means for performing a first image processing on a first image and a second image, and
A first motion vector detecting means for detecting a first motion vector based on the first image and the second image on which the first image processing has been performed, and a motion vector detecting means.
The first reliability calculation means for calculating the reliability of the first motion vector, and
A second image processing means that performs a second image processing on the first image and the second image based on the reliability of the first motion vector.
A second motion vector detecting means for detecting a second motion vector based on the first image and the second image subjected to the second image processing, and
A second reliability calculation means for calculating the reliability of the second motion vector, and
A motion that determines a third motion vector based on the first motion vector, the second motion vector, the reliability of the first motion vector, and the reliability of the second motion vector. An image processing apparatus comprising a vector determining means. An image analysis means for analyzing at least one of the first image and the second image,
A first image processing means that performs a first image processing on the first image and the second image according to the result of the analysis of the image.
A second image processing means that performs a second image processing different from the first image processing on the first image and the second image according to the result of the analysis of the image.
A first motion vector detecting means for detecting a first motion vector based on the first image and the second image on which the first image processing has been performed, and a motion vector detecting means.
A second motion vector detecting means for detecting a second motion vector based on the first image and the second image subjected to the second image processing, and
The first reliability calculation means for calculating the reliability of the first motion vector, and
A second reliability calculation means for calculating the reliability of the second motion vector, and
A motion that determines a third motion vector based on the first motion vector, the second motion vector, the reliability of the first motion vector, and the reliability of the second motion vector. An image processing apparatus comprising a vector determining means. An image analysis means for analyzing at least one of the first image and the second image,
A first image processing means that performs a first image processing on the first image and the second image according to the result of the analysis of the image.
A first motion vector detecting means for detecting a first motion vector based on the first image and the second image on which the first image processing has been performed, and a motion vector detecting means.
The first reliability calculation means for calculating the reliability of the first motion vector, and
A second image processing means that performs a second image processing on the first image and the second image based on the reliability of the first motion vector.
A second motion vector detecting means for detecting a second motion vector based on the first image and the second image subjected to the second image processing, and
A second reliability calculation means for calculating the reliability of the second motion vector, and
A motion that determines a third motion vector based on the first motion vector, the second motion vector, the reliability of the first motion vector, and the reliability of the second motion vector. An image processing apparatus comprising a vector determining means. The first motion vector detecting means calculates a first correlation value based on the first image and the second image on which the first image processing has been performed, and obtains the first correlation value. Based on this, the first motion vector is detected,
The second motion vector detecting means calculates a second correlation value based on the first image and the second image on which the second image processing has been performed, and obtains the second correlation value. Based on this, the second motion vector is detected,
The first reliability calculation means calculates the reliability of the first motion vector based on the first correlation value.
The second reliability calculation means according to any one of claims 1 to 3, wherein the second reliability calculation means calculates the reliability of the second motion vector based on the second correlation value. Image processing device. The first motion vector detecting means calculates a first correlation value for each pixel based on the first image and the second image on which the first image processing has been performed, and the first image processing means. The first motion vector is detected based on the correlation value, and the first motion vector is detected.
The first reliability calculation means calculates the first reliability of the first motion vector based on the reciprocal of the minimum value of the first correlation value, and the maximum value of the first correlation value. The second reliability of the first motion vector is calculated based on the ratio of the difference between the average value and the minimum value of the first correlation value with respect to the difference between the first value and the minimum value.
In the second image processing means, when the first reliability of the first motion vector is lower than the threshold value and the second reliability of the first motion vector is lower than the threshold value, the first image processing means is described. The image processing apparatus according to claim 1 or 3, wherein the image and the second image are processed to emphasize the texture. The first motion vector detecting means calculates a first correlation value for each pixel based on the first image and the second image on which the first image processing has been performed, and the first image processing means. The first motion vector is detected based on the correlation value, and the first motion vector is detected.
The first reliability calculation means calculates the first reliability of the first motion vector based on the reciprocal of the minimum value of the first correlation value, and the maximum value of the first correlation value. The second reliability of the first motion vector is calculated based on the ratio of the difference between the average value and the minimum value of the first correlation value with respect to the difference between the first value and the minimum value.
In the second image processing means, when the first reliability of the first motion vector is higher than the threshold value and the second reliability of the first motion vector is lower than the threshold value, the first image processing means is described. The image processing apparatus according to any one of claims 1, 3 and 5, wherein the image and the second image are processed to reduce noise. The first motion vector detecting means calculates a first correlation value for each pixel based on the first image and the second image on which the first image processing has been performed, and the first image processing means. The first motion vector is detected based on the correlation value, and the first motion vector is detected.
The first reliability calculation means calculates the reliability of the first motion vector based on the difference between the maximum value and the minimum value of the first correlation value.
When the reliability of the first motion vector is lower than the threshold value, the second image processing means enhances the texture and the contrast of the first image and the second image. The image processing apparatus according to any one of claims 1, 3, 5, and 6. The first motion vector detecting means calculates a first correlation value for each pixel based on the first image and the second image on which the first image processing has been performed, and the first image processing means. The first motion vector is detected based on the correlation value, and the first motion vector is detected.
The first reliability calculation means calculates the reliability of the first motion vector based on the difference between the minimum minimum value of the first correlation value and another minimum value.
When the reliability of the first motion vector is lower than the threshold value and the texture repeating pattern exists, the second image processing means repeats the first image and the second image. The image processing apparatus according to any one of claims 1, 3, 5 to 7, wherein a process for removing a pattern is performed. The first motion vector detecting means calculates a first correlation value for each pixel based on the first image and the second image on which the first image processing has been performed, and the first image processing means. The first motion vector is detected based on the correlation value, and the first motion vector is detected.
The first reliability calculation means calculates the reliability of the first motion vector based on the difference between the minimum minimum value of the first correlation value and another minimum value.
When the reliability of the first motion vector is lower than the threshold value and the edge pattern of the texture is present, the second image processing means has an edge with respect to the first image and the second image. The image processing apparatus according to any one of claims 1, 3, 5 to 8, wherein the image processing apparatus is characterized in that the image processing apparatus is performed. The image processing apparatus according to claim 2, wherein the image analysis means analyzes a luminance value for each pixel of an image. The image according to claim 10, wherein the first image processing means enhances the contrast and the texture when the difference between the maximum value and the minimum value of the luminance value is smaller than the threshold value. Processing device. The image processing apparatus according to claim 10, wherein the first image processing means performs a process of removing a texture repetition pattern when the number of repetitions of the luminance value is more than a threshold value. The image analysis means extracts a high frequency component of the brightness value of the image, detects the frequency of the extracted high frequency component, and determines the frequency.
The image according to any one of claims 10 to 12, wherein the first image processing means performs processing for reducing the high frequency component when the frequency of the high frequency component is higher than the threshold value. Processing equipment. The image according to any one of claims 10 to 13, wherein the first image processing means performs a process of suppressing the edge when an edge pattern of an image luminance value is present. Processing device. The image processing apparatus according to any one of claims 4 to 9, wherein the correlation value is a difference absolute value sum, a difference square sum, or a normalized cross-correlation. A first image processing step in which the first image processing is performed on the first image and the second image, and
A first motion vector detection step for detecting a first motion vector based on the first image and the second image on which the first image processing has been performed, and
The first reliability calculation step for calculating the reliability of the first motion vector, and
A second image processing step of performing a second image processing on the first image and the second image based on the reliability of the first motion vector.
A second motion vector detection step for detecting a second motion vector based on the first image and the second image subjected to the second image processing, and
The second reliability calculation step for calculating the reliability of the second motion vector, and
A motion that determines a third motion vector based on the first motion vector, the second motion vector, the reliability of the first motion vector, and the reliability of the second motion vector. An image processing method comprising a vector determination step. An image analysis step that analyzes at least one of the first image and the second image,
A first image processing step of performing a first image processing on the first image and the second image according to the result of the analysis of the image,
A second image processing step of performing a second image processing different from the first image processing on the first image and the second image according to the result of the analysis of the image.
A first motion vector detection step for detecting a first motion vector based on the first image and the second image on which the first image processing has been performed, and
A second motion vector detection step for detecting a second motion vector based on the first image and the second image subjected to the second image processing, and
The first reliability calculation step for calculating the reliability of the first motion vector, and
The second reliability calculation step for calculating the reliability of the second motion vector, and
A motion that determines a third motion vector based on the first motion vector, the second motion vector, the reliability of the first motion vector, and the reliability of the second motion vector. An image processing method comprising a vector determination step. An image analysis step that analyzes at least one of the first image and the second image,
A first image processing step of performing a first image processing on the first image and the second image according to the result of the analysis of the image,
A first motion vector detection step for detecting a first motion vector based on the first image and the second image on which the first image processing has been performed, and
The first reliability calculation step for calculating the reliability of the first motion vector, and
A second image processing step of performing a second image processing on the first image and the second image based on the reliability of the first motion vector.
A second motion vector detection step for detecting a second motion vector based on the first image and the second image subjected to the second image processing, and
The second reliability calculation step for calculating the reliability of the second motion vector, and
A motion that determines a third motion vector based on the first motion vector, the second motion vector, the reliability of the first motion vector, and the reliability of the second motion vector. An image processing method comprising a vector determination step. A program for making a computer function as each means of the image processing apparatus according to any one of claims 1 to 15.

Images