JP2018072942A - Image processing apparatus, image processing method, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of improving detection performance for a motion vector.SOLUTION: An image processing apparatus 100 of a digital camera 10 comprises: a hierarchical image generation unit 104 which generates images in respective hierarchies with a plurality of different resolutions with respect to each of a standard image and a reference image as input images; a template region setting unit 110 which sets a position and size of a template region for a standard image in each hierarchy; a search region setting unit 111 which sets a position and size of a search region for a reference image in each hierarchy; a motion vector calculation unit 106 which calculates a motion vector by correlation value operation on the template region and the search region; and a reliability calculation unit 107 which calculates reliability of the motion vector. The size of the template region and the position and size of the search region are set on the basis of reliability of a motion vector calculated for images in a hierarchy with a lower resolution.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus that detects a motion vector between a plurality of frame images.

  In order to perform image blur correction and dynamic range expansion processing on video shot using an imaging device such as a digital still camera or digital video camera, the amount of motion between frame images is detected and multiple images are captured. Need to align. As a method for detecting the amount of motion between frame images, there are a method of using information of an external device such as a gyro sensor, a method of estimating the amount of motion from a captured frame image, and the like.

  Various methods for estimating the amount of motion using a frame image have been proposed in the past. A typical example is motion vector detection by template matching. As a motion vector detection method, the following hierarchical motion vector detection method is known. In hierarchical motion vector detection, a plurality of hierarchical images having different resolutions are generated from an input image. First, a motion vector is detected from the lowest resolution hierarchical image, and a motion vector is detected from the high resolution hierarchical image while referring to the motion vector detected from the low resolution hierarchical image.

  The advantage of hierarchical motion vector detection is that by referring to the motion vector detected in the low-resolution hierarchical image, the search area in the high-resolution hierarchical image is reduced, and the correlation between the luminance distribution of the template area and the search area is calculated. It is possible to shorten the calculation for this. On the other hand, if the motion vector is not correctly detected in the low resolution hierarchical image, the wrong motion vector is referred to in the high resolution hierarchical image, and the correct motion vector cannot be finally obtained. .

  As a method for preventing such false detection propagation between hierarchical images, Patent Document 1 and Patent Document 2 determine the reliability of a motion vector, and use a motion vector with low reliability as a surrounding motion with high reliability. A technique for correcting using a vector is disclosed.

Japanese Patent Laid-Open No. 2001-148012 JP 2009-282762 A

  However, there is a problem that the detection performance in the hierarchical motion vector detection cannot be sufficiently obtained only by correcting the motion vector with low reliability in the low-resolution hierarchical image.

  In template matching, the smaller the angle of view in the template, the lower the probability of including a characteristic texture in the template, and the lower the possibility that a highly reliable motion vector can be detected. On the other hand, the larger the angle of view, the more easily affected by the motion of the surrounding area other than the feature points, and the motion vector detection accuracy decreases.

  Therefore, in the hierarchical motion vector detection, it is necessary to control the size of the template region in consideration of the relationship between the two. In this regard, the methods disclosed in Patent Literature 1 and Patent Literature 2 do not mention the size of the template region of each hierarchical image.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to improve motion vector detection performance when detecting a motion vector by generating a plurality of hierarchical images having different resolutions from an input image. An object of the present invention is to provide an image processing apparatus that can be improved.

  An image processing apparatus according to the present invention generates hierarchical image generation means for generating images of a plurality of different resolutions for each of a standard image and a reference image as input images, and the hierarchical image generation means Template area setting means for setting the position and size of the template area with respect to the standard image of each hierarchy, and the position and size of the search area with respect to the reference image of each hierarchy generated by the hierarchy image generating means A search area setting means for setting a motion vector calculation means for calculating a motion vector by calculating a correlation value between the template area and the search area, and a reliability calculation means for calculating the reliability of the motion vector, The template region setting means determines the reliability of the motion vector calculated for an image of a lower resolution hierarchy. And setting the size of the template area, and the search area setting means determines the position and size of the search area based on the reliability of the motion vector calculated for the image of the lower resolution hierarchy. It is characterized by setting.

  According to the present invention, it is possible to provide an image processing apparatus capable of improving motion vector detection performance when generating a plurality of hierarchical images having different resolutions from an input image and detecting a motion vector. It becomes.

1 is a block diagram showing the configuration of a digital camera that is a first embodiment of an image processing apparatus of the present invention. 5 is a flowchart showing the operation of the digital camera of the first embodiment. The figure explaining the relationship between the reliability of a motion vector, and a template area | region. The figure which shows the template area | region based on the reliability of a motion vector. The figure explaining the template size when the reliability of a motion vector is low. The figure explaining template size in case the reliability of a motion vector is high. The figure explaining the relationship between the reliability of a motion vector, and a search area | region. The figure explaining template matching. The figure explaining a correlation value map. The figure explaining the expression method of a correlation value map. The figure explaining the correlation value parameter | index showing the reliability of a motion vector. The figure explaining the calculation method of the reliability of a motion vector. The block diagram which shows the structure of the digital camera of 2nd Embodiment. 9 is a flowchart showing the operation of the digital camera of the second embodiment. The figure explaining the area division | segmentation process in 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
FIG. 1 is a block diagram showing a configuration of a digital camera (imaging device) 10 which is a first embodiment of an image processing device of the present invention. In FIG. 1, the photographing optical system 101 includes a lens, a diaphragm, and the like, and performs focus adjustment and exposure adjustment at the time of photographing to form an optical image on the image sensor 102. The image sensor 102 has a photoelectric conversion function for converting an optical image into an electrical signal (analog image signal), and is configured by a CCD, a CMOS sensor, or the like.

  The camera signal processing unit 103 includes an A / D conversion unit, an auto gain control unit (AGC), and an auto white balance unit (not shown), and converts a digital video signal (hereinafter referred to as an input image) from an analog electric signal output from the image sensor 102. Generated). The imaging element 102 and the camera signal processing unit 103 constitute an imaging unit that generates an input image by imaging. Further, the image processing apparatus 100 is configured by a portion surrounded by a broken line in FIG.

  The hierarchical image generation unit 104 performs filter processing and downsampling on the input image to limit the frequency band, and generates a plurality of image data (hereinafter referred to as a hierarchical image) whose resolution is lower than the input image in stages. . The input image itself is also acquired as one hierarchical image (highest resolution hierarchical image). As a result, a plurality of hierarchical images including the input image are acquired using the input image. The hierarchical image generation unit 104 stores the hierarchical images having different resolutions in the hierarchical image holding unit 105.

  The motion vector detection unit (motion vector calculation unit) 106 receives two hierarchical images having the same resolution input from the hierarchical image generation unit 104 and the hierarchical image holding unit 105 (hereinafter, one is referred to as a reference image and the other is referred to as a reference image). From, for example, the motion vector is detected by template matching. The reliability calculation unit 107 calculates the reliability for the motion vector input from the motion vector detection unit 106.

  The motion vector detected by the motion vector detection unit 106 and the reliability calculated by the reliability calculation unit 107 (hereinafter, the motion vector and the reliability of the motion vector are collectively referred to as motion vector information) are referred to as a motion vector information holding unit 108. To store. The motion vector control unit 109 includes a template region setting unit 110 and a search region setting unit 111, and controls the motion vector detection unit 106 based on the motion vector information held by the motion vector information holding unit 108.

  The template area setting unit 110 sets a rectangular area of a predetermined size at a predetermined position as a template area with respect to the reference image input to the motion vector detection unit 106. The search area setting unit 111 sets a rectangular area of a predetermined size at a predetermined position as a search area for the reference image input to the motion vector detection unit 106.

  The image processing apparatus 100 including these components 104 to 111 is configured by an image processing computer such as a CPU or MPU, and executes the image processing shown in the flowchart in FIG. 2 according to an image processing program as a computer program. Hereinafter, the operation of the image processing apparatus 100 will be described with reference to the flowchart of FIG.

  In step S201, the hierarchical image generation unit 104 of the image processing apparatus 100 generates a plurality of hierarchical images. In the following description, it is assumed that the hierarchical image generation unit 104 generates a low resolution image that is a 1/2 reduced image with respect to the input image from a high resolution image as an input image (1/1 image).

  In step S202, the motion vector control unit 109 selects a hierarchical image to be processed. In this embodiment, it is selected to sequentially process from a low resolution image to a high resolution image. In step S203, the template area setting unit 110 sets a template area for the hierarchical image selected in step S202. The position of the template region may be fixedly determined, but is preferably arranged so as to include a texture suitable for vector detection such as a feature point. For example, feature extraction is performed using a hierarchical image obtained from the hierarchical image holding unit 105, and a template region is arranged around the extracted feature points.

  As a feature point extraction method, for example, Harris corner detector or Shi and Tomasi method is used. In these methods, the luminance value at the pixel (x, y) of the image is expressed as I (x, y), and from the results Ix and Iy obtained by applying horizontal and vertical first-order differential filters to the image, An autocorrelation matrix H shown in 1) is created.

In Expression (1), G represents smoothing by the Gaussian distribution shown in Expression (2).

The Harris detector extracts a pixel having a large feature amount as a feature point using the feature evaluation formula shown in Formula (3).

Harris = det (H) −α (tr (H)) ² (3)
(Α = 0.04-0.15)
In Expression (3), det represents a determinant, and tr represents the sum of diagonal components. Further, α is a constant, and a value of 0.04 to 0.15 is experimentally good.

  On the other hand, Shi and Tomasi use the feature evaluation formula shown in Formula (4).

Shi and Tomasi = min (λ1, λ2) (4)
Equation (4) represents that the smaller eigenvalue of the eigenvalues λ1 and λ2 of the autocorrelation matrix H in equation (1) is used as the feature quantity. Even when Shi and Tomasi are used, pixels with large feature values are extracted as feature points.

  The size of the template region is preferably set according to the reliability of the motion vector detected in the lower resolution hierarchical image. FIG. 3 shows the relationship between the reliability of the motion vector and the size of the template area. In the lowest resolution hierarchical image, a predetermined size may be set regardless of the reliability of the motion vector.

  In FIG. 3, 301 is a hierarchical image to be processed, 302 is the coordinates of feature points, and 303 is a template region set by a lower resolution hierarchical image. Reference numerals 304, 306, and 308 denote motion vectors obtained by scaling the motion vectors detected in the lower resolution layer according to the size of the layer image to be processed. Here, it is assumed that the reliability of the motion vector is vector 304 <vector 306 <vector 308. Reference numerals 305, 307, and 309 denote template areas in the hierarchical image to be processed, and the size of the template area is reduced as the reliability of the motion vector of the low-resolution image increases.

  That is, when the reliability is low, the template area 305 is made larger than the template area 303 of the low-resolution image as shown in FIG. 3A (305> 303). When the reliability is medium, as shown in FIG. 3B, the template area 307 is made the same as the template area 303 of the low-resolution image (307 = 303). When the reliability is high, the template area 309 is made smaller than the template area 303 of the low-resolution image as shown in FIG. 3C (309 <303).

  FIG. 3D is a graph showing the relationship between the reliability and the template region size, with the horizontal axis representing the reliability and the vertical axis representing the template region size. In FIG. 3D, the template region size is changed linearly with respect to the reliability, but may be changed nonlinearly.

  As described above, the effect of reducing the size of the template region as the reliability of the motion vector of the low-resolution image becomes higher will be described with reference to FIGS. 4, 5, and 6.

  FIG. 4 shows a case where a template area having the same size as that of the low resolution image is set in the high resolution image regardless of the reliability of the motion vector detected in the low resolution image. In FIG. 4, 401 is a low resolution image, 404 is a high resolution image, 402 and 405 are feature points, and 403 and 406 are template regions.

  Now, the template area 406 of the high resolution image is set to the same size as the template area 403 of the low resolution image. Then, as can be seen from FIG. 4, the angle of view of the template area 406 is smaller than the angle of view of the template area 403 by the ratio of the low resolution image and the high resolution image. If the reliability of the motion vector detected in the low-resolution image is high, the amount of texture information contained in the template is still sufficient even if the angle of view of the template area is reduced in the high-resolution image. It is considered that a high motion vector can be detected. Rather, it is less likely to be affected by the movement of the peripheral area other than the feature points, and it can be expected that the detection accuracy of the motion vector is improved in the high resolution image.

  However, when the reliability of the motion vector detected in the low-resolution image is low, the angle of view of the template region in the high-resolution image becomes small, so that the amount of texture information included in the template is insufficient. And it is considered that the possibility that a highly reliable motion vector can be detected even in a high-resolution image is low. Therefore, it is necessary to control the size of the template region of the high resolution image according to the reliability of the motion vector of the low resolution image.

  FIG. 5 shows a change in the size of the template region when the reliability of the motion vector detected in the low resolution image is low. 501 is a low-resolution image, 504 is a high-resolution image, 502 and 505 are feature points, and 503 and 506 are template regions.

  Assume that the reliability of the motion vector detected in the low-resolution image is low. In this case, according to FIG. 3A, the template region 506 of the high resolution image is set larger than the template region 503 of the low resolution image. Then, as can be seen from FIG. 5, the angle of view included in the template region 503 is maintained in the template region 506. As a result, motion vector detection can be performed on a high-resolution image while maintaining the amount of texture information included in the template, so that the possibility of detecting a highly reliable motion vector in the high-resolution image can be improved.

  FIG. 6 shows a change in the size of the template region when the reliability of the motion vector detected in the low resolution image is high. 601 is a low-resolution image, 604 is a high-resolution image, 602 and 605 are feature points, and 603 and 606 are template regions.

  Assume that the reliability of the motion vector detected in the low-resolution image is high. In this case, according to FIG. 3C, the template region 606 of the high resolution image is set smaller than the template region 603 of the low resolution image. Then, as can be seen from FIG. 6, the angle of view included in the template region 605 is smaller than the angle of view included in the template region 603. As a result, the template matching in the high-resolution image is less affected by the movement of the peripheral area other than the feature points, so that the motion vector detection accuracy in the high-resolution image can be improved.

  Subsequently, in step S204, the search area setting unit 111 sets a search area for the hierarchical image selected in step S202. The position and size of the search area are preferably set according to the motion vector detected in the lower resolution hierarchical image and its reliability.

  FIG. 7 is a diagram showing the relationship between the reliability of the motion vector and the search area. In the lowest resolution hierarchical image, a search area of a predetermined size may be set so as to evenly include the template area vertically and horizontally.

  In FIG. 7, reference numeral 701 denotes a hierarchical image to be processed, and reference numeral 702 denotes the coordinates of feature points. Reference numerals 703, 706, and 709 denote motion vectors obtained by scaling the motion vectors detected in the lower resolution hierarchical images in accordance with the processing target hierarchical image size. Here, it is assumed that the reliability of the motion vector is vector 703 <vector 706 <vector 709.

  Reference numerals 704, 707, and 710 denote template regions. As described with reference to FIG. 3, the size is set smaller as the reliability of the motion vector of the low-resolution image is higher. Reference numerals 705, 708, and 711 denote search areas. The higher the reliability of the motion vector of the low-resolution image, the smaller the search area. That is, search area 705> search area 708> search area 711.

  The position of the search area is preferably set as follows. If the reliability is low, the center of the search area 705 is set to the position of the start point of the motion vector 703 as shown in FIG. When the reliability is medium, the center of the search area 708 is set to a position between the start point and the end point of the motion vector 706 as shown in FIG. When the reliability is high, the center of the search area 711 is set to the end point position of the motion vector 709 as shown in FIG.

  FIG. 7D is a graph showing the relationship between the reliability and the search area size, with the horizontal axis representing the reliability and the vertical axis representing the search area size. In FIG. 7D, the search area size is changed linearly with respect to the reliability, but may be changed nonlinearly. By setting the search area in this way, the contribution of the motion vector of the low-resolution image in the hierarchical motion vector detection can be controlled according to the reliability of the motion vector.

  In step S205, the motion vector detection unit 106 detects a motion vector using the template region and the search region set in steps S203 and S204.

  The motion vector detection unit 106 detects a motion vector by template matching. FIG. 8 is a diagram showing an outline of template matching. FIG. 8A shows a standard image that is one of the two vector detection images, and FIG. 8B shows a reference image that is the other. The hierarchical image used as the vector detection image is a frame image held in the hierarchical image holding unit 105 and image data directly input from the hierarchical image generation unit 104.

  The motion vector detection unit 106 arranges the template area 801 in the base image and the search area 802 in the reference image, and calculates a correlation value between the template area 801 and the search area 802 (correlation value calculation). In this embodiment, a sum of absolute differences (hereinafter abbreviated as SAD) is used as a correlation value calculation method. A formula for calculating SAD is shown in Formula (5).

  In equation (5), f (i, j) represents the luminance value at the coordinates (i, j) in the template region 801. Further, g (i, j) represents a luminance value at each coordinate in a region 803 for which a correlation value is to be calculated (hereinafter referred to as a correlation value calculation region) in the search region 802. In SAD, the correlation value S_SAD is obtained by calculating the absolute value of the difference between the luminance values f (i, j) and g (i, j) in both regions 802 and 803 and obtaining the sum thereof. The smaller the correlation value S_SAD, the higher the texture similarity between the template area 801 and the correlation value calculation area 803. Note that a method other than SAD may be used to calculate the correlation value, and for example, sum of squared differences (SSD) or normalized cross correlation (NCC) may be used.

  The motion vector detection unit 106 calculates the correlation value by moving the correlation value calculation area 803 over the entire search area 802. Thereby, a correlation value map as shown in FIG. 9 is created for the search area 802.

  In FIG. 9A, the X axis and the Y axis represent coordinates in the correlation value map (that is, in the search area 802), and the Z axis represents the magnitude of the correlation value at each coordinate. FIG. 9B shows the contour lines of FIG. 9A. In FIGS. 9A and 9B, the minimum value 901 has the smallest correlation value, and a texture very similar to the template region 801 is found in the region where the minimum value 901 is calculated in the search region 802. Can be determined to exist. 902 represents the second minimum value and 903 represents the third minimum value, which means that a texture similar to 901 exists.

  As described above, the motion vector detection unit 106 calculates the correlation value between the template region 801 and the search region 802, and determines the position of the correlation value calculation region 803 where the value is the smallest. Thereby, the movement destination on the reference image of the template region 801 on the standard image can be specified. Then, it is possible to detect a motion vector in which the direction and the amount of movement to the movement destination on the reference image based on the position of the template region on the standard image are set as the direction and size.

  In step S206, the reliability calculation unit 107 calculates the reliability of the motion vector obtained in step S205. A two-dimensional correlation value map is used to calculate the reliability of the motion vector. FIG. 10A is a diagram schematically showing a state where the correlation value map as shown in FIG. 9 is viewed from above. In the two-dimensional correlation value map 1001, reference numeral 1002 denotes a position corresponding to the minimum value in FIG. FIG. 10B shows the correlation values in this two-dimensional correlation value map arranged in raster order as indicated by an arrow 1003 and expressed in one dimension. In FIG. 10B, the vertical axis represents the correlation value, and the horizontal axis represents the pixel address uniquely determined by the X coordinate and the Y coordinate of the correlation value map. Hereinafter, this expression of FIG. 10B is used to calculate the reliability of the motion vector. FIG. 11 is a diagram illustrating an example of a correlation value index representing the reliability of a motion vector. The horizontal axis in FIG. 11 is the pixel address, and the vertical axis is the correlation value. In FIG. 11A, the difference Da between the minimum value and the maximum value of the correlation value is used as an index. Da represents the range of the correlation value map. When the value of Da is small, it is considered that the contrast of the texture is low and the reliability is low.

  In FIG. 11B, the ratio Db (= B / A) between the difference A between the minimum value and the maximum value of the correlation value and the difference B between the minimum value and the average value is used as an index. Db represents the steepness of the correlation value peak. When Db is small, it is considered that the similarity between the template area and the search area is low, and the reliability is low.

  In FIG. 11C, the difference Dc between the minimum value of the correlation value and the second minimum value is used as an index. Here, 110, 1102, and 1103 correspond to the correlation values 901, 902, and 903 in FIG. 9, respectively. Therefore, FIG. 11C means that it is confirmed whether or not there is a local minimum similar to the minimum value of the correlation value in the contour line of FIG. 9B. Dc represents the periodicity of the correlation value map. When Dc is small, it is considered that the texture is a repetitive pattern or an edge, and the reliability is low. Although the minimum value and the second minimum value are selected here, it is only necessary to determine the periodicity of the correlation value map, so other minimum values may be selected.

  In FIG. 11D, the minimum correlation value Dd is used as an index. When Dd is large, it is considered that the similarity between the template area and the search area is low, and the reliability is low. Since Dd and the reliability are inversely proportional, the reciprocal of Dd (1 / Dd) is used as an index.

  Although the correlation value index described above can be used as the reliability as it is, for example, the correlation value index and the reliability may be associated as shown in FIG. The horizontal axis of FIG. 12 is a correlation value index (any one of the above-mentioned Da, Db, Dc, 1 / Dd), and the vertical axis is the reliability. In this example, two threshold values T1 and T2 are provided, and if the correlation value index is T1 or less, the reliability is 0, and if the correlation value index is T2 or more, the reliability is 1. The threshold value may be changed for each correlation value index. In the section between the threshold values T1 and T2, the correlation value index and the reliability may be associated non-linearly. In the following description, the reliability obtained from each correlation value index Da, Db, Dc, 1 / Dd is expressed as Ra, Rb, Rc, Rd.

  The final motion vector reliability R may be calculated by combining these Ra, Rb, Rc, and Rd. Here, a combination method based on weighted addition (weighted addition) and priority will be described. In the combination by weighted addition, assuming that the weights of Ra, Rb, Rc, and Rd are Wa, Wb, Wc, and Wd, respectively, the reliability R is calculated as shown in Equation (6).

R = Wa * Ra + Wb * Rb + Wc * Rc + Wd * Rd (6)
For example, the weights are set to Wa = 0.4, Wb = 0.3, Wc = 0.2, and Wd = 0.1. When all the reliability levels are sufficiently high and Ra = Rb = Rc = Rd = 1, R = 1.0 is obtained from the equation (6). When Ra = 0.6, Rb = 0.5, Rc = 0.7, Rd = 0.7, R = 0.6 from equation (6).

  On the other hand, in the combination based on the priority order, the reliability R is determined by giving priority to one of Ra, Rb, Rc, and Rd. For example, in the hierarchical motion vector detection, a motion vector is detected by narrowing down the search area using the motion vector of the low resolution hierarchy, so that an area where the correlation value is small (an area where the similarity between the template area and the search area is high) is selected. You will search intensively. Therefore, even if a correct motion vector can be detected, the correlation value map as a whole has a small correlation value, and although Rd increases, Ra, Rb, and Rc tend to decrease. In particular, Rc using a plurality of minimum values tends to be low. Therefore, for example, the priority is set as Rd> Ra = Rb> Rc. An example of reliability calculation in this case will be described below.

  First, it is determined whether Rd is higher than a predetermined value. If it is higher, the reliability is determined to be 1.0 at that time. If Rd is lower than a predetermined value, it is next determined whether Ra and Rb are higher than a predetermined value. If either is higher, the reliability is determined to be 0.5 at that time. If Ra and Rb are lower than a predetermined value, it is finally determined whether Rc is higher than a predetermined value. If it is higher, the reliability is determined to be 0.3, and if it is lower, the reliability is determined to be 0. To do. In this way, the reliability can be determined in accordance with the magnitude relationship by comparing with a predetermined value in order from the reliability set with a high priority.

  In the last step S207, the motion vector control unit 109 determines whether the processing hierarchy has reached the final hierarchy. If the final hierarchy has not been reached, the process returns to step S202, and steps S202 to S207 are repeated.

  As described above, in the present embodiment, the detection performance of the hierarchical motion vector detection can be improved by changing the size of the template area of each hierarchical image according to the reliability of the motion vector.

<Second Embodiment>
FIG. 13 is a block diagram showing a configuration of a digital camera 20 which is the second embodiment of the image processing apparatus of the present invention. 13 that are the same as those of the digital camera 10 shown in FIG. 1 are denoted by the same reference numerals as those in FIG. Further, among the components of the digital camera 20 of FIG. 13, those corresponding to the components of the digital camera 10 shown in FIG. Is attached. The digital camera of this embodiment also includes an image processing apparatus 200. The image processing apparatus 200 includes an area dividing unit 1301. Below, it demonstrates centering on a different part from 1st Embodiment.

  In the first embodiment, the processing in units of frame images has been described. However, the reliability of motion vectors can vary from region to region. Therefore, in the present embodiment, the hierarchical image is divided into a plurality of image areas, a motion vector is detected for each image area, a reliability is calculated, and a template area and a search area are calculated for each image area based on the reliability. To control.

  FIG. 14 is a flowchart showing the operation of the digital camera of this embodiment. 14, the same symbols as those in FIG. 2 are attached to the same components as those in the flowchart shown in FIG. Further, in FIG. 14, “a” is added to the end of the reference numerals in FIG. 2 corresponding to the flow shown in FIG.

  In step S1401, in the image processing apparatus 200, the region dividing unit (dividing unit) 1301 outputs division information for dividing each hierarchical image into a plurality of image regions to the motion vector control unit 109a. Although the shape of each image area is arbitrary, in this embodiment, as shown in FIG. 15A, the hierarchical image is divided into a grid pattern.

  In step S1402, the motion vector control unit 109 selects a divided region to be processed. It should be noted that region division may be performed after the order of steps S1401 and S201 and steps S1402 and S202 is changed to generate a hierarchical image. In that case, steps S207 and S1403, which will be described later, need to be replaced at the same time.

  In step S203a, the template region setting unit 110a first extracts a predetermined number (multiple) of feature points for each image region based on the division information from the region division unit 1301. Specifically, for each image region, the feature amount of the pixel is calculated by the equation (3) or the equation (4), and a predetermined number of pixels are extracted as feature points from the higher feature amount. FIG. 15B shows a state in which one feature point 1501 is extracted for each image area divided in a grid pattern.

  Furthermore, the template area setting unit 110a determines the position of the template area based on the feature points extracted for each area. Further, based on the reliability of the motion vector detected for each divided region of the low resolution hierarchy, the size of the template region is determined as in the first embodiment.

  In step S204, the search area setting unit 111a sets a search area in the same manner as in the first embodiment, based on the motion vector detected for each divided area of the lower resolution hierarchy and its reliability.

  Finally, in step S1403, the motion vector control unit 109a determines whether or not the processed divided area is the last area. If the last divided area has not been reached, the process returns to step S1402, and steps S1402 to S1403 are repeated.

  In this embodiment, the reliability of the motion vector detected for each image area is calculated, and the template area and the search area can be controlled in units of image areas based on the reliability. Therefore, the hierarchy is further increased than in the first embodiment. The performance of motion vector detection can be improved.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101: Imaging optical system, 102: Image sensor, 103: Camera signal processing unit, 104: Hierarchical image generation unit, 105: Hierarchical image holding unit, 106: Motion vector detection unit, 107: Reliability calculation unit, 108: Motion vector Information holding unit, 109: motion vector control unit, 110: template region setting unit, 111: search region setting unit, 1301: region dividing unit

Claims (15)

Hierarchical image generation means for generating a plurality of different resolution images for each of the standard image and the reference image as input images;
Template area setting means for setting the position and size of the template area for the reference image of each hierarchy generated by the hierarchy image generating means;
Search area setting means for setting the position and size of the search area for the reference image of each hierarchy generated by the hierarchy image generating means;
A motion vector calculating means for calculating a motion vector by calculating a correlation value between the template region and the search region;
Reliability calculation means for calculating the reliability of the motion vector;
With
The template area setting means sets the size of the template area based on the reliability of the motion vector calculated for an image of a lower resolution hierarchy, and the search area setting means has the lower resolution resolution. An image processing apparatus, wherein the position and size of the search area are set based on the reliability of the motion vector calculated for a hierarchical image.   The image processing apparatus further includes a dividing unit that divides the image of each layer into a plurality of image regions, and the template region setting unit includes the template region for each image region based on the reliability of the motion vector calculated for each image region. The position and size of the search area are set for each image area based on the reliability of the motion vector calculated for each image area. Image processing apparatus.   The image processing apparatus according to claim 1, wherein the reliability calculation unit calculates the reliability of the motion vector based on a correlation value calculation result obtained from the motion vector calculation unit.   The image processing apparatus according to claim 3, wherein the reliability calculation means calculates the reliability of a final motion vector by weighted addition of the reliability of a plurality of motion vectors.   The image processing apparatus according to claim 3, wherein the reliability calculation unit calculates a reliability of a final motion vector according to a priority of reliability of a plurality of motion vectors.   The image processing apparatus according to claim 3, wherein the reliability calculation unit calculates the reliability of the motion vector higher as the maximum correlation value obtained from the motion vector calculation unit increases.   The reliability calculation unit calculates the reliability of the motion vector higher as the difference between the maximum value and the minimum value of the correlation values obtained from the motion vector calculation unit increases. The image processing apparatus described.   The reliability calculation means increases the ratio of the difference between the maximum value and the average value of the correlation values to the difference between the maximum value and the minimum value of the correlation values obtained from the motion vector calculation means. The image processing apparatus according to claim 3, wherein the reliability is calculated to be high.   The reliability calculation means calculates the reliability of the motion vector higher as the difference between the maximum correlation value obtained from the motion vector calculation means and the minimum value of at least one correlation value increases. The image processing apparatus according to claim 3.   As the reliability of the motion vector becomes higher, the template area setting means sets the size of the template area for the reference image of the next layer to be processed than the size of the template area for the reference image of the previous layer processed. The image processing apparatus according to claim 1, wherein the image processing apparatus is set to be smaller.   As the reliability of the motion vector increases, the search area setting means sets the size of the search area for the reference image of the next layer to be processed from the size of the search area for the reference image of the previous layer processed. The image processing apparatus according to claim 1, wherein the image processing apparatus is set to be smaller.   As the reliability of the motion vector increases, the search region setting means moves the position of the search region with respect to a reference image of the next layer to be processed further from the center of the template region along the direction of the motion vector. The image processing apparatus according to claim 1, wherein the position is set to a position. A hierarchical image generation step of generating images of each hierarchy of a plurality of different resolutions for each of the standard image and the reference image as an input image;
A template region setting step for setting the position and size of the template region for the reference image of each layer generated by the layer image generation step;
A search region setting step for setting the position and size of the search region for the reference image of each layer generated by the layer image generation step;
A motion vector calculating step of calculating a motion vector by calculating a correlation value between the template region and the search region;
A reliability calculation step of calculating the reliability of the motion vector;
Have
In the template area setting step, the size of the template area is set based on the reliability of the motion vector calculated for an image of a lower resolution hierarchy, and in the search area setting step, the lower resolution is set. An image processing method comprising: setting a position and a size of the search area based on a reliability of the motion vector calculated for a hierarchy image.   A program for causing a computer to execute each step of the image processing method according to claim 13.   A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to claim 13.

Images