JP2019020839A - Image processing apparatus, image processing method and program - Google Patents

Abstract

To allow processing to be completed within a prescribed period of time with less degradation in vector detection accuracy in the case of maldistribution of tracked feature points within an image.SOLUTION: A first RDDMAC (221) divides an image into a plurality of regions. A new feature point calculation unit (202) extracts feature points in each of the divided regions. A matching processing unit (203) detects a motion vector value on the basis of the feature points. A tracked feature point determination unit (205) calculates tracked feature points for use in motion vector detection for a next frame on the basis of a motion vector value obtained from a preceding frame and the feature points. If the tracked feature points meet a preliminarily determined condition, the tracked feature point determination unit (205) discards at least a part of the tracked feature points and supplements new feature points as tracked feature points.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for detecting image motion.

  In order to perform camera shake correction on video shot using an imaging device such as a digital still camera or a digital video camera, the amount of motion between frame images is detected and alignment is performed on a plurality of images. There is a need. As a method for detecting the amount of motion between frame images, there are a method of using information of an external device such as a gyro sensor, a method of estimating the amount of motion from a captured frame image, and the like.

  Various methods for estimating the amount of motion using a frame image have been proposed in the past. A typical example is motion vector detection by template matching. In template matching, first, one of two consecutive frame images in a video is used as an original image, and the other is used as a reference image. Then, a rectangular area having a predetermined size arranged on the original image is used as a template block, and a correlation with a distribution of pixel values in the template block is obtained at each position of the reference image. At this time, the position where the correlation is highest in the reference image is the destination of the template block, and the direction and amount of movement to the destination when the position of the template block on the original image is used as a reference is the motion vector. .

  In order to improve the detection rate of motion vectors, there is a technique for extracting feature points, placing template blocks at the extracted feature points, and performing template matching between frame images. Here, if feature points are extracted from the entire image, the distribution of feature points is often non-uniform. When a motion vector obtained for feature points having a non-uniform distribution is used for the purpose of camera shake correction, a region where the feature points are concentrated is the main camera shake correction.

Therefore, in order to distribute the feature points uniformly, the image is divided into grids, feature values representing the feature size are calculated for each pixel, and the pixel with the largest feature value in each grid is extracted as the feature point. There is a technique (for example, Patent Document 1).
Furthermore, there is a technique for tracking feature points in order to improve the detection rate of motion vectors. Feature point tracking can be realized by sequentially detecting motion vectors of feature points extracted from an image over a plurality of continuous frame images (for example, Patent Document 2).

JP 2008-192060 A JP 2007-334625 A

  By the way, when the feature point tracking process is performed as described above, the feature point of the tracking destination may be unevenly distributed in a part of the region in the image. Thus, if the characteristic points of the tracking destination are unevenly distributed in a part of the region, there may be a problem that the motion vector detection performance is deteriorated and the processing cannot be completed within a predetermined period.

  Therefore, an object of the present invention is to complete a process within a predetermined period while suppressing a decrease in vector detection accuracy when feature points of tracking destinations are unevenly distributed in an image.

  The present invention relates to a dividing unit that divides a frame image into a plurality of regions, an extracting unit that extracts image feature points in each of the divided regions, and a motion vector is detected based on the feature points. Detection means, calculation means for calculating a tracking destination feature point used for motion vector detection of the next frame based on the value and feature point of the motion vector detected from the previous frame, and the tracking destination feature point Compensation means for discarding at least a part of the tracking destination feature points and filling the new feature points extracted by the extraction means as tracking destination feature points when a predetermined condition is met. It is characterized by.

  According to the present invention, when the feature point of the tracking destination is unevenly distributed in the image, it is possible to complete the processing within a predetermined period while suppressing a decrease in vector detection accuracy.

It is a block diagram of a digital camera provided with the image processing apparatus of this embodiment. It is a block diagram of the vector detection part of 1st Embodiment. It is explanatory drawing of a grid arrangement | positioning, a feature point, and a template matching area | region. It is explanatory drawing of the outline | summary of a feature point tracking process. It is explanatory drawing of a template matching process timing. It is explanatory drawing of the example of a delay of a template matching process timing. It is a process flowchart of the vector detection part of 1st Embodiment. It is a block diagram of the new feature point calculation unit of the first embodiment. It is a figure used for description of the determination process in the precision determination part of 1st Embodiment. It is a block diagram of the tracking destination feature point determination unit of the first embodiment. It is a figure which shows the positional relationship of a tracking destination feature point and a new feature point. It is a process flowchart of the tracking destination feature point determination part of 1st Embodiment. It is a figure which shows the process timing in an image process part. It is a block diagram of the vector detection part of 2nd Embodiment. It is a block diagram of the tracking destination feature point determination unit of the second embodiment. It is a figure used for description of the process of a range determination part. It is a process flowchart of the tracking destination feature point determination part of 2nd Embodiment. It is a figure used for description of a process of a range determination part and a new feature point compensation part. It is a block diagram of the vector detection part of 3rd Embodiment. It is a figure used for description of the compensation area | region division in 3rd Embodiment. It is a block diagram of the tracking destination feature point determination unit of the third embodiment. It is a figure used for description of the positional relationship of two tracking destination feature points. It is a process flowchart of the tracking destination feature point determination part of 3rd Embodiment. It is a block diagram of the tracking destination feature point determination part of 4th Embodiment. It is a figure which shows the positional relationship of the tracking destination feature point in 4th Embodiment. It is a process flowchart of the tracking destination feature point determination part of 4th Embodiment. It is a figure which shows the process timing in 4th Embodiment. It is a block diagram of the matching process part of 5th Embodiment. It is a process flowchart of a matching process part. It is a figure used for description of the process of the area | region division part of 5th Embodiment. It is a figure used for description of the process of a size change part and an estimation part.

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.
<First Embodiment>
FIG. 1 is a diagram illustrating a schematic configuration example of a digital camera as an application example of the image processing apparatus according to the embodiment. In addition to the digital camera, the image processing apparatus of the present embodiment includes, for example, a digital video camera, various portable terminals such as a smartphone and a tablet terminal having a camera function, an industrial camera, an in-vehicle camera, a medical camera, and the like. The present invention can also be applied to various imaging devices. FIG. 2 is a diagram illustrating a schematic configuration of a motion vector detection unit included in the image processing unit 104 according to the image processing apparatus of the present embodiment. FIG. 2 includes other configurations related to vector detection.

  In FIG. 1, the imaging optical unit 101 includes a lens, a diaphragm, and the like, and performs focus adjustment and exposure adjustment at the time of shooting to form an optical image on the image sensor 102. The image sensor 102 has a photoelectric conversion function for converting an optical image into an electrical signal (analog image signal), and is configured by a CCD, a CMOS sensor, or the like. The A / D conversion unit 103 converts an analog image signal from the image sensor 102 into digital image data.

  The DRAM 107 is a memory for storing data, and has a sufficient storage capacity for storing a predetermined number of still images, moving images for a predetermined time, data such as sound, constants for operating the CPU 112, programs, and the like. . The memory control unit 106 performs data writing to and data reading from the DRAM 107 in accordance with instructions from the CPU 112 or the data transfer unit 105.

  The nonvolatile memory control unit 108 writes and reads data to and from the ROM 109 in accordance with instructions from the CPU 112. The ROM 109 is an electrically erasable / recordable nonvolatile memory, and an EEPROM or the like is used. The ROM 109 stores constants for operating the CPU 112, programs, and the like.

  The CPU 112 is configured by a microcomputer or the like that controls the entire digital camera, and instructs each functional block to execute various control processes. The CPU 112 controls the image processing unit 104, the data transfer unit 105, the memory control unit 106, the nonvolatile memory control unit 108, the display control unit 110, the operation unit 113, the image sensor 102, and the like via the bus 114. In addition, the CPU 112 stores settings in the ROM 109 as settings for reading out analog signals to the A / D conversion unit 103 by controlling the image sensor 102 as reading order settings. Further, the CPU 112 can read the reading order setting recorded in the ROM 109 and control various image processes of the image processing unit 104 according to the setting. The microcomputer of the CPU 112 implements control of each process according to the present embodiment by executing a program recorded in the ROM 109. The bus 114 is a system bus, and the bus 115 is an image data bus.

  The display unit 111 includes a liquid crystal monitor and is controlled by the display control unit 110 to display various images on the display unit 111. The operation unit 113 includes switches and buttons operated by the user, and is used for operations such as power ON / OFF and shutter ON / OFF.

  The image processing unit 104 includes a configuration for performing various types of image processing, a buffer memory, and the like. The various types of image processing include vector detection processing performed by the motion vector detection unit illustrated in FIG. The description of the vector detection unit shown in FIG. 2 will be described later. The data transfer unit 105 includes a plurality of DMACs (Direct Memory Access Controllers) that perform data transfer, and writes and reads data to and from the DRAM 107 via the bus 115.

The outline of the motion vector detection process will be described below.
As the motion vector detection process, a motion vector detection process using template matching as described above is known. In the motion vector detection process, in order to improve the vector detection rate, as described above, a process of extracting feature points by dividing an image into a grid shape, a process of tracking feature points, and the like are performed.

  FIG. 3 is a diagram used for explaining the relationship between the grid arrangement, the feature points, and the template matching area in the vector detection process. In the vector detection process, the frame image is divided into a plurality of grids. Each grid is arranged as a feature extraction grid 302 and a peripheral grid 301 (grid with a dot pattern in FIG. 3) having a preset size in the number set in the horizontal and vertical directions. One feature point 303 is extracted for each feature extraction grid 302. The peripheral grid 301 is arranged around a feature extraction grid group including the feature extraction grids 302. The peripheral grid 301 is used for template matching, but feature points are not extracted. In the template matching process, a rectangular search area 305 and a template area 304 having a predetermined size are set around the feature point 303. Note that the search area 305 and the template area 304 in the template matching process are sequentially set in accordance with the so-called raster scan order in the sensor output of the image sensor 102.

  FIG. 4A is a diagram used for explaining the outline of the tracking process using the extracted feature points. In FIG. 4A, in the feature point tracking process, first, the feature extraction grid 1010 and the peripheral grid 1011 described above are set for the image of the current frame. A feature point 1001 is extracted in the feature extraction grid 1010, and a search region 1005 and a template region 1003 are set and template matching is performed, whereby a vector value 1004 of a motion vector is calculated. In the processing of the next frame, the vector value 1004 calculated in the previous frame is added to the feature point 1001, thereby calculating a feature point that is a tracking destination (referred to as a tracking destination feature point 1002), and the tracking destination feature. Template matching is performed around the point 1002. Thereafter, processing similar to that described above is performed in order over a plurality of frames, and vector values are sequentially added to the tracking destination feature points calculated sequentially in each frame, thereby realizing tracking of the feature points.

  FIG. 5A is a diagram illustrating a timing chart from the image output of the image sensor 102 (hereinafter referred to as sensor output) to the end of the template matching process. FIG. 5B illustrates a feature extraction grid 1010, a peripheral grid 1011, a search area 1005, a template area 1003, and a tracking destination feature point 1002 similar to those in FIG. FIG. 5B also shows an arrow representing a so-called raster scan 1020 and the first (first) template position 1101 at which the template matching process can be started. In FIG. 5A, the image output (sensor output) of the image sensor 102 is performed within the cycle (frame cycle) of the pulse interval of one frame of the vertical synchronization signal VD, and template matching is performed based on the image data of the sensor output. A business image is generated. Generation of the template matching image is started in accordance with the sensor output in the raster scan order, and is completed at a timing delayed by the time required for the image generation processing. The template matching process is performed using a template matching image. Each period T1 to T8 in FIG. 5A represents the time required for the template matching process corresponding to the tracking destination feature point. . Note that the time required for the template matching process increases or decreases according to the size of the template area.

  In the example of FIG. 5B, only eight feature extraction grids 1010 are illustrated, and one tracking destination feature point 1002 is obtained for each of the feature extraction grids 1010. For this reason, in FIG. 5A, eight periods T1 to T8 are shown as the time required for the template matching process for each of the eight tracking destination feature points 1002. Further, as shown in FIG. 5B, the template matching process in the period of one frame is started from the template position 1101 set for the tracking destination feature point 1002 that is the first (first) in the raster scan order. The The template matching process is completed when all the processes in the periods T1 to T8 corresponding to the eight tracking destination feature points 1002 are completed.

  By the way, when the feature point tracking process as described above is performed, a plurality of tracking destination feature points may be unevenly distributed in, for example, a partial region within the image range of the frame. FIG. 4B shows an example in which the tracking destination feature points are unevenly distributed so as to be concentrated in the lower region within the image range of the frame. As shown in FIG. 4B, the state in which the tracking target feature points are unevenly distributed so as to be concentrated in the lower region in the image range of the frame occurs, for example, when the camera is tilted during photographing with a digital camera. There are many. 4B also shows a feature extraction grid 1010, a peripheral grid 1011, a search area 1005, a template area 1003, and a tracking destination feature point 1002 similar to those in FIG.

  FIG. 6A is a diagram showing a timing chart from sensor output to the end of the template matching process, as in FIG. 5A, and a large number of tracking destination features in the lower area as shown in FIG. 4B. The processing timing when points are concentrated is shown. FIG. 6B is a diagram showing a feature extraction grid 1010, a peripheral grid 1011, a search region 1005, a template region 1003, and a tracking destination feature point 1002, which are the same as those in FIG. FIG. 6B also shows an arrow representing the raster scan 1020 and the first (first) template position 1201 at which the template matching process can be started, as in FIG. 5B. In the example of FIG. 6B, the eight tracking destination feature points 1002 are unevenly distributed so as to be concentrated in the lower region within the image range of the frame. Then, the eight periods T1 to T8 in FIG. 6A represent the time required for the template matching process for each of the eight tracking destination feature points 1002 in FIG.

  Since the template matching process is performed in accordance with the raster scan order of the sensor output as described above, the lower area in the image range of the frame is an area that is processed at the end in the raster scan order. When many tracking destination feature points are concentrated in the lower region as shown in FIG. 4B and FIG. 6B, as shown in FIG. 6A, within the period of the pulse interval of the vertical synchronization signal VD, It may become impossible to complete all template matching processes in the periods T1 to T8. That is, when the tracking destination feature points are concentrated in the lower area, as shown in FIG. 6B, the first template position 1201 at which the template matching process is started is a position corresponding to the last stage in the raster scan order. Then, when the template matching process starts from the template position 1201 corresponding to the last stage of the raster scan order, as shown in FIG. 6A, the period 1200 until the start of the processes in the periods T1 to T8 becomes longer. For this reason, before the processing in the periods T1 to T8 is completed, the next pulse of the vertical synchronization signal VD may come and the processing may not be completed.

Here, when a motion vector is used for camera shake correction at the time of shooting as in the digital camera of the present embodiment, the motion vector is detected by completing the template matching process within one frame period, and the motion vector is detected as the next frame. It is necessary to feed back to the process.
In the example of FIGS. 5A and 5B described above, all the processes in the periods T1 to T8 are completed within one frame period, and the template matching process is completed. For this reason, the motion vector can be fed back to the processing of the next frame, and as a result, camera shake correction at the time of photographing with the digital camera can be performed correctly.
On the other hand, if the template matching process cannot be completed within the period of one frame as in the examples of FIGS. 6A and 6B, the correct motion vector can be fed back to the process of the next frame. Disappear. In this case, the vector detection performance is lowered, and as a result, the camera shake correction performance at the time of shooting of a digital camera or the like is also lowered.

  Therefore, in the vector detection unit according to the first embodiment, when tracking destination feature points are unevenly distributed within the image range of the frame, tracking destination feature point discard compensation processing according to the number of tracking destination feature points in the predetermined region is performed. As a result, the processing is completed within a predetermined period such as one frame period.

  FIG. 7 is a flowchart showing a flow of processing in the vector detection unit shown in FIG. 2 and the related configuration according to the present embodiment. Note that the processing of the flowchart of FIG. 7 may be executed only by a software configuration or only by a hardware configuration, or a part may be realized by a software configuration and the rest by a hardware configuration. When the process is executed by the software configuration, the process of the flowchart in FIG. 7 may be realized by the CPU 112 executing a program stored in the ROM 109, for example. The program according to the present embodiment may be prepared in advance in the ROM 109, read from a detachable semiconductor memory, or downloaded from a network such as the Internet (not shown). In the following description, steps S401 to S409 of each process in FIG. 7 are abbreviated as S401 to S409. The same applies to other flowcharts described later.

Hereinafter, an outline of processing performed by the vector detection unit shown in FIG. 2 and related configurations will be described with reference to the flowchart of FIG.
First, as the processing of S401, the first RDDMAC 221 provided in the data transfer unit 105 in FIG. 1 reads out the image data 241 of the current frame that is the target of vector detection processing from the DRAM 107 via the bus 115. In the following description, the description that the data transferred from the data transfer unit 105 passes through the bus 115 is omitted. The data read at this time is data in units of grids as described above, and data is read for each of the feature extraction grid 302 and the peripheral grid 301 shown in FIG. It is assumed that the image data 241 is subjected to various other image processing by the image processing unit 104. The image data 241 read out in grid units by the first RDDMAC 221 is sent to the matching image generation unit 201 and the new feature point calculation unit 202. Also, the first RDDMAC 221 outputs grid coordinate information 252 indicating the read grid coordinate position to the tracking destination feature point determination unit 205 described later. After S401, the processing of the vector detection unit transitions to S402.

  As processing of S402, the matching image generation unit 201 generates a template matching image used for template matching at the time of vector detection from the image data 241 read out in grid units by the first RDDMAC 221. The matching image generation unit 201 includes, for example, a band-pass filter circuit, and cuts high-frequency components and low-frequency components of image signals that are unnecessary for template matching processing from the image data 241. Then, the matching image generation unit 201 outputs the template matching image generated for the current frame to the WRDMAC 231 provided in the data transfer unit 105. After S402, the processing of the vector detection unit transitions to S403.

  As the process of S403, the WRDMAC 231 performs a process of writing the template matching image data 242 generated in the current frame into the DRAM 107. It is assumed that the template matching image data 243 generated in the previous frame is also stored in the DRAM 107 at this time. After S403, the processing of the vector detection unit transitions to S405.

In parallel with the processing of S402 and S403 described above, as the processing of S404, the new feature point calculation unit 202 calculates a new feature point for each feature extraction grid from the image of the current frame. A new feature point is a feature point newly extracted for each feature extraction grid from the image of the current frame, unlike the tracking destination feature point calculated by adding the motion vector values described above. After S404, the processing of the vector detection unit transitions to S405.
FIG. 8 is a block diagram illustrating a schematic configuration example of the new feature point calculation unit 202. The new feature point calculation unit 202 includes a feature filter unit 501, a feature evaluation unit 502, and a feature point determination unit 503.

  The feature filter unit 501 includes a plurality of filters such as a band pass filter, a horizontal differential filter, a vertical differential filter, and a smoothing filter. The feature filter unit 501 cuts unnecessary high-frequency components and low-frequency components from the image data 241 by using a bandpass filter, and further performs horizontal differential filter processing and vertical direction processing on the image data after the bandpass filter processing. The differential filter processing is performed. The feature filter unit 501 performs a smoothing filter process on each of the image data that has been subjected to the differential filter process in the horizontal direction and the image data that has been subjected to the differential filter process in the vertical direction, and the feature evaluation unit 502 Output to.

  The feature evaluation unit 502 calculates a feature value from the image data in units of grids after the filter processing by the feature filter unit 501. Specifically, the feature evaluation unit 502 performs a calculation using a predetermined feature evaluation formula for each pixel, and a differential value around the pixel such as an intersection of two edges or a curved point having a maximum curvature. A point where is large in multiple directions is calculated as a feature point.

Hereinafter, for example, a so-called Shi and Tomasi technique will be described as an example, and a feature value calculation example in the feature evaluation unit 502 of this embodiment will be described.
First, the feature evaluation unit 502 uses the data subjected to the above-described horizontal differential filter processing and the data subjected to the vertical differential filter processing to calculate the autocorrelation matrix by the arithmetic expression shown in Expression (1). Create H. In Expression (1), Ix represents the result of applying the horizontal differential filter processing, and Iy represents the result of applying the vertical differential filter, which is an arithmetic expression for convolving the Gaussian filter G.

Then, the feature evaluation unit 502 calculates the feature value using the Shi and Tomasi feature evaluation formula shown in Formula (2). Expression (2) represents that the smaller eigenvalue of the eigenvalues (λ1, λ2) of the autocorrelation matrix H of Expression (1) is used as the feature value.
Shi and Tomasi = min (λ1, λ2) Equation (2)
The feature value calculated for each pixel by the feature evaluation unit 502 in this way is sent to the feature point determination unit 503.

  The feature point determination unit 503 newly determines a pixel having the largest value among the feature values calculated for each pixel in the grid by the feature evaluation unit 502 as a feature point in the grid. In the case of the present embodiment, the feature point information uses relative coordinates (PX, PY) with the upper left corner of the grid as coordinates (0, 0). Note that the coordinates of the feature points may be expressed by coordinates in the frame image. The feature point coordinate information newly determined by the feature point determination unit 503 is stored in an internal memory provided in the new feature point calculation unit 202. This internal memory has a capacity sufficient to store each feature point coordinate information acquired from the previous frame and each feature point coordinate information acquired from the current frame.

The new feature point calculation unit 202 reads out the feature point coordinate information 251 calculated for the previous frame and the current frame from the internal memory when the matching template 203 starts the corresponding grid template matching process. The result is output to the tracking destination feature point determination unit 205.
The tracking destination feature point determination unit 205 outputs the tracking destination feature point coordinate information 257 to the second RDDMAC 222 included in the data transfer unit 105.

Returning to the flowchart of FIG. 7 and FIG.
As the processing of S405, the second RDDMAC 222 reads out data of a rectangular area having a preset size centering on the tracking destination feature point coordinate information 257 from the template matching image data 242 and 243 in the DRAM 107. That is, from the template matching image data 242 generated in the current frame, the data in the search area 305 in FIG. 3 is stored, while from the template matching image data 243 generated in the previous frame, the data in the template area 304 is stored. Read out. Then, the read search area image data 253 and template area image data 254 are sent to the matching processing unit 203. However, in the first template matching process, since the tracking destination feature point coordinates cannot be calculated from the previous frame, processing using the new feature point coordinates calculated in the current frame is performed. After S405, the vector detection unit transitions to S406.

  As processing of S406, the matching processing unit 203 calculates a correlation value using the input search area image data 253 and template area image data 254, and calculates a vector value based on the correlation value. After S406, the process of the vector detection unit transitions to S407.

In the present embodiment, the sum of absolute difference (hereinafter abbreviated as SAD) is used as a correlation value calculation method. Equation (3) is a formula for calculating SAD.

  In Expression (3), f (i, j) represents the pixel value of the coordinates (i, j) in the template area image data 254, and g (i, j) represents the correlation value calculation in the search area image data 253. Each pixel value in the target area (hereinafter referred to as a correlation value calculation area) is represented. The correlation value calculation area is the same size as the area in the template area image data 254. Then, the matching processing unit 203 calculates the absolute value of the difference for each pixel value f (i, j) and g (i, j) in those regions, and calculates the correlation value S_SAD by obtaining the sum of them. Here, the smaller the value of correlation value S_SAD, the smaller the difference in luminance value between the two regions, that is, the template region image data 254 and the texture in the correlation value calculation region are similar. In the present embodiment, SAD is used as an example of the correlation value, but the present invention is not limited to this. For example, other correlation values such as sum of squares of differences (SSD) and normalized cross-correlation (NCC) may be used. May be used.

  Then, the matching processing unit 203 obtains the position of the minimum value of the correlation value as the grid motion vector value. The matching processing unit 203 outputs the vector value and the correlation value information 255 calculated as described above to the accuracy determining unit 204.

  As the process of S407, the accuracy determination unit 204 obtains the maximum value, the minimum value, the average value, and the minimum value of the correlation values using the correlation value among the vector values and the correlation value information 255 calculated in the process of S406. . Then, the accuracy determination unit 204 uses the maximum value, the minimum value, the average value, and the minimum value of the correlation values to perform low contrast determination, pixel value maximum value protrusion determination, and repeated pattern determination processing as described below. I do. After S407, the processing of the vector detection unit transitions to S408.

FIG. 9A to FIG. 9D are graphs used for explaining the relationship between the maximum value, the minimum value, the average value, and the minimum value of the correlation values and the determination example using them. However, in this embodiment, the smaller the correlation value is, the higher the similarity is. Therefore, in FIGS. 9A to 9D, the maximum value of the pixel value is the minimum value of the correlation value, and the pixel value of FIG. The minimum value represents the maximum value in the correlation value, and the maximum value of the pixel value represents the minimum value in the correlation value.
FIG. 9A shows an example in which each determination result is a good result.

  FIG. 9B shows an example in which low contrast is determined in low contrast determination. The low contrast determination is a process for determining whether or not the correlation value calculation area has low contrast. In the low contrast determination in the present embodiment, when the difference between the maximum value and the minimum value of the correlation value in the correlation value calculation area is smaller than a predetermined threshold value, the correlation value calculation area has a low contrast. It is determined that In the case of FIG. 9B, the difference between the maximum value and the minimum value of the pixel value is small compared to FIG. 9A, and the difference between the maximum value and the minimum value of the correlation value in the correlation value calculation region is also a predetermined value. It is assumed that it is smaller than the threshold value. Therefore, in the case of FIG. 9B, the accuracy determination unit 204 has a low contrast in the correlation value calculation area because the difference between the maximum value and the minimum value of the correlation value in the correlation value calculation area is smaller than the predetermined threshold value. Judge that there is.

  FIG. 9C shows an example in which it is determined that the peak is low in the maximum value protrusion determination of the pixel value. The pixel value maximum value protrusion determination is a process for determining how much the minimum value of the correlation value in the correlation value calculation region stands out. In the maximum value protrusion determination of the pixel value in the present embodiment, the value of the division result of the difference between the maximum value and the average value of the pixel values and the difference between the maximum value and the minimum value of the pixel values is set in advance. When it is smaller than the predetermined threshold, it is determined that the correlation value calculation region has a low peak. On the other hand, when the value of the division result is larger than the predetermined threshold value, it is determined that the correlation value calculation area has a high peak. In the case of FIG. 9C, compared with FIG. 9A, the value of the division result between the difference between the maximum value and the average value of the pixel values and the difference between the maximum value and the minimum value of the pixel values is small. It is assumed that it is smaller than the threshold value. Therefore, in the case of FIG. 9C, the accuracy determination unit 204 determines that the correlation value calculation region has a low peak because the value of the division result is smaller than the predetermined threshold value.

  FIG. 9D shows an example in which a repetitive pattern is determined as a repetitive pattern. The repetitive pattern determination is a process for determining whether or not the pixel value fluctuates repeatedly. In the repetitive pattern determination in the present embodiment, a repetitive pattern is determined when the difference between the maximum value and the maximum value of the pixel values in the correlation value calculation area is smaller than a predetermined threshold value set in advance. . In the case of FIG. 9D, it is assumed that the difference between the maximum value and the maximum value of the pixel values is smaller than that in FIG. Therefore, in the case of FIG. 9D, the accuracy determination unit 204 determines that the pattern is a repetitive pattern because the difference between the maximum value and the maximum value of the pixel values in the correlation value calculation region is smaller than the predetermined threshold value.

Returning to the flowchart of FIG. 7 and FIG.
As the processing of S408, the accuracy determination unit 204 writes the above-described determination information of low contrast, low peak, and repetitive pattern, and vector information 256 into the SRAM 206. As a result, each information is stored in the storage area 244 of the SRAM 206. After S408, the processing of the vector detection unit transitions to S409.

In step S409, the tracking destination feature point determination unit 205 calculates a tracking destination feature point to be used for template matching processing of the next frame.
Here, FIG. 10 is a block diagram illustrating a schematic configuration example of the tracking destination feature point determination unit 205 of FIG. As illustrated in FIG. 10, the tracking destination feature point determination unit 205 includes a count unit 701, a count number comparison unit 702, and a new feature point compensation unit 703. FIG. 10 also shows another configuration (new feature point calculation unit 202, SRAM 206, second RDDMAC 222) related to the tracking destination feature point determination process in the tracking destination feature point determination unit 205. FIG. 11 is a diagram showing the positional relationship between the tracking destination feature point and the new feature point processed by the tracking destination feature point determination unit 205. Further, FIG. 12 is a flowchart of processing of the tracking destination feature point determination unit 205 according to the present embodiment. FIG. 13 is a diagram illustrating a processing timing chart in the image processing apparatus according to the present embodiment.

  The outline of the tracking destination feature point determination process in the tracking destination feature point determination unit 205 will be described below with reference to FIGS. In the grid arrangement of FIG. 11, an example in which the number of feature extraction grids is 25 is shown. Accordingly, the new feature point calculation unit 202 calculates 25 new feature point coordinates, which are the same number as the number of grids, and feature values at these coordinates. Similarly, in the template matching process, it is assumed that 25 vector values equal to the number of grids are calculated around each tracking destination feature point coordinate. Also, in FIG. 11, each feature point 901, 902, 903 represents a tracking destination feature point, and each feature point 901 represents a new feature point to be compensated as a tracking destination feature point, as will be described later. Assume that reference numeral 903 denotes a tracking destination feature point that is discarded as will be described later.

  First, as the processing of S <b> 901 in FIG. 12, the tracking destination feature point determination unit 205 acquires a vector value and the above-described determination information 258 from the SRAM 206. The tracking destination feature point determination unit 205 calculates the tracking destination feature point of the current frame using the acquired vector value, and further performs template matching processing centered on the tracking destination feature point coordinates to add the calculated vector value. The coordinates of the tracking destination feature point of the next frame are calculated. In the example of the grid arrangement shown in FIG. 11, a total of 25 tracking destination feature point coordinates are calculated. After S901, the process of the tracking destination feature point determination unit 205 transitions to S902.

  Next, as the processing of S902, the tracking destination feature point determination unit 205 can use the tracking destination feature point calculated in S901 for template matching processing of the next frame based on the above-described determination information acquired from the SRAM 206. It is determined whether the feature point is a valid tracking destination feature point. The tracking destination feature point determination unit 205 determines that the low contrast is determined by the low contrast determination, the low peak is determined by the maximum value protrusion determination of the pixel value, or the repeated pattern determination determines that the pattern is a repeated pattern. It is determined that the feature point is not valid (NG determination). On the other hand, the tracking destination feature point determination unit 205 determines that the tracking destination feature point is valid (OK determination) when none of the low contrast, the low peak, and the repetitive pattern. Then, in S902, the tracking destination feature point determination unit 205 advances the process to S904 when the determination is NG (NO), and advances the process to S903 when the determination is OK (YES).

  As the processing of S903, the tracking destination feature point determination unit 205 determines whether or not the tracking destination feature point coordinates calculated in S901 are within the image range (angle of view) of the frame. Specifically, the tracking destination feature point determination unit 205 makes an OK determination when the template region centered on the tracking destination feature point coordinates is within the peripheral grid, and when the template region is not within the peripheral grid. Is judged as NG. Then, in S903, the tracking destination feature point determination unit 205 advances the process to S904 if the determination is NG (NO), and advances the process to S905 if the determination is OK (YES).

  When the processing proceeds to S904, the tracking destination feature point determination unit 205 deletes the tracking destination feature point coordinates corresponding to the coordinate position determined as NG in S902 or S903. Then, the tracking destination feature point determination unit 205 uses the new feature calculated at the corresponding position of the feature point extraction grid at the position where the new feature point is calculated first in the current frame as the tracking destination feature point coordinates. Performs replacement with point coordinates. After S904, the process of the tracking destination feature point determination unit 205 transitions to step S905.

  In step S905, the tracking destination feature point determination unit 205 determines whether all tracking destination feature point coordinates for the number of feature extraction grids have been calculated. If the tracking destination feature point determination unit 205 determines in step S905 that all the tracking destination feature point coordinates have been calculated (YES), the processing proceeds to step S906. On the other hand, if the tracking destination feature point determination unit 205 determines in S905 that all the tracking destination feature point coordinates have not been calculated (NO), the process returns to S901 and the above-described processing is performed. The processing from S901 to S905 is repeated until it is determined (YES) that all the tracking destination feature point coordinates have been calculated in S905. In the case of the example in FIG. 11, since the calculation processing of 25 tracking destination feature point coordinates is performed, the processing from S901 to 905 is repeated 25 times.

  Next, when proceeding to S906, the count unit 701 included in the tracking destination feature point determination unit 205 counts how many tracking destination feature point coordinates are in a predetermined area section. In the case of the present embodiment, each feature extraction grid group in FIG. 11 is divided into an area section below the broken line 957 in the figure and an area section above the broken line 954 in the figure. Then, the counting unit 701 counts the number of tracking destination feature point coordinates in the area section below the broken line 957 (hereinafter referred to as the counting area section) as the predetermined area section. That is, the counting area section below the broken line 957 is an area that is processed in the last stage in the raster scan order. When a large number of tracking destination feature points are concentrated in this area, the template matching process is performed within one frame period. It is an area that may not be able to be completed. In the example of FIG. 11, the total number of tracking destination feature point coordinates in the counting area section is 17. As will be described later, these 17 tracking destination feature points are divided into a tracking destination feature point 903 to be discarded later and a tracking destination feature point 902 that is not to be discarded. Details of the area section above the broken line 954 in FIG. 11 will be described later.

  Details of the processing in the tracking destination feature point determination unit 205 will be described with reference to the timing chart of FIG. Times 1953 to 1959 in FIG. 13 represent timings at which the areas indicated by broken lines 953 to 959 in FIG.

  Here, the time required to perform the template matching process for one template region is uniquely determined by the size of the template region. Each period represented by each break in the period 920 in FIG. 13 represents a processing period required for each template matching process. In the case of the above-described example of FIG. 11, the total number of tracking destination feature point coordinates in the counting area section below the broken line 957 is 17, and thus 17 template areas are set. Therefore, the number of divisions (the number of processing periods) in the period 920 in FIG. 13 is 17 corresponding to the number of template matching processes for the 17 template regions.

  On the other hand, the entire template matching process for one frame must be completed within the period of the pulse interval of the vertical synchronization signal VD (within one frame period), that is, before the processing of the next frame is started. In the case of the timing example in FIG. 13, the processing can be completed before the processing of the next frame is started up to 12 processing periods in the period 920. That is, in the case of the timing example in FIG. 13, if the number of tracking destination feature point coordinates in the counting area section below the broken line 957 in FIG. 11 is up to 12, the template matching process is performed before starting the process of the next frame. Can be completed. However, in the case of the example in FIG. 11, since the number of tracking destination feature point coordinates is 17, as shown in a period 921 in FIG. 13, the template matching process corresponding to the 13th and subsequent five processing periods is performed. Cannot be completed before processing of the next frame starts.

  For this reason, in this embodiment, the count number comparison unit 702 of the tracking destination feature point determination unit 205 performs the processing of S907 as the number of tracking destination feature point coordinates counted in S906 (the tracking destination feature in the counting area section described above). It is determined whether the number of point coordinates is equal to or less than a predetermined set number. In the example of FIGS. 11 and 13, the number of tracking destination feature point coordinates that can be processed before the start of processing of the next frame is 12, and therefore, 12 is used as the predetermined set number. Then, in S907, when the count number comparison unit 702 determines (NO) that the number of coordinates of the tracking destination feature point is equal to or less than the set number (12 or less), the tracking destination feature point determination unit 205 displays the flowchart of FIG. End the process. On the other hand, in S907, when the count comparison unit 702 determines (YES) that the number of coordinates of the tracking destination feature point exceeds the set number (13 or more), the processing of the tracking destination feature point determination unit 205 transitions to S908. . In the case of the examples of FIGS. 11 and 13, the number of coordinates of the tracking destination feature point is 17, which exceeds the set number of 12, so that the processing of the tracking destination feature point determination unit 205 proceeds to S908.

  In step S908, the new feature point compensation unit 703 of the tracking destination feature point determination unit 205 performs discarding compensation processing for discarding the tracking destination feature point and supplementing the new feature point. In the example of FIGS. 11 and 13, the new feature point compensation unit 703 has a number (5) of tracking destinations exceeding a predetermined set number (12) from each tracking destination feature point in the counting area section. A feature point (tracking destination feature point 903) is selected and discarded. Then, the new feature point compensation unit 703 compensates the new feature point calculated by the new feature point calculation unit 202 as a tracking destination feature point in the designated area. In the case of the example of FIG. 11, the new feature point compensation unit 703 includes five new feature points corresponding to the number discarded in the region segment above the broken line 954 in FIG. The feature point 901 is compensated. The compensation area segment above the broken line 954 is an area that is initially processed in the raster scan order. Even if a new feature point is supplemented in this area segment, the template matching process cannot be completed within one frame period. This is an area where there is no possibility.

  A processing period required for the template matching process for each feature point coordinate of the five new feature points 901 compensated as tracking destination feature points by the discard compensation process can be represented by a period 930 in FIG. That is, the processing for 17 processing periods of the above-described period 920 is performed for the period 930 for 5 processing periods performed in the first half of one frame period and the remaining 12 processing periods performed in the second half of one frame period. The period 931 is distributed. Therefore, in the present embodiment, the template processing corresponding to the total number of 17 tracking destination feature point coordinates is completed before the processing of the next frame is started.

  When determining which tracking destination feature points are to be discarded among the tracking destination feature points in the counting area section below the broken line 957 in FIG. 11, for example, It is possible to use a method of selecting the objects to be discarded in order from the smallest feature value. Further, when determining feature points to be compensated, for example, new feature points in the compensation area section above the broken line 954 in FIG. 11 are selected in order from the largest feature value. Such a technique can be used.

  The above is the flow of the vector detection process in the first embodiment. In the present embodiment, processing for one frame has been described as an example, but the processing for one frame described above is performed in the same manner for each frame, and a vector value is calculated for each frame.

  In the present embodiment, the target area (counting area section) and the set number described above when counting the number of feature point coordinates described above can be determined according to the processing capability of the image processing apparatus. For example, based on the template matching image generation processing speed and the template matching processing speed in the image processing apparatus, the counting area division and the set number can be determined so that the image processing apparatus can complete the processing within a certain processing period. it can.

  In the present embodiment, the number of tracking destination feature point coordinates is counted in the counting area section below the broken line 957 in FIG. 11 to determine whether or not the processing is completed within one frame period. It is not limited to this example. For example, the tracking destination feature point determination unit 205 counts the number of tracking destination feature points in the three region segments from the broken line 955 to the broken line 956, the broken line 956 to the broken line 957, and the broken line 957 to the broken line 958 in FIG. You may compare with a setting number for every area division. Then, when the number of tracking destination feature points in each area section counted is larger than the set number, the tracking destination feature point determination unit 205 performs the tracking destination feature point discard compensation process as described above. Further, the tracking destination feature point determination unit 205 divides the region finely by increasing the number of region divisions, and the template matching process within one frame period is more evenly distributed within the frame image range (within the angle of view). You may make it let it. As a result, even when processing is performed with a larger number of template matches or a larger template size, the processing can be completed within one frame period.

  In the present embodiment, the case where the reading order of the image sensor 102 is the raster scan order is taken as an example. Therefore, the counting area section for counting the number of feature point coordinates is the lower area section of the image range (view angle) of the frame. However, it is not limited to this. For example, when the reading order of the image sensor 102 is scanned sequentially from the left side in the vertical direction, the counting area section for counting the number of feature point coordinates may be the right area section in the image range of the frame. Similarly, when the reading order of the image sensor 102 is scanned sequentially from the left side in the vertical direction, the supplemental area segment that compensates for the new feature point as the tracking destination feature point is the left region segment in the image range of the frame. Also good. In other words, in this embodiment, when the tracking destination feature point is discarded and compensated, an area segment in which an image used to detect a motion vector is generated first is set as the compensation region segment. On the other hand, in the present embodiment, when the tracking destination feature point is discarded and compensated, an area segment in which generation of an image used for motion vector detection is performed after the compensation region segment is set as the count region segment.

  In the vector detection that performs feature point tracking, the image processing apparatus according to the first embodiment counts the number of tracking destination feature points in a predetermined area and discards compensation processing when the tracking destination feature points are unevenly distributed within the image range of the frame. By performing the above, the processing can be completed within a predetermined period. Therefore, according to the image processing apparatus of the first embodiment, even when the tracking destination feature points are unevenly distributed within the image range of the frame, the vector detection is speeded up while preventing a decrease in the vector detection performance, and within a predetermined period. It is possible to complete the processing.

<Second Embodiment>
Hereinafter, the second embodiment will be described.
Since the configuration of the digital camera to which the image processing apparatus of the second embodiment is applied is the same as that of the digital camera of the first embodiment shown in FIG. 1, its illustration and description are omitted.
The image processing apparatus according to the second embodiment sets a tracking range based on the extracted feature points, and calculates a tracking destination feature point from the set range. That is, in the second embodiment, the tracking range is set from the initial coordinates of the feature point in the vector detection for tracking the feature point, and the tracking destination feature point is discarded and compensated when the tracking range is exceeded. The processing is completed within a predetermined period such as a frame period. The tracking range in the second embodiment represents a range in which feature points are tracked, and tracking destination feature points outside the tracking range are discarded.

  FIG. 14 is a block diagram illustrating a schematic configuration example of a vector detection unit included in the image processing unit 104 according to the second embodiment. In FIG. 14, the same components as those shown in FIG. 2 of the first embodiment are denoted by the same reference numerals, and description thereof is omitted. Further, the flowchart of the process in the vector detection unit of the second embodiment is substantially the same as the flowchart of FIG. 7 of the first embodiment described above, and the first embodiment also relates to the aforementioned FIG. 8 and FIG. Since these are the same, their illustration and description are omitted.

In the case of the second embodiment, the first RDDMAC 221 outputs the grid coordinate information 252 indicating the upper left coordinate position of the grid read from the DRAM 107 to the second RDDMAC 222 in S401 of FIG.
In the case of the vector detection unit of the second embodiment, in S409 of FIG. 7, the tracking destination feature point determination unit 205 stores the calculated tracking destination feature point coordinate information as tracking destination feature point coordinate information 257 as second information. And output to the SRAM 206 as well.

  Hereinafter, the configuration and processing contents of the tracking destination feature point determination unit 205 of the second embodiment illustrated in FIG. 14 will be described with reference to FIGS. 15 to 18. FIG. 15 is a block diagram illustrating a schematic configuration of the tracking destination feature point determination unit 205 according to the second embodiment. As illustrated in FIG. 15, the tracking destination feature point determination unit 205 according to the second embodiment includes a feature point management unit 801, a range setting unit 802, a range determination unit 803, and a new feature point compensation unit 804. FIGS. 16A and 16B are diagrams used for explaining the processing of the range determination unit 803 provided in the tracking destination feature point determination unit 205 of FIG. FIG. 17 is a flowchart illustrating processing of the tracking destination feature point determination unit 205 according to the second embodiment. FIG. 18 is a diagram used for explaining the processing of the range determination unit 803 and the new feature point compensation unit 804 provided in the tracking destination feature point determination unit 205 of FIG.

  First, as the processing of S1701 in FIG. 17, the feature point management unit 801, based on the current frame vector information and the previous frame tracking destination feature point coordinate information input from the SRAM 206, tracks the current frame tracking destination feature point coordinates. Update information. Specifically, the feature point management unit 801 updates the tracking destination feature point coordinate information of the current frame by adding the vector value of the current frame to the tracking destination feature point coordinates of the previous frame. Then, the feature point management unit 801 stores the updated tracking destination feature point coordinate information of the current frame in the internal memory. The internal memory of the feature point management unit 801 has a capacity capable of storing at least the tracking destination feature point coordinate information of the current frame and the initial coordinates of the feature points associated with the tracking destination feature point coordinate information. .

  As the next processing of S1702, the feature point management unit 801 determines whether or not the initial coordinates of the feature points associated with the tracking destination feature point coordinate information updated in S1701 are stored in the internal memory. If the feature point management unit 801 determines (YES) in S1702 that the initial coordinates of the associated feature point exist (stored in the internal memory) (YES), the process proceeds to S1704. On the other hand, if the feature point management unit 801 determines in S1702 that the initial coordinates of the associated feature point do not exist (NO), the feature point management unit 801 advances the process to S1703.

  In step S1703, the feature point management unit 801 determines that the tracking destination feature point coordinate information updated this time is the initial coordinate, and uses the tracking destination feature point coordinate of the current frame updated in step S1701 as the initial coordinate, and updated this time. After associating with the tracking destination feature point coordinate information, the process proceeds to S1704. Specifically, the feature point management unit 801 performs a process of adding a common ID to the initial coordinates and the tracking destination feature point coordinate information updated this time as the association process. Thereby, the feature point management unit 801 can determine whether or not the initial coordinates of the associated feature point exist by referring to the ID added to the tracking destination feature point coordinate information.

  In step S1704, the feature point management unit 801 outputs the tracking destination feature point coordinate information of the current frame stored in the internal memory to the range determination unit 803 and the new feature point compensation unit 804, thereby initializing the feature points. The coordinates are output to the range setting unit 802. After S1704, the process of the tracking destination feature point determination unit 205 transitions to S1705.

  In step S1705, the range setting unit 802 sets tracking ranges (SX, SY) to (EX, EY) based on the input initial coordinates of the feature points. Information on the set tracking range is sent to the range determination unit 803. After S1705, the process of the tracking destination feature point determination unit 205 transitions to S1706.

Here, a method for setting the tracking ranges (SX, SY) to (EX, EY) in the second embodiment will be specifically described with reference to FIGS.
FIG. 16A is a conceptual diagram showing a coordinate system and an image reading order of the current frame when the image of the current frame is divided into 5 in the horizontal direction and 5 in the vertical direction to set a total of 25 grids. is there. Each grid includes the feature extraction grid 302 and the peripheral grid 301 described above. In the case of FIG. 16A, the grid is divided for each coordinate (gx, gy) in the horizontal direction GX and the vertical direction GY. In the coordinate system of FIG. 16A, the upper left corner coordinates of the upper left corner feature extraction grid are (0, 0), and the upper right corner coordinates of the upper right corner feature extraction grid are (5gx, 0). Further, the coordinates of the lower left corner of the feature extraction grid at the lower left corner are (0, 5 gy), and the coordinates of the lower right corner of the feature extraction grid at the lower right corner are (5 gx, 5 gy). Note that the grid image reading order in the current frame is the reading order corresponding to the raster scan method, as indicated by an arrow 1601 in FIG.

  FIG. 16B is a diagram showing an example in which the range setting unit 802 reads the reading order setting recorded in the ROM 109 of FIG. 1 and sets the tracking range. In FIG. 16B, the vertical tracking range in each grid when the current frame is divided into a total of 25 grids divided into 5 parts in the horizontal direction and 5 parts in the vertical direction is set by the values of SY and EY. An example is shown. In the case of the second embodiment, for example, in order to prevent the tracking target feature points from concentrating on the lower region in the vertical direction and not ending the processing within a predetermined period, The range in which feature points are tracked is narrowed. In other words, the tracking range in the second embodiment is set as a range in which tracking is not performed in an area where processing is performed at the end in the raster scan order.

  When the grid 441 in FIG. 16B is taken as an example, the range setting unit 802 reads the tracking range value set in the grid 441 to which the initial coordinates of the input feature points belong, and the tracking range (SX, SY) to (EX, EY) are set. In the example of FIG. 16B, since SY: 0 and EY: 3gy are set as the tracking range values of the grid 441, the range setting unit 802 includes tracking ranges (SX, SY) to (EX, EY). Is set to (0, 0) to (5 gx, 3 gy). That is, the tracking range is narrower than the range (0,0) to (5gx, 5gy) of the feature extraction grid group in the coordinate system of FIG. 16A, and the tracking range (0,0) to (5gx, 3gy) is narrowed. ).

  In the second embodiment, the case where the reading order of the image sensor 102 is raster scan (horizontal direction) has been described, but the present invention is not particularly limited thereto. For example, when the reading order of the image sensor 102 is scanned in the vertical direction from the left side, the range setting unit 802 may narrow the horizontal tracking range according to the reading order. In this example, for example, even when the tracking target feature points are concentrated on the right side area in the horizontal direction, the processing range is prevented from being finished within a predetermined period by narrowing the horizontal tracking range. Can do.

Returning to FIG.
When the processing proceeds to S1706 in FIG. 17, the range determination unit 803 determines whether or not the input tracking destination feature point coordinates of the current frame exceed the tracking range (whether or not the tracking range is exceeded). Then, in S1706, the range determination unit 803 provides information on the determination result NG when it is determined that the tracking range is exceeded, and information on the determination result OK when it is determined that the tracking range is not exceeded. The data is output to the point compensation unit 804.

  With reference to FIG. 18, the determination process in the range determination unit 803 will be described using examples of tracking ranges (0, 0) to (5 gx, 3 gy). In FIG. 18, each grid in the hatched portion in the figure represents a grid in the tracking range, and the others represent grids outside the tracking range. In the example of FIG. 18, it is assumed that the input tracking destination feature point coordinates 1803 of the current frame are outside the tracking range. Therefore, in the example of FIG. 18, the range determination unit 803 sets the determination result NG for the tracking destination feature point coordinates 1803 of the current frame, and outputs information on the determination result NG to the new feature point compensation unit 804. After S1706, the process of the tracking destination feature point determination unit 205 transitions to S1707.

  In step S1707, the new feature point compensation unit 804 determines whether or not to perform compensation processing on the tracking destination feature point coordinate information of the current frame based on the determination result sent from the range determination unit 803. . The new feature point compensation unit 804 advances the process to S1709 when the determination result OK is sent from the range determination unit 803, and advances the process to S1708 when the determination result NG is sent.

  When the processing proceeds to S1708, the new feature point compensation unit 804 discards the tracking destination feature point from the tracking destination feature point coordinates of the current frame, compensates with the new feature point coordinate information, and proceeds to S1709. In the case of the example of FIG. 18 described above, the new feature point compensation unit 804 discards the tracking destination feature point coordinates 1803 of the current frame and calculates the new feature point coordinates 1804 calculated at the position of the grid 441 in FIG. Is compensated as the tracking destination feature point coordinates. After S1708, the process of the tracking destination feature point determination unit 205 transitions to S1709.

  In step S1709, the tracking destination feature point determination unit 205 determines whether all tracking destination feature point coordinates (tracking destination feature point coordinates for the number of feature extraction grids) in the current frame have been calculated. When the tracking destination feature point determination unit 205 determines that all the tracking destination feature point coordinates have been calculated (YES), the processing of the flowchart of FIG. 17 ends. On the other hand, if it is determined in S1709 that all the tracking destination feature point coordinates of the current frame have not been calculated (NO), the tracking destination feature point determination unit 205 returns the processing to S1701, and performs the above-described processing after S1701. Do. The processing from S1701 to S1709 is repeated until it is determined (YES) that all the tracking destination feature point coordinates have been calculated in S1709.

The above is the flow of processing in the tracking destination feature point determination unit 205 of the second embodiment. In the present embodiment, processing for one frame has been described as an example, but the processing for one frame described above is performed in the same manner for each frame, and a vector value is calculated for each frame.
Further, the tracking range setting in the second embodiment can be determined according to the processing capability of the image processing apparatus. For example, the tracking range can be set so that the image processing apparatus can complete the processing within a predetermined processing period based on the template matching image generation processing speed and the template matching processing speed in the image processing apparatus.

  The image processing apparatus according to the second embodiment sets a tracking range based on initial coordinates of feature points in vector detection for performing feature point tracking, and performs tracking point feature point discard compensation processing when the tracking range is exceeded. By performing the processing, the processing can be completed within a predetermined period such as one frame period. Therefore, according to the image processing apparatus of the second embodiment, even when the tracking destination feature points are unevenly distributed within the image range of the frame, the vector detection is speeded up while preventing the deterioration of the vector detection performance, and within a predetermined period. It is possible to complete the processing.

<Third Embodiment>
Hereinafter, a third embodiment will be described.
Since the configuration of the digital camera to which the image processing apparatus of the third embodiment is applied is the same as that of the digital camera of the first embodiment shown in FIG. 1, its illustration and description are omitted.
The image processing apparatus according to the third embodiment calculates a distance between tracking destination feature points based on the coordinate information of each tracking destination feature point, and performs a discard compensation process for the tracking destination feature point based on the calculated distance. Thus, the processing is completed within a predetermined period such as one frame period. In the case of the third embodiment, for example, when the distance between two tracking destination feature points is short, one is discarded and a new feature point is compensated as the tracking destination feature point.

  FIG. 19 is a block diagram illustrating a schematic configuration example of a vector detection unit included in the image processing unit 104 according to the third embodiment. In FIG. 19, the same components as those shown in FIG. 2 of the first embodiment are denoted by the same reference numerals, and description thereof is omitted. Although not shown in FIG. 19, the grid coordinate information is sent to the matching processing unit 203 via the second RDDMAC 222 as in FIG. 14. Further, the processing flowchart of the vector detection unit of the third embodiment is substantially the same as the flowchart of FIG. 7 described above, and the above-described FIG. 8 and FIG. .

  FIG. 20 is a diagram used for explaining the relationship between the grid layout, the feature points, and the template matching area in the vector detection process according to the third embodiment. In FIG. 20, the feature extraction grid 302, the peripheral grid 301, the search area 305, and the template area 304 are the same as those described with reference to FIG. In FIG. 20, the area composed of the grids indicated by the hatched area is an area in which the new feature point is compensated as the tracking destination feature point. In the third embodiment, this area is represented as a compensation area section 310. . The compensation area section 310 is set to an area where an image used for motion vector detection is generated first (an area where processing is initially performed in the raster scan order).

  FIG. 21 is a block diagram illustrating a schematic configuration example of the tracking destination feature point determination unit 205 according to the third embodiment. The tracking destination feature point determination unit 205 according to the third embodiment includes a distance calculation unit 715, a process determination unit 716, and a new feature point compensation unit 717. In FIG. 21, the same components as those in FIG. 10 described above are denoted by the same reference numerals, and description thereof is omitted. FIG. 22 is a diagram used for explaining the positional relationship of the tracking destination feature points processed by the tracking destination feature point determination unit 205 according to the third embodiment. In the example of FIG. 22, only one grid and a part of the surrounding grid are shown for easy viewing of the figure. Furthermore, FIG. 23 is a process flowchart of the tracking destination feature point determination unit 205 according to the third embodiment. Since the processing timing in the third embodiment is the same as that in the example of FIG. 13 described above, illustration and description thereof are omitted.

Hereinafter, with reference to FIG. 20 to FIG. 23, an outline of the configuration and processing in the tracking destination feature point determination unit 205 of the third embodiment will be described. Note that the processing from S901 to S905 in FIG. 23 is the same as the processing from S901 to S905 in FIG. 12 described above, and therefore a description thereof will be omitted, and S2301 to S2304, which are processing different from FIG.
In the flowchart of FIG. 23, after S905, the process of the tracking destination feature point determination unit 205 transitions to S2301.

In step S2301, the distance calculation unit 715 of the tracking destination feature point determination unit 205 calculates the distance between the two tracking destination feature points. Specifically, as shown in FIG. 22, when the coordinates of the two tracking destination feature points 2210 and 2211 are (X1, Y1) and (X2, Y2), the distance calculation unit 715 The distance D between the tracking destination feature points is calculated by the following equation (4). After S2301, the process of the tracking destination feature point determination unit 205 transitions to S2302.

  In S2302, the process determination unit 716 of the tracking destination feature point determination unit 205 discards the two tracking destination feature points based on the distance D between the two tracking destination feature points obtained by the distance calculation unit 715. It is determined whether or not compensation processing is performed. Specifically, the process determination unit 716 determines whether or not the distance D between the two tracking destination feature points obtained in S2301 is smaller than a predetermined threshold value Dth. The threshold value Dth is an index when determining whether or not to perform the discard compensation process for the two tracking destination feature points. In the case of the third embodiment, the threshold value Dth is, for example, a distance 2202 that is half the length of one side of the template region 304 in FIG. In the present embodiment, the threshold value Dth is set to the distance 2202 if the distance D between the two tracking destination feature points is shorter (if closer) than half the length of one side of the template area. This is because there are two feature points inside. In step S2302, the process determining unit 716 advances the process to step S2304 when the distance D between the two tracking destination feature points is equal to or greater than the threshold (D ≧ Dth) (NO). If <Dth) (YES), the process proceeds to S2303.

  Note that the threshold value Dth may be adjusted so that the template matching process can be completed within one frame period. For example, the degree of congestion of a plurality of tracking destination feature points is evaluated so as to distribute the template matching process, and the threshold Dth value is increased as the density increases, so that the discarding compensation process for the tracking destination feature points is easily performed. May be. Also, the coordinates of the tracking target feature points are evaluated so that the template matching process does not concentrate on the lower region within the frame image range (within the angle of view), and the threshold Dth is increased as it approaches the lower end of the frame image range. Thus, it may be easy to perform the discard compensation process for the tracking destination feature point.

  In step S2303, the new feature point compensation unit 717 of the tracking destination feature point determination unit 205 performs discarding processing for discarding the tracking destination feature point and compensating for the new feature point as the tracking destination feature point in the third embodiment. carry out. First, the new feature point compensation unit 717 compares the feature values of the two tracking destination feature points. Then, the new feature point compensation unit 717 sets a tracking destination feature point having a smaller feature value as a discard target. Note that the discard target may be selected based on the coordinates of the two tracking destination feature points instead of the feature values. For example, based on the coordinates of two tracking destination feature points, it is determined which tracking destination feature point coordinates are on the lower area side of the image range of the frame, and the one that is on the lower area side is subject to destruction Select as Then, the new feature point compensation unit 717 discards one of the tracking destination feature points, and then compensates the feature extraction grid of the compensation region section 310 illustrated in FIG. 20 as the tracking destination feature point. By this discard compensation process, the period 930 required for the template matching process for the compensated tracking destination feature points is distributed in the first half of one frame period, as described with reference to FIG. After S2303, the process of the tracking destination feature point determination unit 205 transitions to S2304.

  In S2304, the tracking destination feature point determination unit 205 determines whether or not the processing from S2301 to S2303 has been performed between all the tracking destination feature points. If it is determined in S2304 that the processing between all the tracking destination feature points has not been performed (NO), the tracking destination feature point determination unit 205 returns the processing to S2301 and discards the processing from S2301 to S2303 described above. Perform compensation processing. On the other hand, if it is determined in S2304 that the processing for all the tracking destination feature points has been performed (YES), the tracking destination feature point determination unit 205 ends the processing of FIG.

  The above is the flow of processing in the tracking destination feature point determination unit 205 of the third embodiment. In the present embodiment, processing for one frame has been described as an example, but the processing for one frame described above is performed in the same manner for each frame, and a vector value is calculated for each frame.

  The threshold value Dth and the compensation region classification 310 that serve as an index for discard compensation processing in the third embodiment can be determined according to the processing capability of the image processing apparatus. For example, based on the template matching image generation processing speed and the template matching processing speed in the image processing apparatus, the threshold value Dth and the compensation region section 310 are set so that the image processing apparatus can complete the processing within a predetermined processing period. It can be carried out.

Further, the tracking destination feature point determination unit 205 preferentially discards, for example, tracking destination feature points with small feature values when selecting two tracking destination feature points for calculating the distance from among the plurality of tracking destination feature points. As described above, it may be selected from two tracking destination feature points in the order of the tracking destination feature points with small feature values.
Further, the tracking destination feature point determination unit 205 may not obtain the distance between all the tracking destination feature points of the plurality of tracking destination feature points. For example, the tracking destination feature point determination unit 205 may set the distance calculation target only to the tracking destination feature point group in the lower region, which easily affects the completion time of the template matching process. Further, for example, when the tracking destination feature points are dense in only a part of the region and the tracking destination feature points are widely dispersed in the other image range, the tracking destination feature point determination unit 205 performs S2301 to S2304 in FIG. This processing may be skipped and the discard compensation processing may not be performed. As described above, when the discard compensation process is skipped, the processing load on the image processing unit 104 is reduced, and the processing speed can be increased.
In addition, the tracking-destination feature point determination unit 205 may preferentially compensate for new feature points in the compensation region segment from those having large feature values. In addition, the tracking destination feature point determination unit 205 preferentially assigns new feature points from the grid on the upper region side so that the distribution of the tracking destination feature points is more widely dispersed when the new feature points are compensated for in the compensation region classification. You may make it supplement.

  The image processing apparatus according to the third embodiment completes the processing within a predetermined period such as one frame period by performing a discarding compensation process for the tracking destination feature points based on the distance between the plurality of tracking destination feature points. Can do. Therefore, according to the image processing apparatus of the third embodiment, even when the tracking destination feature points are unevenly distributed within the image range of the frame, the vector detection is speeded up while preventing a decrease in the vector detection performance, and within a predetermined period It is possible to complete the processing.

<Fourth Embodiment>
Hereinafter, a fourth embodiment will be described.
Since the configuration of the digital camera to which the image processing apparatus of the fourth embodiment is applied is the same as that of the digital camera of the first embodiment shown in FIG. 1, its illustration and description are omitted.
The image processing apparatus according to the fourth embodiment divides the plurality of feature extraction grids described above into a non-tracking area section and a tracking area section, and detects a motion vector based on a tracking destination feature point in the tracking area section, In the non-tracking area section, a motion vector is detected based on a new feature point.

  Since the configuration of the vector detection unit included in the image processing unit 104 of the fourth embodiment is the same as that shown in FIG. 19 of the third embodiment described above, its illustration and description are omitted. Further, the processing flowchart in the vector detection unit of the fourth embodiment is substantially the same as the flowchart of FIG. 7 described above, and also the above-described FIG. 8 and FIG. To do.

  FIG. 24 is a block diagram illustrating a schematic configuration example of the tracking destination feature point determination unit 205 according to the fourth embodiment. The tracking destination feature point determination unit 205 according to the fourth embodiment includes an area determination unit 720 and a new feature point compensation unit 721. 24, the same components as those in FIG. 10 described above are denoted by the same reference numerals, and description thereof is omitted. FIG. 25 is a diagram used for explaining the positional relationship between the tracking destination feature point and the new feature point processed by the tracking destination feature point determination unit 205 of the fourth embodiment, and the tracking region segment and the non-tracking region segment. Note that the feature point 2501, the tracking destination feature point 2502, the vector value 2503, the feature extraction grid 2504, and the peripheral grid 2505 in FIG. 25 are the same as described above.

  In the fourth embodiment, the plurality of feature extraction grid groups in the lower region of FIG. 25 are set as the non-tracking area section 2506, and the other feature extraction grid groups are set as the tracking area section. The tracking destination feature point determination unit 205 according to the fourth embodiment discards the tracking destination feature point when the tracking destination feature point enters the non-tracking area section 2506. In the case of the present embodiment, the matching processing unit 203 uses the new feature points calculated by the new feature point calculation unit 202 for each grid of the non-tracking region section 2506 (five feature extraction grids in the example of FIG. 25). Template matching centering on coordinates 2507 is performed.

  FIG. 26 is a process flowchart of the tracking destination feature point determination unit 205 according to the fourth embodiment. Further, FIG. 27 is a diagram showing a processing timing chart in the fourth embodiment. In FIG. 26, processes similar to those in FIG. 12 described above are denoted by the same reference numerals as those in FIG. 12, and description thereof is omitted. Also, in FIG. 27, the same parts as those in FIG. 13 are given the same reference numerals as those in FIG.

  At the processing timing shown in FIG. 27, the time required to perform the template matching process for one template region is uniquely determined by the size of the template region as described above. For example, when 17 tracking destination feature points are concentrated in the lower region (non-tracking region classification of the fourth embodiment) in the image range of the frame as in the example of FIG. The matching process starts at the timing of time 1959 in FIG. Further, as described above, all of the 17 template matching processes are not completed within the time 910 from the time 1959 until the end of the frame period (the time until the start of the next frame period). In the case of the example of FIG. 27, as shown in the period 921, the template matching process for at least five tracking destination feature points will not end.

  For this reason, the tracking destination feature point determination unit 205 of the fourth embodiment sets the non-tracking area segment 2506 for five grids shown in FIG. 25 and discards the tracking destination feature point of this non-tracking area segment 2506. . Therefore, when the tracking destination feature point of the non-tracking area segment 2506 is discarded, the template matching process is a template matching process using only the tracking destination feature point in the tracking area segment excluding the non-tracking area segment 2506. The template matching process using only the tracking destination feature points in the tracking area section excluding the non-tracking area section 2506 is completed at the timing of the period 2000 at the maximum (even if it takes the longest). In other words, if there are 17 tracking destination feature points in the non-tracking area segment, these tracking destination feature points are discarded and no processing is required, so it takes some time to process the tracking destination feature points in the tracking area segment. Even if it takes time, the process ends by the period 2000.

  On the other hand, in the case of the fourth embodiment, the tracking destination feature point determination unit 205 extracts five new feature points (5 pieces) extracted by the new feature point calculation unit 202 in each of the five grids of the non-tracking region section 2506. Based on this, a motion vector is detected and a template matching process is performed. In the case of FIG. 27, a period 930 represents a period required for template matching processing based on five new feature points. A time 2010 from the period 2000 to the end of the frame period (time until the start of the next frame period) is for performing template matching processing (processing in the period 930) on the five new feature points in the non-tracking area section 2506. Is enough time. Therefore, in the case of the fourth embodiment, the process up to the detection of the motion vector value by the template matching process can be completed within one frame period.

  The outline of the configuration and processing in the tracking destination feature point determination unit 205 of the fourth embodiment will be described below with reference to FIGS. The processing from S901 to S905 in FIG. 26 is the same as the processing in FIG. 12 described above, and here, S2601 to S2603, which are different from FIG. 12, will be described.

  The example of FIG. 11 is an example in which there are 17 tracking destination feature points in the non-tracking area section 2506, but in order to make the explanation easier to understand, a total of five grids in the non-tracking area section 2506 are used. An example will be given in which five tracking destination feature points are entered and these five tracking destination feature points are discarded. For this reason, it is assumed that the number of tracking destination feature points to be tracked in the tracking area section is 20. Therefore, the tracking destination feature point coordinates calculated in S901 are 20. On the other hand, five new feature points are extracted by the new feature point calculation unit 202 from the five grids of the non-tracking region section 2506. Therefore, the vector value is obtained by performing template matching processing using the five new feature points extracted in the non-tracking region segment 2506 and the 20 tracking destination feature points in the tracking region segment, thereby obtaining 25 vector values. Will be calculated.

When it is determined YES in S903 of FIG. 26, or after the process of S904, the process of the tracking destination feature point determination unit 205 transitions to S2601.
In step S2601, the area determination unit 720 of the tracking destination feature point determination unit 205 determines whether the tracking destination feature point coordinates calculated in step S901 have entered the non-tracking area section 2506 in FIG. Specifically, the area determination unit 720 determines that the vertical coordinate value of the tracking destination feature point is within the non-tracking area section 2506 when the vertical coordinate value is equal to or greater than the upper end coordinate value of the non-tracking area section 2506 (YES) ) And the process proceeds to S2602. On the other hand, if the vertical coordinate of the tracking destination feature point is less than the value of the upper end coordinate of the non-tracking area segment 2506, the area determination unit 720 determines that it is not within the non-tracking area segment 2506 (NO), and processing in S2603 To proceed.

  When the process proceeds to S2602, the new feature point compensation unit 721 discards the tracking destination feature point coordinates determined to be in the non-tracking area section 2506 in S2601, and the new feature point calculated by the new feature point calculation unit 202 Replace with point coordinates. The tracking destination feature point determination unit 205 uses the deleted tracking destination feature point coordinates at the corresponding position in the feature point extraction grid at the position where the new feature point is calculated first in the current frame. It may be replaced by point coordinates. After S2602, the process of the tracking destination feature point determination unit 205 transitions to S2603.

  In step S2603, the tracking destination feature point determination unit 205 determines whether all the tracking destination feature point coordinates have been calculated. In S2603, if the tracking destination feature point determination unit 205 determines that all the tracking destination feature point coordinates have not been calculated (NO), the process returns to S901, and the processing after S901 is performed. . In the case of the example of FIG. 25, since the calculation processing of 20 tracking destination feature point coordinates is performed, the processing from S901 to Step 2603 is repeated 20 times. On the other hand, if the tracking destination feature point determination unit 205 determines in S2603 that all the tracking destination feature point coordinates have been calculated (YES), the processing of the flowchart in FIG.

  The above is the flow of processing in the tracking destination feature point determination unit 205 of the fourth embodiment. In the present embodiment, processing for one frame has been described as an example, but the processing for one frame described above is performed in the same manner for each frame, and a vector value is calculated for each frame.

  The non-tracking area section and the tracking area section in the fourth embodiment can be determined according to the processing capability of the image processing apparatus. For example, based on the template matching image generation processing speed and the template matching processing speed in the image processing apparatus, the non-tracking area section and the tracking area section are set so that the image processing apparatus can complete the processing within a certain processing period. It can be performed.

In the fourth embodiment, the template matching processing period is distributed by providing a non-tracking area section, and the non-tracking area section is a lower area. The non-tracking area section can be set to an arbitrary area, and can be set to at least one of an upper area, a right area, a left area, a center area, etc. of the image range of the frame, for example. . Furthermore, the non-tracking area sections may be distributed and arranged in the image range of the frame.
Further, in the fourth embodiment, new feature points to be compensated may be preferentially compensated from the grid with the largest feature value in the region in which the new feature points are compensated, or may be easily dispersed. You may preferentially compensate from what was calculated by the grid of the upper area.

  The image processing apparatus according to the fourth embodiment includes a tracking area section and a non-tracking area section. In the tracking area section, a motion vector is detected based on the tracking destination feature point, and in the non-tracking area section, a new feature point is used. A motion vector is detected. As a result, the image processing apparatus according to the fourth embodiment speeds up the vector detection while preventing deterioration in vector detection performance even when the tracking destination feature points are unevenly distributed within the image range of the frame, and within a predetermined period. The process can be completed.

<Fifth Embodiment>
The fifth embodiment will be described below.
Since the configuration of the digital camera to which the image processing apparatus of the fifth embodiment is applied is the same as that of the digital camera of the first embodiment shown in FIG. 1, its illustration and description are omitted.
The image processing apparatus according to the fifth embodiment divides a plurality of feature extraction grid groups into at least two area sections, a first area section and a second area section. Then, the image processing apparatus according to the fifth embodiment determines to which region section the tracking destination feature point belongs based on the vector value of the tracking destination feature point and other vector values, and the determination result Accordingly, the template size (size) used for the template matching process is determined.

  Since the configuration of the vector detection unit of the image processing unit 104 of the fifth embodiment is the same as that shown in FIG. 14 of the second embodiment described above, for example, its illustration and description are omitted. In addition, the flowchart of the processing in the vector detection unit of the fifth embodiment is almost the same as the flowchart of FIG. 7 described above, and also the above-described FIG. 8 and FIG. Omitted.

  FIG. 28 is a diagram illustrating a schematic configuration of the matching processing unit 203 according to the fifth embodiment. As shown in FIG. 28, the matching processing unit 203 according to the fifth embodiment includes an area dividing unit 2801, a counting unit 2802, a size changing unit 2803, an estimating unit 2804, and a process executing unit 2805. FIG. 29 is a flowchart showing the flow of template matching processing in the matching processing unit 203 of the fifth embodiment. FIG. 30A and FIG. 30B are diagrams used for explaining the processing of the area sorting unit 2801 in the matching processing unit 203. FIG. 31A and FIG. 31B are diagrams used for explaining the processing of the size changing unit 2803 and the estimation unit 2804 in the matching processing unit 203.

Hereinafter, the configuration and processing outline of the matching processing unit 203 according to the fifth embodiment will be described with reference to FIGS. 28 to 31.
In the fifth embodiment, the grid coordinate information 252 is read from the second RDDMAC 222 together with the search area image data 253 and the template area image data 254 described above, and is also sent to the matching processing unit 203.

  The area classification unit 2801 of the matching processing unit 203 divides the plurality of feature extraction grids of the current frame into at least two area classifications of the first area classification and the second area classification as processing of S2901. Specifically, the region segmentation unit 2801 calculates the region segment to which each feature extraction grid belongs based on the grid coordinate information of the current frame. After S2901, the processing of the matching processing unit 203 transitions to S2902.

  Here, the region segment calculation setting method in the region segmentation unit 2801 will be specifically described with reference to FIGS. FIG. 30 (a) is a conceptual diagram showing the image reading order when the current frame is divided into five grids in the horizontal direction and five in the vertical direction. Further, as shown in FIG. 30A, image reading of the current frame is performed by a raster scan method as indicated by an arrow in the figure. FIG. 30B is a conceptual diagram showing the area divisions assigned to each of the 25 grids in the raster scan method. The area division unit 2801 sets the feature extraction grid group (second half of the raster scan) for two rows above the lower end of FIG. 30B as the first area division. That is, the first area segment is an area segment where an image used for detection of the motion vector is generated later. In other words, the first area segment is an area segment where reading is performed at the end of the image reading order. In addition, the area segmenting unit 2801 sets the feature extraction grid group (the first half of the raster scan) for three rows below the upper end of FIG. 30B as the second area segment. That is, the second region segment is a region segment where the image used for the detection of the motion vector is first generated later, in other words, a region segment where reading is performed at the beginning of the image reading order. In the example of FIG. 30B, the feature extraction grid group for two rows above the lower end is set as the first area section. However, the present invention is not limited to this, and an arbitrary area is defined as the first area section, It is also possible to set the second area section.

  In step S2902, the matching processing unit 203 determines whether or not the feature extraction grid to which the template region image data that is the template matching processing target belongs is the feature extraction grid in the first region section. judge. If the matching processing unit 203 determines that it is not the first region section (NO), it sends the current template region image data to the processing execution unit 2805 via the size changing unit 2803. Thereafter, the processing of the matching processing unit 203 transitions to S2909. On the other hand, if the matching processing unit 203 determines in S2902 that the region is the first area division (YES), the process proceeds to S2903. Note that the determination of whether or not the feature extraction grid is the first region section may be performed by the CPU 112 in FIG.

  The counting unit 2802 of the matching processing unit 203 is a counter that counts tracking destination feature points for each grid. When the processing proceeds to S2903, the counting unit 2802 counts the tracking destination feature points in the corresponding grid from the grid coordinate information of the current frame, and updates the counter value. After S2903, the processing of the matching processing unit 203 transitions to S2904.

  In S2904, the matching processing unit 203 uses a predetermined threshold value stored in the ROM 109 of FIG. 1, and determines whether the number of tracking destination feature points counted in the grid by the counting unit 2802 exceeds the predetermined threshold value. Judging. If the matching processing unit 203 determines that the number of tracking destination feature points in the grid does not exceed the threshold (NO), the matching processing unit 203 sends the current template region image data to the processing execution unit 2805 via the size changing unit 2803. Then, the process proceeds to S2909. On the other hand, if the matching processing unit 203 determines that the number of tracking destination feature points in the grid exceeds the threshold (YES), the matching processing unit 203 advances the process to S2905. Note that the CPU 112 in FIG. 1 may determine whether or not the number of tracking destination feature points in the grid exceeds a threshold value. In addition, the threshold value is set for each grid in advance, and can be updated as appropriate according to shooting conditions and the like. Note that the threshold value is updated according to the shooting conditions and the like by the CPU 112 in FIG.

  In step S2905, the matching processing unit 203 determines whether or not the template size of the input template area image data is a predetermined minimum size. When it is determined that the template size is the minimum size (YES), the matching processing unit 203 determines that the process cannot be completed within a predetermined period, and ends the process of the flowchart of FIG. On the other hand, if it is determined that the template size is not the minimum size (NO), the matching processing unit 203 advances the process to S2906. Note that the CPU 112 may determine whether or not the template size is a predetermined minimum size.

  In step S2906, the size changing unit 2803 reduces the size of the input template area image data (that is, the size of the template), and outputs the reduced template area image data to the estimation unit 2804. After S2906, the processing of the matching processing unit 203 transitions to S2907.

  A template size changing process in the size changing unit 2803 will be described with reference to FIG. FIG. 31A shows an example of the template size. In this embodiment, the template region used for template matching is, for example, a square. In the example of FIG. 31A, three types with one side of n, 2n, and 3n (n is the number of pixels) are used as size examples. Cite. If the input template area image data is a template area having one side of 3n, the size changing unit 2803 trims the upper, lower, left, and right areas with respect to the center of the template area in step S2907 to obtain a template area image having one side of 2n. to shrink. Note that the size changing unit 2803 may reduce a template area having one side of 3n to a template area image having one side of n.

  In step S2907, the estimation unit 2804 calculates a processing time coefficient for estimating the processing time based on the size of the input template area image. After S2907, the processing of the matching processing unit 203 transitions to S2908.

  The time calculation process in the estimation unit 2804 will be described with reference to FIG. FIG. 31B is a diagram used for explaining an example of processing time coefficient calculation in the estimation unit 2804. In the present embodiment, the template area is a square as shown in FIG. 31A, and n, 2n, and 3n in FIG. 31B represent the size of one side of the template area. Further, the SAD calculation amount in FIG. 31B indicates the calculation amount when the correlation value is calculated by template matching, and the SAD calculation is performed by the above-described equation (3). In the correlation value calculation, calculation is performed on all the pixel values in the template region. Therefore, as one side of the template size increases as n, 2n, and 3n, the SAD calculation amount becomes n, 4n in FIG. , 9n. In addition, the processing time coefficient in FIG. 31B represents the processing time coefficient of the template matching process when the size of one side of the template region is n as 1. Therefore, as one side of the template region becomes larger as n, 2n, 3n, the processing time coefficient becomes larger (longer) as 1, 4, 9, and so on. On the other hand, if the template area is changed from the size of one side 3n to the size of one side n by the size changing unit 2803, the processing time coefficient of the template matching process is reduced from 1 to 9, and the processing time is reduced to 1/9. Will be.

  When the processing proceeds to S2908, the matching processing unit 203 performs template matching processing centered on the tracking destination feature point within a predetermined time (within one frame period) based on the processing time coefficient calculated by the estimation unit 2804. ) To determine whether or not to end. If the matching processing unit 203 determines that the processing is completed within a predetermined time (YES), the template region image data reduced by the size changing unit 2803 is sent to the processing execution unit 2805, and the process is performed in S2909. Proceed. On the other hand, if the matching processing unit 203 determines that the process does not end within the predetermined time (NO), the matching processing unit 203 returns the process to S2905 to continue the process of further reducing the template size. That is, the template size change processing performed in steps S2905 to S2908 is repeated until the processing time required for template matching falls within a predetermined time such as one frame period. Note that the CPU 112 may determine whether or not the template matching process is completed within a predetermined time.

  In step S2909, the process execution unit 2805 calculates a correlation value using the input template area image data and search area image data, and detects a motion vector based on the correlation value. Then, the matching processing unit 203 outputs the detected vector value information and correlation value information of the motion vector to the accuracy determination unit 204 described above. Thereafter, the matching processing unit 203 ends the process of the flowchart of FIG.

  The above is the flow of processing in the matching processing unit 203 of the fifth embodiment. In the present embodiment, processing for one frame has been described as an example, but the processing for one frame described above is performed in the same manner for each frame, and a vector value is calculated for each frame.

  As described above, the image processing apparatus according to the fifth embodiment reduces the SAD calculation amount by reducing the above-described correlation value calculation region by reducing the size of the template region by the size changing unit 2803. As a result, the image processing apparatus according to the fifth embodiment can complete the template matching process within one frame period. Therefore, the image processing apparatus according to the fifth embodiment speeds up vector detection while preventing deterioration in vector detection performance even when feature points of tracking destinations are unevenly distributed within the image range of the frame, and within a predetermined period. The process can be completed.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  104: Image processing unit, 107: DRAM, 112: CPU, 221: First RDDMAC, 201: Image generation unit for matching, 202: New feature point calculation unit, 203: Matching processing unit, 204: Accuracy determination unit, 205 : Tracking destination feature point determination unit, 206: SRAM, 222: second RDDMAC, 231: WRDMAC

Claims (30)

A dividing means for dividing the frame image into a plurality of regions;
Extraction means for extracting feature points of an image in each of the divided areas;
Detecting means for detecting a motion vector based on the feature points;
Calculation means for calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame;
Compensation for discarding at least a part of the tracking destination feature point and supplementing the new feature point extracted by the extraction unit as the tracking destination feature point when the tracking destination feature point satisfies a predetermined condition Means,
An image processing apparatus comprising: A dividing means for dividing the plurality of divided areas into a counting area section and a compensation area section;
Counting means for counting the number of the tracking destination feature points in the counting area section,
When the number of the tracking destination feature points in the counting area section exceeds a predetermined number, the compensation means discards the tracking destination feature points in the counting area section, and the extracting means The image processing apparatus according to claim 1, wherein the extracted new feature point is compensated in the compensation region section as a tracking destination feature point.   The compensation means discards the number of tracking destination feature points exceeding the set number in the counting area section, and sets the new feature points according to the number of the discarded tracking destination feature points to the compensation area section. The image processing apparatus according to claim 2, wherein the image processing apparatus compensates for the above.   The image processing apparatus according to claim 2, wherein the segmentation unit sets, as the compensation region segment, a region segment in which an image used for detecting the motion vector is generated first.   5. The image according to claim 4, wherein the segmentation unit sets, as the count region segment, a region segment in which generation of an image used for detection of the motion vector is performed after the compensation region segment. Processing equipment.   6. The image processing apparatus according to claim 4, wherein generation of an image used for detecting the motion vector is performed in a raster scan order.   7. The compensation means selects the tracking destination feature points to be discarded as a plurality of tracking destination feature points in the counting area section in ascending order of feature values. The image processing apparatus according to any one of the above.   The said compensation means selects the new feature point used as the object of the said compensation from the several new feature point in the said compensation area | region division in order with a high feature value, The any one of Claim 2 to 7 characterized by the above-mentioned. The image processing apparatus according to item 1. A range setting means for setting a tracking range based on the extracted feature points;
The calculation means calculates the tracking destination feature point from the tracking range,
When the calculated tracking destination feature point is out of the tracking range, the compensation unit discards the tracking destination feature point outside the tracking range and is extracted by the extraction unit within the tracking range. The image processing apparatus according to claim 1, wherein a new feature point is compensated in the tracking range as a tracking destination feature point.   The image processing apparatus according to claim 9, wherein the range setting unit sets the tracking range according to a setting of a reading order of the image of the frame.   The image processing apparatus according to claim 10, wherein the range setting unit sets the tracking range to be narrower than the image range of the frame according to the setting of the reading order of the image of the frame.   The range setting means, when the setting of the reading order of the image of the frame is the setting of the reading order of the image in the horizontal direction, in accordance with the setting of the reading order in the horizontal direction, The image processing apparatus according to claim 11, wherein the tracking range is set to be narrowed.   The range setting means, when the setting of the reading order of the image of the frame is the setting of the reading order of the image in the vertical direction, in accordance with the setting of the reading order in the vertical direction, The image processing apparatus according to claim 11, wherein the tracking range is set to be narrowed.   The image processing apparatus according to claim 9, wherein the setting of the reading order of the image is a setting of a reading order from an imaging unit that picks up an image of the frame. It further has a distance calculation means for calculating the distance between the plurality of tracking destination feature points,
The calculating means calculates the tracking destination feature point based on the distance calculated between the tracking destination feature points;
When the distance calculated between the tracking destination feature points is less than a predetermined threshold, the compensation means discards one of the two tracking destination feature points whose distances are calculated, and the extraction means The image processing apparatus according to claim 1, wherein the new feature point extracted in step (b) is supplemented in a predetermined area section as a tracking destination feature point.   The discarded tracking destination feature point is determined based on at least one of a feature value and a coordinate position of two tracking destination feature points whose calculated distance is less than the threshold. The image processing apparatus according to claim 15, characterized in that:   The image processing apparatus according to claim 15 or 16, wherein the threshold value is a value determined based on a size of a rectangular region used for detecting the motion vector.   The image processing apparatus according to claim 15, wherein the threshold value is a value determined based on coordinate positions of a plurality of tracking destination feature points. A dividing means for dividing the plurality of divided areas into a tracking area section and a non-tracking area section;
The detection means detects a motion vector based on the tracking destination feature point in the tracking area section, and calculates a motion vector based on the new feature point extracted by the extraction means in the non-tracking area section,
The compensation means discards the tracking destination feature point when the tracking destination feature point is in the non-tracking area section, and compensates the new feature point extracted by the extraction means. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 19, wherein the compensation unit compensates the new feature point extracted by the extraction unit from the tracking area section.   21. The non-tracking area segment is at least one of a lower area, a left area, an upper area, and a right area of an image range of a frame. Image processing device.   The image processing apparatus according to any one of claims 19 to 21, wherein the non-tracking area sections are distributed and arranged in an image range of a frame. A dividing means for dividing the image into a plurality of regions;
Extraction means for extracting feature points of an image in each of the divided areas;
Detecting means for detecting a motion vector by template matching based on the feature points;
Calculation means for calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame;
Dividing means for dividing the plurality of divided areas into a first area section and a second area section;
The detecting means determines a size of a template used for template matching for detection of the motion vector according to which of the first area section and the second area section the tracking destination feature point belongs to. An image processing apparatus characterized by determining. The sorting means includes
An area segment that is read at the end of the image readout order is set as the first area segment,
24. The image processing apparatus according to claim 23, wherein an area section that is read out in an initial order of image reading is set as the second area section.   25. The image processing apparatus according to claim 23, wherein the detection unit changes a size of a template used for the template matching when the tracking destination feature point belongs to the first region section. . Counting means for counting the number of tracking destination feature points for each of the divided areas;
The detection means includes a template when a tracking destination feature point of the divided area belongs to the first area section and the number of tracking destination feature points counted for the area exceeds a predetermined threshold. The image processing apparatus according to claim 24, wherein the size of the template using matching is changed to be small. Having time calculation means for calculating the processing time of vector detection;
27. The image according to any one of claims 23 to 26, wherein the detection unit repeats changing the size of the template until the calculated processing time is within a predetermined time. Processing equipment. A division step of dividing the frame image into a plurality of regions;
An extraction step of extracting feature points of an image in each of the divided areas;
A detection step of detecting a motion vector based on the feature points;
A calculation step of calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame;
Compensation for discarding at least a part of the tracking destination feature point and supplementing the new feature point extracted in the extraction step as the tracking destination feature point when the tracking destination feature point satisfies a predetermined condition Process,
An image processing method for an image processing apparatus, comprising: A division step of dividing the image into a plurality of regions;
An extraction step of extracting feature points of an image in each of the divided areas;
A detection step of detecting a motion vector by template matching based on the feature points;
A calculation step of calculating a tracking destination feature point used for motion vector detection of the next frame based on the value of the motion vector and the feature point detected from the previous frame;
A dividing step of dividing the plurality of divided areas into a first area section and a second area section;
In the detection step, a size of a template used for template matching for the detection of the motion vector is determined according to which of the first region segment and the second region segment the tracking destination feature point belongs to. An image processing method of an image processing apparatus characterized by determining.   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 27.

Images