JP2019192048A - Imaging apparatus - Google Patents

Abstract

To provide an imaging apparatus capable of completing processing in a predetermined period such as one frame period, even when a template area to be processed is biased in a field angle in vector detection for performing feature point tracking.SOLUTION: The imaging apparatus reduces the search range of template matching, to increase the speed of vector detection processing and complete processing in the predetermined period such as one frame period, when a vector detection area is biased in the vector detection for performing the feature point tracking.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a hardware configuration for tracking feature points between a plurality of frame images and detecting a motion vector.

  In order to perform camera shake correction on video shot using an imaging device such as a digital still camera or a digital video camera, the amount of motion between frame images is detected and alignment is performed on a plurality of images. There is a need. As a method for detecting the amount of motion between frame images, there are a method of using information of an external device such as a gyro sensor, a method of estimating the amount of motion from a captured frame image, and the like.

  Various methods for estimating the amount of motion using a frame image have been proposed in the past. A typical example is motion vector detection by template matching. In template matching, first, one of two frame images in a video is used as an original image and the other is used as a reference image. Then, a rectangular area having a predetermined size arranged on the original image is used as a template block, and a correlation with a distribution of pixel values in the template block is obtained at each position of the reference image.

  At this time, the position where the correlation is highest in the reference image is the destination of the template block, and the direction and amount of movement to the destination when the position of the template block on the original image is used as a reference is the motion vector. . In order to improve the detection rate of motion vectors, there is a technique for extracting feature points, placing template blocks at the extracted feature points, and performing template matching with frame images.

  Here, if feature points are extracted from the entire image, the distribution of feature points is often non-uniform. When motion vectors obtained for non-uniform feature points are used for the purpose of camera shake correction, an area where features are concentrated is the main camera shake correction. Therefore, in order to distribute the feature points uniformly, the image is divided into grids, feature values representing the feature size are calculated for each pixel, and the pixel with the largest feature value in each grid is extracted as the feature point. There is a technique (for example, Patent Document 1).

  Furthermore, there is a technique for tracking feature points in order to improve the detection rate of motion vectors. Feature point tracking can be realized by sequentially detecting motion vectors of feature points extracted from an image over a plurality of continuous frame images (for example, Patent Document 2).

  FIG. 12 is a diagram showing an outline of the feature point tracking process. In FIG. 12A, a feature point 1001 is calculated in a feature point grid 1004 for performing feature point extraction, and template matching is performed in the template region 1003 to calculate a vector value 1004. Around the feature point extraction grid, feature point extraction is not performed, but a peripheral grid 1005 used for template matching is arranged.

  In the processing of the next frame, template matching is performed around a feature point (hereinafter referred to as a tracking destination feature point) 1002 as a tracking destination obtained by adding the calculated vector value 1004 to the feature point 1001. Thereafter, the feature points are tracked by adding the vector value to the tracking destination feature points over a plurality of frames.

  However, in the feature point tracking, the feature point that is being tracked disappears outside the angle of view or disappears because it is hidden behind some object, so that the feature point disappears and the tracking fails. There is. When the number of feature points to be tracked decreases due to such a tracking failure, it is necessary to set (complement) another feature point as a tracked object.

JP 2008-192060 A JP 2007-334625 A

  When template matching including feature point tracking processing is implemented as hardware, if tracking destination feature points are concentrated at the lower end of the angle of view as shown in FIG. Due to this processing time, there arises a problem that it does not end within a predetermined time.

  FIG. 13 is a diagram illustrating the processing timing of the feature point tracking processing (an example in which the processing time falls within one frame output period of the sensor). FIG. 14 is a diagram illustrating the processing timing of the feature point tracking processing (an example in which the processing time does not fit in one frame output period of the sensor).

  FIGS. 13A and 14A show the processing timing from sensor output to the end of template matching, and FIGS. 13B and 14B show the distribution of the tracking destination feature points within the angle of view. FIG. In FIG. 13A, the interval between pulses of the VD signal indicates a period of one frame of sensor output. A template matching image is generated for the image data output from the sensor. The template matching process is performed on the template matching image. Each period of the template matching circles 1 to 8 is the time required to execute one template matching process, and increases or decreases depending on the size of the template. In order to improve the performance of the image stabilization process, it is necessary to complete the template matching process within a certain period such as one frame period of the sensor output and feed it back to the image stabilization process for the next frame.

  When the tracking destination feature points are distributed and distributed within the angle of view as shown in FIG. 13B, the tracking destination feature points are shown in FIG. 14B even when the processing is finished. May concentrate on the lower side of the angle of view. In such a case, the period up to the template position 1201 in which the template matching process in FIG. 14B can be started with respect to the period up to the first template position 1101 in which the template matching process in FIG. 13B can be started (1100). (1200) becomes longer. Due to this delay, there may occur a case where the period of circle 1 to circle 8 of template matching is not completed within a certain period such as 1 V period of sensor output.

  In the technique disclosed in the related art, in the template matching for performing the feature point tracking as described above, the template is biased within the angle of view, and the solution to the problem that the template matching process does not end within a predetermined period is not mentioned. .

  The present invention has been made in view of the above problems. An object of the present invention is to set a tracking range from the initial coordinates of feature points in vector detection for performing feature point tracking, and when exceeding the tracking range, discard / compensation processing is performed within a predetermined period such as one frame period. It is to provide an imaging device that completes processing.

  An area dividing means for dividing the image into a plurality of areas, a feature point extracting means for extracting a predetermined number of feature points in each area, one of the two frame images in the video as an original image, and the other as a reference image, Vector detection means for detecting a motion vector by performing correlation value calculation processing on the pixel values in the rectangular area of the reference image centered on the feature point and in the rectangular area of the original image, and the vector value of the previous frame obtained by the vector detection means The tracking destination feature point calculating means for calculating the tracking destination feature point to be tracked of the subsequent frame from the feature point of the previous frame acquired by the feature point extracting means, and the feature point calculated by the tracking destination feature point calculating means is retained. The feature point holding means is characterized in that the vector detecting means reduces the search range of the feature points to be used in accordance with the held feature points.

  According to the present invention, when the vector detection area is biased in vector detection for performing feature point tracking, the vector detection processing is speeded up by reducing the template matching search range, and processing is performed within a predetermined period such as one frame period. Can be completed.

1 is a block diagram of a digital camera that is an imaging apparatus according to an embodiment of the present invention. It is a block diagram of the vector detection circuit with which the imaging device concerning an embodiment of the present invention is provided. It is a figure which shows the relationship between the grid arrangement | positioning which the vector detection circuit with which the imaging device which concerns on embodiment of this invention comprises, a feature point, and a template matching area | region is read. It is a figure which shows the processing flow of the vector detection circuit with which the imaging device which concerns on embodiment of this invention comprises. It is a block diagram of the feature point calculation means with which the imaging device concerning an embodiment of the present invention is provided. It is a graph which shows the relationship between a pixel value and determination by the image analysis in the accuracy determination means with which the imaging device which concerns on embodiment of this invention comprises. It is a block diagram of the template matching process means 203 with which the imaging device which concerns on embodiment of this invention comprises. It is a flowchart which shows the process of the template matching process means 203 with which the imaging device which concerns on embodiment of this invention comprises. It is a figure explaining the process of the area | region division means 801 with which the imaging device which concerns on embodiment of this invention comprises. It is a figure explaining the 1st template tracking range reduction 804 with which the imaging device concerning an embodiment of the present invention is provided. It is a figure explaining the 2nd template tracking range reduction 805 with which the imaging device which concerns on embodiment of this invention comprises. It is a figure which shows the outline of a feature point tracking process. It is a figure which shows the process timing of a feature point tracking process (example in which processing time is settled in 1 frame output period of a sensor). It is a figure which shows the process timing of a feature point tracking process (an example in which the processing time does not fit in one frame output period of the sensor).

  The best mode for carrying out the present invention will be described below with reference to the accompanying drawings. FIG. 1 is a block diagram of a digital camera which is an imaging apparatus according to the present invention. In FIG. 1, the imaging optical unit 101 includes a lens, a diaphragm, and the like. During imaging, the imaging optical unit 101 performs focus adjustment and exposure adjustment, and forms an optical image on the image sensor 102. The image sensor 102 has a photoelectric conversion function for converting an optical image into an electrical signal (analog image signal), and is configured by a CCD, a CMOS sensor, or the like. The A / D conversion unit 103 converts the analog image signal from the image sensor unit 102 into a digital image signal.

  The CPU 112 is configured by a microcomputer or the like that controls the entire imaging apparatus, and issues operation instructions to each functional block to execute various control processes. The bus 114 is a system bus, and the bus 115 is an image data bus. A DRAM (memory) 107 is a memory for storing data, and has a sufficient storage capacity for storing a predetermined number of still images, a moving image for a predetermined time, audio data, constants for operating the CPU 112, programs, and the like. Prepare. The memory control unit 106 performs data writing to and data reading from the DRAM 107 in accordance with instructions from the CPU 112 or the data transfer unit 105.

  The nonvolatile memory control unit 108 writes and reads data to and from a ROM (nonvolatile memory) 109 in accordance with instructions from the CPU 112. The ROM 109 is an electrically erasable / recordable memory, and an EEPROM or the like is used. The ROM 109 stores constants for operating the CPU 112, programs, and the like.

  The CPU 112 controls the image processing unit 104, the data transfer unit 105, the memory control unit 106, the nonvolatile memory control unit 108, the display control unit 110, the operation unit 113, and the image sensor 102 via the bus 114. The execution of the microcomputer realizes each process of the present embodiment by executing a program recorded in the ROM 109. The display unit 111 includes a liquid crystal monitor and the like, and is controlled by the display control unit 110 to display various image data and the like on the display unit 111. The operation unit 113 includes switches, buttons, and the like operated by the user, and is used for operations such as power ON / OFF and shutter ON / OFF.

  The image processing unit 104 includes various image processing units and a buffer memory, and includes a vector detection circuit. The data transfer unit 105 includes a plurality of direct memory access controllers (DMACs) that perform data transfer.

  The CPU 112 related to the image pickup apparatus according to the present invention controls the image pickup device unit 101 and stores a setting for reading an analog signal to the A / D conversion unit 103 in the ROM 109 as a read order setting. Further, the CPU 112 can read the reading order setting recorded in the ROM 109 and control each main image processing unit of the image processing unit 104 according to the setting.

  FIG. 3 is a diagram showing the relationship between the grid layout read by the vector detection circuit included in the imaging apparatus according to the embodiment of the present invention, the feature points, and the template matching area. Feature point extraction grids 0302 (white grid in FIG. 3) and peripheral grids 0301 (halftone grids in FIG. 3) of the set size are arranged in the number set in the horizontal and vertical directions. 0303 is extracted in each feature point extraction grid 0302. A peripheral grid 0301 is arranged around the feature point extraction grid and is not subjected to feature point extraction, but is a region used for template matching. A rectangular search area 0305 and a template area 0304 having a set size are provided around the extracted feature point 0303.

  FIG. 2 is a block diagram of a vector detection circuit included in the imaging apparatus according to the embodiment of the present invention. FIG. 4 is a diagram illustrating a processing flow of the vector detection circuit included in the imaging apparatus according to the embodiment of the present invention.

  Hereinafter, the outline of the processing of each block of the vector detection circuit will be described based on the processing flow of FIG.

Step S401 processing:
RDDMAC 1 (0221) included in the data transfer unit 105 reads the input image data 0241 of the current frame as a vector detection target from the DRAM 107 via the bus 115. The amount of data to be read depends on the size of the feature point extraction grid 0302 and the peripheral grid 0301 set as shown in FIG. The input image data 0241 is subjected to various image processing by the image processing unit 104. The image data 0241 read in units of grids by the RDDMAC 1 (0221) is output to the template matching image generation unit 0201 and the new feature point calculation unit 0202. Further, RDDMAC1 (0221) outputs grid coordinate information 0252 indicating the upper left coordinate position of the read grid to RDDMAC2 (0222).

Processing in step S402:
The template matching image generation unit 0201 generates a template matching image used for template matching at the time of vector detection, and outputs it to WRDMAC1 (0231). The vector detection image generation means 0201 is a bandpass filter circuit, and cuts the high frequency component and low frequency component of the image signal unnecessary for the template matching process.

Processing in step S403:
The WRDMAC 1 (0231) included in the data transfer unit 105 performs processing for writing the template matching image data 1 (0242) input to the DRAM 107 via the bus 115. The DRAM 107 stores template matching image data 2 (0243) generated in the previous frame.

Step S404:
In parallel with the processes in steps S402 and S403, the new feature point calculation unit 0302 calculates a new feature point of the current frame.

  Here, FIG. 5 is a block diagram of the new feature point calculation means 0302. A feature filter unit 0501, a feature evaluation unit 0502, and a feature point determination unit 0503 are included. The feature filter unit 0501 includes a plurality of filters such as a band pass filter, a horizontal differential filter, a vertical differential filter, and a smoothing filter. In the present embodiment, the band-pass filter cuts the high-frequency component and the low-frequency component of the unnecessary image signal, the signal subjected to the horizontal differential filter processing, and the signal subjected to the vertical differential filter processing, respectively In contrast, the result of smoothing filter processing is output.

  The feature evaluation unit 0502 has a large differential value around the pixel in many directions such as an intersection of two edges or a curved point having a maximum curvature for each pixel with respect to the grid filtered by the feature filter unit 0501. Is calculated as a feature value by a feature evaluation formula.

  Hereinafter, the Shi and Tomasi method used in the present embodiment will be described. An autocorrelation matrix H is created from the result of applying the horizontal differential filter and the vertical differential filter. Formula 1 of the autocorrelation matrix H is shown in Formula 1.

  In Expression 1, the result of applying the horizontal differential filter to Ix and the result of applying the vertical differential filter to Iy represent the Gaussian filter G. Shi and Tomasi's characteristic evaluation formula is shown in Formula 2.

  (Expression 2) indicates that the smaller eigenvalue of the eigenvalues λ1 and λ2 of the autocorrelation matrix H of (Expression 1) is used as the feature value.

  The feature point determination unit 0503 determines, for each grid, a pixel having the largest feature value calculated for each pixel by the feature evaluation unit 0502 as a feature point. In the present embodiment, the coordinates of the feature points are expressed by relative coordinates (PX, PY) where the upper left corner of the grid is (0, 0), but may be expressed by absolute coordinates in the image signal.

  Note that the calculated new feature point is stored in a memory included in the new feature point calculation unit 0202. The memory has a capacity for storing the feature points of the previous frame and the feature points of the current frame.

  When the template matching processing unit 0203 starts template matching processing for the corresponding grid, the feature points calculated in the previous frame and the feature points calculated in the current frame are tracked destination feature information 0251. It is output to the point determination means 205.

Step S405 processing:
In RDDMAC2 (0222), based on the input tracking feature point coordinate information 0257, a rectangular area centered on the feature point calculated in the previous frame is converted into the template matching image data 1 generated in the current frame and the previous frame. Is read out from the template matching image data 2 generated in the above.

  Note that the search area 0305 is read from the template matching image data 1, and the template area 0304 is read from the template matching image data 2 generated in the previous frame. Each read image data is output as search area image data 0253 and template area image data 0254 to the template matching processing unit 0203. However, since the feature point coordinates cannot be calculated from the previous frame in the first template matching process, all the feature point coordinates calculated in the current frame are used.

Step S406:
The template matching processing unit 0203 calculates a correlation value using the input search area image data 0253, template area image data 0254, and grid coordinate information 0252, and calculates a vector value from the correlation value. The calculated vector value information and correlation value information 0255 are output to the accuracy determining means 0204.

Step S407 processing:
In the accuracy determination unit 204, the correlation value information 0255 calculated in step S406 is used to obtain the maximum value, minimum value, average value, and minimum value of the correlation value, and low contrast determination, pixel value maximum value protrusion determination, and repetition. Perform pattern determination.

  FIG. 6 is a graph showing the relationship between the pixel value and each determination. However, in this embodiment, the smaller the correlation value is, the higher the similarity is. Therefore, the maximum value of the pixel value in the figure is the minimum value in the correlation value, the minimum value of the pixel value is the maximum value in the correlation value, and the maximum value of the pixel value is the correlation value Represents the local minimum.

  FIG. 6A shows a good result for each determination. In the low contrast determination, when the difference between the maximum value and the minimum value of the correlation values in the correlation value calculation area is smaller than a preset threshold value, the correlation value calculation area is determined to be low contrast.

  FIG. 5B shows the result of low contrast in the low contrast determination, and the difference between the maximum value and the minimum value of the pixel value is smaller in (b) than in (a). The maximum value protrusion determination of the pixel value determines how the minimum correlation value in the correlation value calculation region stands out. If the difference between the maximum and average pixel values and the difference between the maximum and minimum pixel values are smaller than a preset threshold value, the correlation value calculation area is determined to have a low peak. If the value is larger than the threshold value, it is determined that the correlation value calculation region has a high peak.

  FIG. 6C shows a result of a low peak in the maximum value protrusion determination of the pixel value. In FIG. 5C, the difference between the maximum value and the average value of the pixel value and the maximum value of the pixel value are compared in FIG. The difference between the minimum value and the minimum value is small. The repeated pattern determination is determined to be a repeated pattern when the difference between the minimum value and the minimum value of the pixel values in the correlation value calculation region is smaller than a preset threshold value. FIG. 4D shows the result of the repeated pattern determination, and the difference between the maximum pixel value and the maximum value is smaller in (d) than in (a).

Step S408 processing:
Low contrast, low peak, repetitive pattern determination information and vector information 0256 are output to SRAM (memory) 0206.

Processing in step S409:
The tracking destination feature point determination unit 205 calculates a tracking destination feature point to be used for template matching processing of the next frame. The calculated tracking destination feature points are output as tracking destination setting information 0257 to the SRAM (memory) 206 and RDDMAC 2 (0222).

  The above is the processing flow of the vector detection circuit in the embodiment of the present invention.

[First Embodiment]
Processing contents of the template matching processing unit 203 in the imaging apparatus according to the embodiment of the present invention will be described with reference to FIGS. FIG. 7 is a block diagram of the template matching processing means 203. FIG. 8 is a flowchart showing the processing of the template matching processing means 203. FIG. 9 is a diagram for explaining the processing of the area sorting unit 801. FIG. 10 is a diagram for explaining the processing of the first template area reduction unit 804. FIG. 11 is a diagram for explaining the processing of the second template area reduction unit 805.

  The outline of the template matching processing means 203 will be described below with reference to the flowchart of FIG. In this embodiment, the CPU 112 can control each processing block in the template matching processing unit 203. Further, the vector information holding unit 803 includes a memory or the like, and associates the vector value information and the correlation value information calculated by the template matching process execution unit 807 for each tracking destination feature point, and holds for four frames in advance. The CPU 112 can read out desired vector value information and correlation value information held in the vector value holding unit 803.

□ Step S901:
The CPU 112 inputs the grid coordinate information of the current frame to the area sorting unit 801, calculates the designated area to which the corresponding grid belongs, and proceeds to step S902.

  Here, the method for calculating and setting the designated area in the embodiment of the present invention will be specifically described with reference to FIG. FIG. 9A is a conceptual diagram showing the image reading order when the current frame is divided into a total of 25 grids of 5 horizontal divisions and 5 vertical divisions. In the case of FIG. 9A, image reading of the current frame is performed by a raster scan method as indicated by an arrow. FIG. 9B is a conceptual diagram showing designated areas allocated to 25 grids in the case of the raster scan method. The second designated area is assigned as the first half of the three vertical grids from the upper end. The first designated area is assigned as the second half of the vertical 2 grid from the lower end.

  In the present embodiment, the first designated area is assigned as the second half of the vertical two grids from the lower end, but there is no particular limitation, and the first designated area and the second designated area may be assigned to any area. it can.

□ Step S902 processing:
The CPU 112 determines whether or not the designated area calculated by the area sorting unit 801 is the first designated area. If it is the first designated area, the process proceeds to step S903. If it is not the first designated area, the search area image data is not updated and is output to the template matching process execution unit 807, and the process proceeds to step S911.

□ Step S903 processing:
The CPU 112 updates the counter of the corresponding grid from the grid coordinate information of the current frame from the counter group for each grid provided in the feature point counting means 802, and the process proceeds to step S904.

□ Step S904 processing:
The CPU 112 reads the threshold value set in the ROM 109 and determines whether or not the counter of the corresponding grid exceeds the threshold value. If the counter does not exceed the threshold, the CPU 112 outputs the current search area image data to the template matching process execution unit 807, and the process proceeds to step S911. On the other hand, if the counter exceeds the threshold, the process proceeds to step S905.

  Note that the threshold value is set in advance for each grid, and can be appropriately updated via the CPU 112 in accordance with shooting conditions and the like.

□ Step S905 processing:
The CPU 112 reads the tracking destination feature points existing in the grid from the vector value holding unit 803, and determines whether the vector direction is constant horizontally and vertically from the difference between the correlation values. If it is determined that the vector direction is constant, the vector direction is output to the first template tracking range reduction 804, and the process proceeds to step S906. On the other hand, if it is determined that the vector direction is not constant, the process proceeds to step S907.

  Here, a method for determining whether or not the direction of the vector in the embodiment of the present invention is constant horizontally and vertically will be described in detail with reference to FIG. FIG. 10A shows the correlation values for four frames held in the vector information holding unit 803. Here, in FIG. 10A, x and y indicate the horizontal and vertical coordinates of the correlation value. Further, dx and dy indicate differences in correlation values between the frames. The CPU 112 determines whether or not the vector direction is constant using (Equation 3).

dmin1 <dx <dmax1 (Formula 3)
dmin1 <dy <dmax1
Here, the upper limit threshold value dmin1 and the lower limit threshold value dmax1 are shown in Expression 3, and the CPU 112 determines that the vector direction is constant when dx or dy is within the range of (Expression 3) between all the held frames. To do. Note that the threshold value is set in advance for each grid, and can be appropriately updated via the CPU 112 in accordance with shooting conditions and the like.

□ Step S906 processing:
The first template tracking range reduction 804 reduces the template tracking range based on the input vector direction, and proceeds to step S907.

  Here, the processing of the first template tracking range reduction 804 in the embodiment of the present invention will be described with reference to FIGS. 10B and 10C. FIG. 10B shows a conceptual diagram of the template tracking range before executing the first template tracking range reduction 804 in this embodiment. The template 304 used for template matching in the present embodiment is a square, and the template tracking range 305 moves the template 304 by moving ± mx in the horizontal direction and ± my in the vertical direction. FIG. 10C shows a conceptual diagram of the template tracking range after the first template tracking range reduction 804 is executed when the CPU 112 determines that the horizontal direction is constant. When the CPU 112 determines that the horizontal direction is constant, the first template tracking range reduction 804 reduces the vertical movement amount from ± my to ± my / 2.

□ Step S907 processing:
The CPU 112 reads the tracking destination feature points existing in the grid from the vector value holding unit 803, calculates the difference of the motion vector value information, and determines whether or not the speed is less than the threshold value. If it is determined that the speed is less than the threshold, the process proceeds to step S908. On the other hand, if it is determined that the speed is greater than or equal to the threshold, the process proceeds to step S909.

□ Step S908 processing:
The second template tracking range reduction 805 reduces the template tracking range, and the process proceeds to step S909.

□ Step S909 processing:
The estimation unit 804 calculates a processing time coefficient for estimating the processing time from the number of operations for calculating the correlation value according to the updated template region image data size, and the process proceeds to step S910.

□ Step S910 processing:
Based on the calculated processing time coefficient, the CPU 112 determines whether or not the template matching process centered on the tracking destination feature point is completed within a predetermined time. If it is determined that the process is to be completed within a predetermined time, the reduced template area image data is output to the template matching process execution unit 805, and the process proceeds to step S911. On the other hand, if it is determined that the process does not end within a predetermined time, the template matching process for the corresponding tracking feature point is ended.

□ Step S911 processing:
The template matching process execution unit 805 calculates a correlation value using the input search area image data and template area image data, and calculates a vector value from the correlation value. The calculated vector value information and correlation value information are output to the accuracy determining means 0204, and the template matching process is terminated.

  In this embodiment, the correlation value is calculated using a sum of absolute difference (hereinafter abbreviated as SAD).

  In Expression 4, f (i, j) represents a pixel value at the coordinates (i, j) in the template area image data 0254, and g (i, j) is an object of correlation value calculation in the search area image data 0253. Represents each pixel value in the region. The correlation value calculation target area is the same size as the template area image data 0254. In SAD, the absolute value of the difference is calculated for each pixel value f (i, j) and g (i, j) in both blocks, and the correlation value S_SAD can be obtained by obtaining the sum. Therefore, the smaller the correlation value S_SAD, the smaller the difference in luminance value between the two blocks, that is, the template area image data 0254 and the texture in the correlation value calculation area are similar.

  In the present embodiment, the size of the search area image data 0253 is reduced by the first template tracking range reduction 804 and the second template tracking range reduction 805, thereby reducing the SAD calculation amount described above and the period of one frame. Within the template matching process can be completed.

  The above is the processing flow of template matching in the embodiment of the present invention. Although this processing frame shows processing for one frame, the same processing is performed for each frame, and a vector value is calculated for each frame.

101 Imaging Optical Unit 102 Image Sensor 103 A / D Converter 112 CPU

Claims (8)

Area dividing means for dividing an image into a plurality of areas;
Feature point extraction means for extracting a predetermined number of feature points in each region;
One of the two frame images in the video is the original image, the other is the reference image,
A vector detection means for performing a correlation value calculation process on the pixel values in the rectangular area of the reference image centered on the feature point and in the rectangular area of the original image, and detecting a motion vector;
A tracking destination feature point calculating means for calculating a tracking destination feature point to be tracked of a subsequent frame from the vector value of the previous frame acquired by the vector detecting means and the feature point of the previous frame acquired by the feature point extracting means;
Comprising feature point holding means for holding the feature points calculated by the tracking destination feature point calculating means;
An image pickup apparatus characterized in that the vector detection means reduces a search range of feature points to be used in accordance with held feature points. A region segmentation means for determining which region of the at least two region segments the feature point belongs to, based on the acquired vector value and another vector value, the tracking destination feature point;
The region classifying means designates a region corresponding to the second half of the sequentially input pixels of the frame image as a first designated region, and a region corresponding to the first half of the pixels sequentially input of the frame image as a second designation. The imaging apparatus according to claim 1, wherein the imaging apparatus is a region. The vector detection means reduces the search range of feature points used for template matching when the area classification means determines that the feature points belong to a first designated area. 2. The imaging device according to 2. The imaging apparatus according to claim 1, further comprising a feature point attribute calculating unit that calculates a direction and a speed by comparing the feature points held by the feature point holding unit. . The imaging apparatus according to claim 4, further comprising a direction determining unit that determines whether or not the direction of the feature point calculated by the feature point attribute calculating unit is constant. The imaging apparatus according to claim 4, wherein when the direction is determined to be constant in the horizontal direction by the direction unit, the search range in the vertical direction is limited during vector detection. The imaging apparatus according to claim 4, wherein when the direction is determined to be constant in the vertical direction by the direction unit, a search range in a horizontal direction is limited when detecting a vector. A threshold setting means for setting a speed threshold; and a speed determining means for determining whether or not the speed of the feature point calculated by the feature point attribute calculating means is equal to or lower than the threshold. The speed is determined by the speed determining means. The imaging apparatus according to claim 1, wherein when it is determined that the threshold value is equal to or less than the threshold value, the search range is deleted when the vector is detected.