JP2018072895A - Image processing device, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce occurrence of a redundant process in detecting a motion vector.SOLUTION: An image processing device defines a search area to perform a correlation operation in a second image for a matching process to search a high correlation area in the second image based on a first image. The device determines a feature point for each divided area for the first image in which a plurality of divided areas are defined, and sets the search area in the second image based on the determined feature point for the divided area for each of the plurality of divided areas. In the setting, when the feature point satisfies a predetermined condition among at least two divided areas of the plurality of divided areas, one search area is set to the at least two divided areas such that the matching processes related to the at least two divided areas are integrated.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly to a motion vector detection technique based on correlation between images.

  In the field of imaging devices such as digital video cameras, one of the methods for reducing image blur caused by camera shake during movie shooting is to estimate the amount of motion of the imaging device from a plurality of frame images, and to form an image image There are some which control the optical system so that is stabilized. The estimation of the amount of motion based on a plurality of frame images is performed based on a motion vector obtained by specifying a region in another frame image having the highest correlation, for example, by template matching using a specific pattern as a template image. In Patent Document 1, in order to appropriately estimate a motion amount that reduces image blur at high speed, an image is divided into a plurality of blocks, and the feature point indicating the maximum feature amount determined for each block is described as the feature point. A technique for deriving a motion vector using a template image including the template image is employed.

JP 2008-192060 A

  By the way, in the method of deriving a motion vector for each of a plurality of blocks as in Patent Document 1, an inefficient calculation can be performed when an image shows a specific pattern.

  For example, as shown in the block 1201a of FIG. 12, when the feature point 1202a indicating the maximum feature amount exists near the center of the block, the template image 1203a centering on the feature point also stays in the block, so that other blocks It is hard to overlap with the template image. That is, in the template matching process (hereinafter referred to as matching process) for the search area 1204a using the template image 1203a that does not overlap the template image related to the other block, an operation similar to the motion vector derivation related to the other block does not occur. .

  On the other hand, for example, in a mode in which a pattern showing a high feature amount is divided into different blocks, the feature points 1202b to 1202e can be determined close to the block periphery as shown in the adjacent blocks 1201b to 120e in FIG. In such a case, since the template image and the search area determined for each block are similar in content, in the matching process, the same correlation calculation is performed for the same overlapping pattern, or redundant memory access occurs, which is inefficient. there were.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide an image processing apparatus, an image processing method, and a program that reduce the occurrence of redundant processing when detecting a motion vector.

  In order to achieve the above-described object, the image processing apparatus according to the present invention sets a search area for performing a correlation operation to a second area for a matching process for searching for a highly correlated area in the second image based on the first image. An image processing apparatus defined for an image, wherein a determination unit that determines a feature point for each divided region for a first image in which a plurality of divided regions are determined, and for each of the divided regions Setting means for setting a search area in the second image based on the determined feature point, and the setting means has a predetermined feature point between at least two of the plurality of divided areas. When the condition is satisfied, one search area is set for the at least two divided areas so as to integrate the matching processing related to the at least two divided areas.

  With such a configuration, according to the present invention, it is possible to reduce the occurrence of redundant processing when detecting a motion vector.

The block diagram which showed the function structure of the digital video camera 100 which concerns on embodiment and the modification of this invention The block diagram which showed the detailed structure of the motion vector detection part 105 The figure for demonstrating the division | segmentation aspect of the image which concerns on embodiment and modification of this invention The figure for demonstrating the output of the image of a surplus area | region and a block based on embodiment and the modification of this invention The block diagram which showed the detailed structure of the characteristic analysis part 203 The flowchart which illustrated the setting process which the setting part 205 which concerns on embodiment of this invention performs The figure for demonstrating the setting of the various area | region performed per object block based on embodiment of this invention The figure which showed the processing timing of each block of the motion vector detection part 105 based on embodiment of this invention. The figure for demonstrating the setting of the various area | region when integrating the matching process based on embodiment of this invention The flowchart which illustrated the setting process which the setting part 205 which concerns on the modification of this invention performs The figure for demonstrating the setting of the various area | region when integrating the matching process based on the modification of this invention The figure for demonstrating generation | occurrence | production of the redundant process in the case of calculating a motion vector for every area | region which divided | segmented the image based on a prior art.

[Embodiment]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the embodiment described below, an example in which the present invention is applied to a digital video camera that performs camera shake correction processing based on a motion vector between two consecutively captured images as an example of an image processing apparatus. Will be explained. However, the present invention is applicable to any device that can set a search region for matching processing related to motion vector detection for each of a plurality of divided regions (blocks) included in an image.

<< Configuration of Digital Video Camera 100 >>
FIG. 1 is a block diagram showing a functional configuration of a digital video camera 100 according to an embodiment of the present invention.

  The imaging optical system 101 includes a plurality of lens groups including a focus lens and an anti-vibration lens, and a stop. The imaging optical system 101 acts to guide the reflected light of the subject to the imaging unit 102 described later. More specifically, the light beam that has passed through the imaging optical system 101 forms an optical image of the subject on the imaging unit 102. Here, the anti-vibration lens is driven and controlled by the drive unit 112 operating based on a drive command output by the drive control unit 111 described later. In the digital video camera 100 of the present embodiment, mechanical vibration isolation is realized by the operation of such an image stabilization lens, and image blur due to camera shake or the like is reduced. Note that the focus lens may also be controlled by the drive control unit 111 and the drive unit 112, and operates to adjust the focus position. The drive of the focus lens may be realized, for example, by controlling the rotation of the focus ring.

  The imaging unit 102 is an imaging element such as a CCD or a CMOS sensor. The imaging unit 102 photoelectrically converts an optical image formed on the imaging surface of the imaging element and outputs an analog image signal. The output analog image signal is A / D converted by the A / D conversion unit 103 and output to the image processing unit 104 as a digital image signal (hereinafter simply referred to as an image signal).

  The image processing unit 104 executes various image processes performed in the digital video camera 100 of the present embodiment. The image processing unit 104 performs development processing such as WB correction processing, conversion to a color difference signal or RGB signal including application of color gain, and gamma correction processing on a digital image signal input from the A / D conversion unit 103, for example. The image data for recording or display is generated. In the present embodiment, among the image processing performed by the image processing unit 104, the motion vector detection processing is shown as a separate functional configuration that the motion vector detection unit 105 is independently responsible for. However, it will be easily understood that the image processing unit 104 and the motion vector detection unit 105 may be realized as one hardware or a plurality of hardware. .

  The system control unit 106 is a CPU, for example, and controls the operation of each block included in the digital video camera 100. For example, the system control unit 106 reads out an operation program for each block recorded in the recording medium 109, develops it in the memory 110, and executes it to control the operation of each block. The system control unit 106 also acquires state information such as lens position information of the imaging optical system 101.

  The recording medium 109 is a recording device configured to be able to hold permanent information such as a nonvolatile memory built in the digital video camera 100 or a memory card detachably attached to the digital video camera 100. . The operation program for each block included in the digital video camera 100 described above may be stored in, for example, a built-in nonvolatile memory, and other information such as parameters necessary for the operation of each block is stored in the memory. May be recorded. On the other hand, the memory 110 is a recording device such as a volatile memory used for a work area or temporary information storage. The memory 110 is used not only as an operation program development area, but also as a storage area for various data output in the operation of each block or required to be held in the middle of the operation of each block.

  The operation unit 107 is a user interface such as various switches, dials, shooting buttons, and a touch panel provided in the digital video camera 100. When the operation unit 107 detects that an operation input to the user interface has been made, the operation unit 107 analyzes the input and outputs a corresponding control signal to the system control unit 106.

  The display unit 108 is, for example, a display device such as an LCD, and various graphical user interfaces such as an image related to imaging subjected to image processing by the image processing unit 104, an image recorded on the recording medium 109, and other setting screens. Used for display.

  In the present embodiment, the functional configuration of the digital video camera 100 will be described assuming that processing is realized by hardware such as a circuit and a processor corresponding to each block. However, the embodiment of the present invention is not limited to this, and the processing of each block may be realized by a program that performs similar processing.

<< Configuration of Motion Vector Detection Unit 105 >>
Next, the configuration of the motion vector detection unit 105 included in the digital video camera 100 according to the present embodiment will be described in detail with reference to FIG. For example, in the mode in which execution of a function related to image stabilization is set, the motion vector detection unit 105 receives an image signal and performs processing. In the present embodiment, for the sake of simplicity, the description will be made assuming that the target of motion vector detection processing in the motion vector detection unit 105 is an image signal picked up in two consecutive frames. However, the present invention is not limited to the motion vector detection application related to the image stabilization function, and can be used if the correlation between a plurality of images is obtained for each of a plurality of blocks obtained by dividing a reference image. It will be understood that this is possible.

  In the description of each configuration of the motion vector detection unit 105 shown below, an image signal serving as a reference for motion vector detection, that is, an image (subsequent frame image) captured later in time series, Let it be a “reference image” as one image. In addition, when detecting a motion vector, an image signal in which a region having a high correlation with the reference image is searched, that is, an image (previous frame image) captured earlier in time series (before the reference image) is added to the present invention. This will be described as a “target image” as the second image. In addition, at the time of detecting a motion vector for a reference image, it is assumed that at least pixels of a target image necessary for searching are stored in an image memory 204 described later.

  The block transfer unit 201 separates the input reference image into an image having a size (processing unit) that is handled in the subsequent processing, and transfers the image to the subsequent block. In the present embodiment, the reference image 300 is separated into blocks (rectangular region groups indicated by white background) separated into a plurality of divided regions in the horizontal direction 6 × vertical direction 4 by a grid as illustrated in FIG. 3. Note that the area outside the block in the matching process is also set as a reference area according to the present invention as a template image used for the matching process or a search area for searching for the position with the highest correlation for the template image. obtain.

  In the present embodiment, a feature point is determined for each block, and a rectangular area having a predetermined size around the feature point is set as a template image. A rectangular region larger than the template image is set as a search region (indicating a coordinate range in which the matching processing is performed in the target image) that is a range in which correlation calculation with the template image is performed in the target image. In each block in FIG. 3, the search area is shown on the reference image in order to indicate the relative positional relationship between the template image set for the reference image 300 and the search area set for the target image. For example, when the feature point 302a is determined near the center of the block as in the block 301a, both the template image 303a and the search area 304a fit in the block. On the other hand, when the feature point 302b is determined near the block upper boundary as in the block 301b, the template image 303b and the search region 304b can be set including pixels outside the block.

  Therefore, the block transfer unit 201 considers the case where the template image 303 including the pixels outside the block and the search region 304 are set for the block located near the end of the reference image, and the surplus region indicated by hatching in FIG. The reference image is separated to provide 305. That is, by providing the surplus area 305, the template image 303 and the search area 304 including pixels outside the block are set, and even if the pixels outside the block do not belong to any block, the surplus area 305 is It is configured to be referable.

  In this embodiment, the block transfer unit 201 stores the reference image related to the separated block and the surplus area in the image memory 204 that performs reading (reference image and target image) in the matching process. 202. The block transfer unit 201 also transfers the partial image related to the block to the feature analysis unit 203 in order to determine the feature point of the reference image related to the block. As shown in FIG. 4, the partial image is transferred by sequentially switching the pixels of each row of the partial image from the left to the lowermost row of the partial image. Therefore, the block transfer unit 201 reads the image of the surplus area 402a from the storage area (not shown) and transfers it to the image conversion unit 202, then reads and transfers the surplus area 402b, and sequentially selects and transfers the right surplus area. I will do it. Thereafter, when all the surplus areas at the upper end of the reference image 400 are transferred to the image conversion unit 202, the block transfer unit 201 reads out the surplus area 402c existing on the left of the block 401a and transfers it to the image conversion unit 202. Next, the block transfer unit 201 performs processing of reading the image of the block 401a and transferring it to the image conversion unit 202 and the feature analysis unit 203 by sequentially switching the block 401b, the block 401c,. Can be transferred.

  In addition, since the image of the surplus area is transferred only to the image conversion unit 202, when the transfer is being performed, a configuration may be adopted in which power consumption is reduced by, for example, stopping the clock supplied to the feature analysis unit 203. . In the present embodiment, for the sake of simplicity, the entire reference image is input to the block transfer unit 201. The block transfer unit 201 stores the input reference image in a storage area (not shown), and based on a determined grid, A description will be given assuming that the reference images for transfer are sequentially read from the area. However, for example, if the reference image is read by the raster scan method and the sequentially read pixels are input to the block transfer unit 201, the raster block conversion is performed by a known method, so that the image May be realized.

  The image conversion unit 202 converts the input image into an image (characteristic for detection) having characteristics suitable for matching processing for detecting a motion vector, and stores the image in the image memory 204. The image conversion unit 202 is, for example, a band-pass filter circuit, and may be configured to cut high-frequency components and low-frequency components of image signals that are unnecessary for matching processing.

  The image memory 204 is a storage device that stores a detection image used for matching processing. When detecting the motion vector related to the reference image, the image memory 204 stores a detection image obtained by converting the reference image (related to the current frame), a detection image obtained by converting the target image (related to the previous frame), and Is stored.

  The feature analysis unit 203 determines a feature point for each input image of the block, and outputs the information (feature point information) to the setting unit 205. The process in the feature analysis unit 203 may be performed in parallel with the conversion process in the image conversion unit 202.

<Configuration of Feature Analysis Unit 203>
Here, the functional configuration related to the output of the feature point information in the feature analysis unit 203 will be described in detail with reference to the block diagram of FIG.

  The filter unit 501 performs a filter process for converting an input block image into an image having characteristics suitable for feature amount evaluation. The filter unit 501 may include a plurality of filters such as a band pass filter, a horizontal differential filter, a vertical differential filter, and a smoothing filter. For example, the filter unit 501 cuts an unnecessary high-frequency component and low-frequency component of the image signal with a band-pass filter and applies a horizontal differential filter process and a vertical differential filter process to the signal. On the other hand, the result of smoothing filter processing is output.

  The feature amount calculation unit 502 calculates a feature amount for each pixel of the block image filtered by the filter unit 501. The feature amount calculation unit 502 performs predetermined feature evaluation so that a high feature amount is indicated at a point where the differential value around the pixel is large in many directions, such as an intersection of two edges or a curved point having a maximum curvature. A feature amount is calculated using an equation. The feature amount calculation unit 502 may calculate the feature amount of a pixel using, for example, the Shi and Tomasi feature evaluation formula as follows.

First, the feature amount calculation unit 502 creates an autocorrelation matrix H based on the result of applying the horizontal differential filter and the vertical differential filter. The autocorrelation matrix H is
Is derived by Here, Ix represents the result of applying the horizontal differential filter, Iy represents the result of applying the vertical differential filter, and the autocorrelation matrix H appears by convolution of the Gaussian filter G.

Shi and Tomasi's feature evaluation formula is
The smaller eigenvalue of the eigenvalues λ1 and λ2 of the autocorrelation matrix H is calculated as the feature amount of the pixel.

  The feature point determination unit 503 determines the feature point of the block based on the feature amount of each pixel included in the block calculated by the feature amount calculation unit 502. In the present embodiment, the feature point determination unit 503 will be described as determining the pixel having the highest feature amount among the pixels in the block as the feature point, but the present invention is not limited to this. There is no need. For example, the feature point determination unit 503 selects, as a feature point, a pixel in a block that is closest to a pixel that is estimated to be able to show a peak of the feature amount, not only inside or outside the block, based on the distribution of the feature amount in the block. It is good also as a structure. When the feature point determination unit 503 determines a pixel to be a feature point, the feature point information is generated by associating the information specifying the block with the coordinate information of the pixel as the feature point in the block. The coordinates in the block are expressed by relative coordinates (PX, PY) where the coordinates of the pixel at the upper left corner of the block are (0, 0). The generated feature point information is temporarily stored in a memory or register (not shown) in the feature point determination unit 503, and then, when the setting unit 205 starts preprocessing of matching processing related to the block. And output to the setting unit 205.

  For the matching process performed for each block of the reference image, the setting unit 205 determines the region in the reference image that is used as the template image and the region in the target image that is used as the search region for the feature points determined for the block. Determine based on information. Further, the setting unit 205 determines the reading position of the image memory 204 from the image memory 204 based on the information of the area to be a template image and the search area determined for the matching process related to each block, and issues a read request (memory access). Do. In this embodiment, based on the memory access made by the setting unit 205, a memory controller (not shown) sequentially reads out the pixels stored at the corresponding address, and outputs them to the matching processing unit 206 as a template image or a search region image. To do.

  The matching processing unit 206 executes matching processing for each block using the template image input for each block of the standard image and the image of the reference area. The matching process is a process of searching for a region having the highest correlation with the template image defined for the reference image in the search region defined for the target image as described above. The matching processing unit 206 derives the degree of correlation (correlation value) by a predetermined calculation while sequentially changing the region (range) to be calculated in the search region, and specifies the region having the highest correlation. When the matching processing unit 206 identifies a region having the highest correlation for one block, the matching processing unit 206 calculates a motion vector related to the block based on a relative positional relationship with the template image. When the calculation of the motion vector for all the blocks of the reference image is completed, the matching processing unit 206 calculates and outputs a motion vector related to the entire image between the reference image and the target image.

As the correlation value, for example, a sum of absolute differences (SAD) is used,
It may be what you want. Here, f (i, j) is a pixel value at the coordinates (i, j) in the template image 704, and g (i, j) is each pixel value in the region for which the correlation value is calculated in the search region 705. is there. The target area for correlation value calculation is set to have the same size as the template image 704 in the search area. As shown in the equation, in the SAD, the absolute value of the difference between the pixel value f (i, j) in the template image and the pixel value g (i, j) at the corresponding position in the target region is calculated. The correlation value S_SAD is obtained by calculating the sum. Therefore, the smaller the correlation value S_SAD, the smaller the difference between the brightness values of the template image and the target area, that is, the texture of the template image 704 and the target area image is similar.

  Note that the calculation of the correlation value is not limited to the method using SAD, and it will be easily understood that other methods using sum of squares of differences (SSD), normalized cross-correlation (NCC), and the like may be used.

<Overview of reducing redundant processing>
In addition, in the motion vector detection unit 105 of this embodiment, in order to reduce the occurrence of redundant processing for the derivation of motion vectors performed by dividing the reference image into blocks, the feature points of adjacent blocks are reduced. Depending on the relationship, whether or not the matching process is executed is varied. More specifically, the motion vector detection unit 105 of the present embodiment integrates the matching processing performed on these blocks into one for a plurality of blocks in which the feature points are not separated from each other by a predetermined distance.

  As illustrated in FIG. 7, the feature point 703 determined for the target block 701 is separated from the block boundary by a predetermined distance (half the length of one side of the template image), or exists near the block boundary. If not, there is little occurrence of redundant processing. In other words, matching processing performed based on the feature points determined for the peripheral blocks 709a to 709h (blocks adjacent to the target block 701 or touching only the vertices) existing around the target block 701, and an overlapping operation may occur. The nature is low. This is because, even if a template image 704 centering on the feature point 703 is set as shown in the drawing, it does not overlap with the template image set for the peripheral block 709, or even if it overlaps, the number of pixels is relatively small. by.

  On the other hand, as illustrated in FIG. 9A, when the feature point 703 determined for the target block 701 exists in the vicinity of the block boundary, redundant processing may occur. In the example of FIG. 9A, the feature point 703 is near the lower right boundary of the target block 701, and the feature point 901e determined for the peripheral block 709e and the feature point 901h determined for the peripheral block 709h are close to each other. Yes. In such a case, the matching process performed by setting a template image of a predetermined size with each feature point as the center includes a correlation operation for the same search region for pixels overlapping between the template images. Redundant processing with similar results can occur. The situation where the feature points as shown in FIG. 9A are close to each other can occur in a case where there are characteristic textures such as dense edges, and these are separated into a plurality of blocks.

  Therefore, in the motion vector detection unit 105 according to the present embodiment, if there are feature points that are determined to be close in the distribution of feature points in the reference image, redundant blocks generated in the matching process for blocks related to these feature points are included. Reduce computation. More specifically, when there are blocks that are determined to be close to each other's feature points, the setting unit 205 does not set a template image and a search area for each of these blocks, but instead sets one for each of these blocks. A template image and a search area are set. That is, the matching process that was scheduled to be performed for these blocks is integrated and performed as one matching process instead.

  In the example of FIG. 9A, when the setting unit 205 calculates the reading range of the template image 902 and the search area 903 for one target block 701, the search area 903 exceeds the right and lower boundaries of the block. . For this reason, the setting unit 205 further calculates distances from the feature points 901e, g, and h of the peripheral blocks 709e, g, and h included in the search region 903 based on the feature point information obtained from the feature analysis unit 203. To determine whether they are close to each other. The determination of whether or not they are close may be made according to whether or not the distance D calculated for the feature points is below a predetermined threshold. At this time, since the setting unit 205 determines that the feature points 901e and h are close to the feature point 703, the setting unit 205 does not set the template image 902 and the search region 903 based on the calculation result for the target block 701. It is determined that the block and the matching process are integrated.

  In other words, in this embodiment, the setting unit 205 performs matching processing on a certain block when there is a block for which a feature point is determined to be close to at least one of the peripheral blocks of the block. Integrate. In other words, if there are at least two blocks that are determined to be close to each other among the plurality of blocks defined for the reference image, the motion setting unit 205 has one template image for each of these blocks. And set only the search area.

  In the motion vector detection unit 105 of this embodiment, the setting unit 205 selects each block according to the order of the blocks output by the block transfer unit 201, that is, the order in which the detection images are stored in the image memory 204. Set the template image and search area. Therefore, the setting unit 205 performs a single matching process in which the template image and the search area determined for the block with the slowest storage order among the blocks determined to have feature points close to each other are integrated for these blocks. Set as used. In the example of FIG. 9A, the peripheral block 709h is the block in which the detection image is stored in the image memory 204 in the slowest order. For this reason, a template image 911 and a search area 912 defined around a feature point 901h as shown in FIG. 9B are set for a matching process performed by integrating the target block 701 and the peripheral blocks 709e and h. The

《Setting process》
Hereinafter, with respect to the setting process for setting the template image and the search area for the matching process related to each block, which is performed by the setting unit 205 of the present embodiment having such a configuration, specific processing is performed using the flowchart of FIG. explain. The processing corresponding to the flowchart can be realized by the system control unit 106 reading, for example, a corresponding processing program stored in the recording medium 109, developing it in the memory 110, and causing the setting unit 205 to execute it. This setting process will be described as being started when, for example, feature point information about the first block (upper left block) related to the reference image is input to the setting unit 205.

  In step S601, the setting unit 205 calculates, for a block (target block) corresponding to the input feature point information, a coordinate range of a template image and a coordinate range of a search area used in matching processing related to the block. Hereinafter, the calculation method will be described by taking as an example the case where the feature points 703 are arranged for the target block 701 as shown in FIG. In the following description, “absolute coordinates” refers to the coordinates of pixels determined based on the upper left pixel of the reference image, and “relative coordinates” refers to the upper left pixel (start position 702) of the target block 701. The pixel coordinates in the block determined by the relative relationship are indicated.

When the absolute coordinates (GPX, GPY) of the feature point 703 are (GSX, GSY) as the absolute coordinates of the start position 702 and (PX, PY) are the relative coordinates of the feature point 703 obtained from the feature point information,
Can be expressed as At this time, the absolute coordinates (GTX) of the reading start position 706 of the template image 704 determined with the horizontal size TX and the vertical size TY around the feature point 703 is
Can be expressed as The absolute coordinates of the reading completion position of the template image 704 (the lower right pixel of the template image 704) are values obtained by adding the horizontal direction TX and the vertical direction TY to the absolute coordinates of the start position 706, respectively.

Similarly, the absolute coordinates (GMX, GMY) of the image reading start position 707 in the search area 705 determined by having the horizontal size SX and the vertical size SY around the feature point 703 are as follows.
Can be expressed as Further, the absolute coordinates (GEX, GEY) of the image reading completion position of the search area 705 (the lower right pixel of the search area 705) are:
Can be expressed as

  In step S602, the setting unit 205 determines whether the matching process related to the target block can be started. The determination in this step is that the coordinates of the image read completion position calculated in S601 are in a block in which no image is stored in the image memory 204 or in a block that has not yet received feature point information. Is done based on whether or not. For simplicity, the determination in this step is based on the coordinates (GEX, GEY) of the readout completion position of the search area and the coordinates (GSX, GSY) of the start position of the peripheral block for each of a plurality of peripheral blocks. It may be performed by comparison.

  Here, generation and storage of detection images, generation of feature point information, and execution timing of matching processing are illustrated in FIG. In FIG. 8, the right direction indicates the direction of time progression. Further, in order from the top, the processing of the image conversion unit 202, the processing of the feature analysis unit 203, the processing of the matching processing unit 206 when the matching processing can be started at the time of S602 (case A), and the case where the matching processing cannot be started (case B) The processing timing of the matching processing unit 206 is shown. The example illustrated in FIG. 8 illustrates an example in which the peripheral blocks 709a to 709d and f illustrated in FIG. 7 are surplus areas.

  Since the matching process related to the target block 701 needs to read out pixels outside the target block 701 as a template image and a search area, the matching process can be executed in case A and case B as shown in the figure. Timing changes. That is, the timing at which the matching process can be started varies depending on the coordinates (PX, PY) of the feature point 703 in the target block 701. In case A as shown in FIG. 7, the template image and the image of the search area are stored in the target block. Therefore, the matching process related to the target block 701 is performed before the detection image after the neighboring block 709e is stored. You can start. On the other hand, in case B as shown in FIG. 9A, since the template image and the image of the search area also include the outside of the target block, the matching process related to the target block 701 corresponds to at least the detection image of the peripheral block 709h. If the storage of the pixel to be completed is not completed, it cannot be started.

  Note that, as described above, the setting unit 205 of the present embodiment determines whether or not matching processing can be integrated based on the search area, that is, whether or not there is a block having feature points close to each other. Considering whether or not the determination can be performed, the determination of this step is performed. However, the target image from which the image in the search area is read has already been input to the image memory 204. For this reason, whether or not the matching process can be simply started is determined based on whether or not the calculated coordinates of the read completion position of the template image indicate a block in which no image is stored in the image memory 204. It may be performed based on how.

  As a result of such determination, the setting unit 205 moves the process to S606 when it is determined that the matching process related to the target block 701 can be started, and moves the process to S603 when it is determined that the matching process cannot be started.

  In step S603, the setting unit 205 waits for processing until matching processing can be started. The determination of whether the matching process can be started may be the same as that performed in S602. For example, in case B as illustrated in FIG. 9, the setting unit 205 waits for processing related to the target block 701 until the detection image storage and feature point information generation related to the peripheral block 709 h are completed. That is, the standby in this step is performed until the storage of the detection image in the image memory 204 and the generation of the feature point information are completed for the peripheral blocks determined to include the search region pixels related to the target block.

In step S <b> 604, the setting unit 205 determines whether there is a feature point of a neighboring block that is close to the feature point of the target block. As described above, whether or not the feature points are close to each other is determined based on whether or not the distance D between the feature points is less than a predetermined threshold value. When the absolute coordinates of the feature points of the target block are (GPX, GPY) and the absolute coordinates of the feature points of the surrounding blocks are (GPX ′, GPY ′)
It is possible to calculate by Note that the peripheral block for which the distance between feature points is to be calculated is a peripheral block having pixels included in the search area, preferably included in the search area, and the distance between feature points has not yet been calculated. It may be limited to no surrounding blocks. Note that the threshold used for determining whether or not they are close may be dynamically changed according to the size of the set template image. For example, as shown in FIG. 9, the shortest distance from the feature point 703 is inevitably longer than the peripheral blocks e and g in the peripheral block 709 h that touches only at the vertex (right lower end position) of the target block 701. Accordingly, for the distance between the feature points calculated between the target block 701 and the peripheral block 709h, a threshold value for determining that the feature points are close is set larger than the value used for the peripheral blocks 709e and g. Also good. In other words, the threshold used for determining whether or not they are close may be determined according to the positional relationship between the target block and the surrounding blocks.

  If the setting unit 205 determines that there is a feature point of a neighboring block that is close to the feature point of the target block, the setting unit 205 moves the process to S605, and if it does not exist, moves the process to S606.

  In step S <b> 605, the setting unit 205 determines whether the peripheral block having the feature point determined to be close is a block in which the corresponding detection image is stored after the target block. In the setting unit 205 of the present embodiment, matching processing is integrated for a block group determined to have feature points close to each other, and the result of matching processing of the block with the slowest storage order of detection images in the image memory 204 is obtained. Substitute as the processing result of each integrated block. Therefore, in the processing of this step, the setting unit 205 is performed to determine that the block with the slowest storage order of the detection image is the target block. When the setting unit 205 determines that the neighboring block having the feature point determined to be close is not a block in which the detection image is stored after the target block, the setting unit 205 moves the process to S606 and determines that the block is the other block. If so, the process moves to S606.

  In step S <b> 606, the setting unit 205 sets the memory address related to the memory access in order to extract the template image calculated for the target block and the rectangular image related to the search range from the corresponding detection image stored in the image memory 204. Issue. That is, in this step, the setting unit 205 reads out a rectangular area of the horizontal size TX and the vertical size TY as a template image from the coordinates (GTX, GTY) of the detection image related to the reference image stored in the image memory 204. Issue a memory address. Further, the setting unit 205 reads out a rectangular area having a horizontal size SX and a vertical size SY as a search area image from the coordinates (GMX, GMY) of the detection image related to the target image stored in the image memory 204. Issue an address. Then, the setting unit 205 transmits the issued memory address to the memory controller of the image memory 204, reads the corresponding image, and causes the matching processing unit 206 to output it.

  The setting unit 205 also outputs information for identifying the target block that is the processing target of the matching processing to the matching processing unit 206. Note that for a block group that has been determined that feature points are close to each other, the setting unit 205 outputs not only information for identifying the target block but also information for identifying a block in which matching processing is integrated. Thereby, in the calculation of the motion vector for the entire image based on the calculation result of the motion vector for each block, the matching processing unit 206 is informed of how much the motion vector calculated for the block group in which the matching processing is integrated should be weighted. It can be tightened.

  In step S607, the setting unit 205 determines whether all blocks defined for the reference image have been selected as target blocks. When the setting unit 205 determines that there is a block that has not yet been selected as the target block, the block for which the corresponding detection image is stored or the feature amount information is generated is added as a new target after the current target block. The block is selected, and the process returns to S601. If the setting unit 205 determines that all blocks have been selected as target blocks, the setting unit 205 completes the setting process.

  As described above, according to the motion vector detection unit 105 of this embodiment, it is possible to reduce the occurrence of redundant processing when an image is separated into a plurality of divided regions and a motion vector is detected. Specifically, the setting unit 205 according to the present embodiment identifies a group of blocks with close feature points that can be similar in matching processing results for the blocks, and integrates matching processing performed on the block groups. As a result, the number of blocks for which matching processing is executed can be reduced to reduce the amount of calculation, and further, memory access related to reading of overlapping detection images can be reduced, so that the occurrence of redundant processing can be suppressed. it can.

  In the present embodiment, the template image and the search area are set based on the feature point of the block having the slowest storage of the detection image in the image memory 204 in the block group integrating the matching processing. The implementation of the present invention is not limited to this. That is, as a result of the matching process for each block in the block group, in a mode in which the result of the matching process for one block is substituted (weighted), the template image and the search area are set based on the feature point of any block. May be.

  Further, in the setting process of the present embodiment, it has been described that the process waits until the matching process related to the target block can be started in S603. However, it is not necessary to wait for the storage of the detection image of the peripheral block in the image memory 204 as long as the feature amount information that can calculate the distance between the feature points at least with the peripheral block can be obtained. . If it is such an aspect, when the block with the slowest storage order of the detection image is set as the target block as in the setting process of the present embodiment, the integrated matching process is configured to be executed. It is possible to reduce the time required to wait due to the absence of the image for use.

  In the present embodiment, the condition for integrating the matching processing is that the distance between the feature points between the divided regions (between blocks) is not a predetermined distance. However, the present invention is not limited to this. Absent. For example, if it is estimated that similar matching processing results are obtained across multiple blocks, such as a subject that is unlikely to cause image distortion due to camera shake such as a distant view, the matching processing should be integrated for these blocks. You may comprise. Alternatively, in order to determine whether adjacent feature points indicate a specific texture, for example, not only the feature points are close but also the close feature points indicate the same level of feature amount This may be a condition.

[Modification]
In the above-described embodiment, the configuration in which the template image and the search region are determined based on the feature point determined for one of the blocks when it is determined that the matching processing is integrated for a plurality of blocks has been described. In other words, the configuration has been described in which the result of the matching process performed on one block is substituted as the result of the matching process related to another block to be integrated. On the other hand, the configuration that substitutes the result of the matching processing related to any of the blocks in this way can obtain a result with suitable accuracy because the amount of template image overlap between blocks increases or decreases depending on the arrangement of feature points. Sometimes it is not possible. In this modification, a mode in which the accuracy of the motion vector calculated for the entire image is more preferable will be described.

《Setting process》
Hereinafter, the setting process of this modification will be described in detail with reference to FIG. In the setting process of this modification, the same reference numerals are assigned to the steps for performing the same process as the setting process of the above-described embodiment, and the description thereof will be omitted. Hereinafter, a characteristic process according to this modification will be performed. The steps will be mainly described.

If it is determined in S605 that the neighboring block having the adjacent feature point is not a block in which the detection image is stored after the target block, the setting unit 205 performs template processing for matching processing to be performed in S1001. And set the search area. First, the setting unit 205 determines one point (reference point) that serves as a reference for setting a template image and a search region, based on the coordinates of the feature points of the blocks for which matching processing is integrated. In the present embodiment, when a template image is set around each feature point, the reference point is determined at the barycentric position of the feature points of the block group so that more overlapping pixels are included. That is, if the absolute coordinates of the feature points determined to be close are (GPXn, GPYn) (n = 2 to 4), the absolute coordinates (NPX, NPY) of the reference point to be obtained are
Can be calculated. Then, the setting unit 205 calculates the template image centered on the reference point and the coordinate range of the search area based on the obtained absolute coordinates of the reference point, and moves the process to S606. In this way, for example, when the feature point 703 and the feature points 901e and h are close as shown in FIG. 9, the reference point 1101 as shown in FIG. 11 can be determined. By using the reference point 1101 as the center, it is estimated that there is a feature based on three adjacent feature points, and a template image including an image region that can be overlapped when matching processing is performed for each block. 1102 can be used for the matching process. Similarly, the search area 1103 can be set around the reference point 1101.

  According to the above-described method, the calculation for calculating the coordinates of the reference point and the calculation of the new template image and the coordinate range of the search area increase, but the template that is set by biasing the matching process to be integrated to a specific feature point It can be expected to improve the accuracy of motion vectors rather than using images. In this modification, the method of determining the reference point at the center of gravity of the feature point has been described. However, the accuracy of the motion vector can be improved by other methods. For example, a position obtained by weighted averaging a plurality of adjacent feature points based on the feature values may be used as the reference point. In this case, a region that can be estimated to be more characteristic can be included in the template image. Further, for example, the reference points may be set so that all of the adjacent feature points are included in the template image region (in the reference region). At this time, for example, the size of the template image or the template image and the search area may be changed.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  105: motion vector detection unit, 201: block transfer unit, 202: image conversion unit, 203: feature analysis unit, 204: image memory, 205: setting unit, 206: matching processing unit, 501: filter unit, 502: feature quantity Calculation unit, 503: feature point determination unit

Claims (14)

An image processing apparatus for determining a search area for performing a correlation operation in the second image for a matching process for searching for a highly correlated area in the second image based on the first image,
Determining means for determining a feature point for each divided region for the first image in which a plurality of divided regions are defined;
Setting means for setting the search region in the second image based on feature points determined for the divided region for each of the plurality of divided regions;
The setting means is configured to integrate the matching processing related to the at least two divided areas when a feature point satisfies a predetermined condition between at least two divided areas of the plurality of divided areas. An image processing apparatus, wherein one search area is set for a divided area.   The said predetermined condition is satisfy | filled when the feature point determined by each of the said at least 2 division area exists in the said 1st image without separating predetermined distance. The image processing apparatus described.   The image processing apparatus according to claim 2, wherein the predetermined distance is determined according to a positional relationship between the at least two divided regions.   If the predetermined condition is satisfied, the setting means further uses the first image used as a template image for the matching process to be integrated on the basis of one feature point determined for one of the at least two divided regions. The image processing apparatus according to claim 1, wherein a reference area is set.   When the predetermined condition is satisfied, the setting unit further uses the matching unit as a template image for the matching process based on one point determined based on a plurality of feature points determined for each of the at least two divided regions. The image processing apparatus according to claim 1, wherein a reference region in the first image is set.   The one point may be the centroid position of the plurality of feature points, the weighted average position of the plurality of feature points based on the feature amount, or the reference region having a predetermined size as the center. The image processing apparatus according to claim 5, wherein the feature point is determined at a position included in the reference region.   The image processing apparatus further comprises detection means for performing a matching process on each of the plurality of divided regions of the first image and detecting a motion vector of the entire image between the first image and the second image. The image processing apparatus according to any one of claims 1 to 6.   The detection means detects a motion vector of the entire image based on a motion vector of each feature point obtained by a matching process performed for each of the plurality of divided regions, and satisfies the predetermined condition 8. The image processing apparatus according to claim 7, wherein the at least two divided areas are weighted according to the number of divided areas obtained by integrating matching processing into the obtained motion vector. A predetermined conversion process related to a matching process is applied to the plurality of divided regions of the first image, and a storage order stored in a storage unit is determined,
For the at least two divided regions satisfying the predetermined condition, the detection means sets the one search region as the one search region when a divided region having the latest storage order among the at least two divided regions is to be processed. The image processing apparatus according to claim 7, wherein the matching process is performed.   The image processing apparatus according to claim 1, wherein the determining unit determines one feature point for each of the plurality of divided regions.   The image processing apparatus according to claim 10, wherein the determining unit determines a pixel having the highest feature amount in one divided region as the one feature point.   The image processing apparatus according to claim 1, wherein the first image and the second image are images related to frames that are continuously captured. An image processing method for determining, in the second image, a search area for performing a correlation operation for a matching process for searching for a highly correlated area in the second image based on the first image,
A determination step of determining a feature point for each divided region for the first image in which a plurality of divided regions are defined;
For each of the plurality of divided areas, a setting step for setting the search area in the second image based on the feature points determined for the divided areas,
In the setting step, when a feature point satisfies a predetermined condition between at least two divided regions of the plurality of divided regions, the at least two matching regions are integrated so that the matching process is integrated. An image processing method, wherein one search area is set for a divided area.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 12.