JP4877243B2 - Corresponding point search apparatus and method - Google Patents

Description

  The present invention relates to a corresponding point search apparatus and a corresponding point search method for searching for points corresponding to each other between two images.

  Conventionally, there is known a corresponding point search method for searching for a corresponding point of the reference image corresponding to a point of interest in the reference image when one of the two images is a reference image and the other is a reference image. It has been. This corresponding point search method is also called image matching, and is important as a basic technique in various fields such as image analysis, recognition and understanding by a computer.

  For example, this corresponding point search method uses a set of stereo images in three-dimensional vision of a robot in FA (Factory Automation) and recognition of a preceding object in a moving body such as a vehicle and measurement of a distance to the preceding object. It is used for corresponding point search processing using. Further, for example, the corresponding point search method is used for motion vector search processing using a set of moving images in tracking of a moving object, tracking of a face part (tracking), and the like in a security system. Also, for example, it is used for comparison processing between a registered image and a reference image in face authentication, fingerprint authentication, image search, and the like in biometric authentication.

  This corresponding point search method has been researched and developed, and various methods have been proposed. For example, as shown in FIG. 17, the size (area of the template TP) serving as the calculation area for the corresponding point search in the reference image I1. Depending on the size, the template TP may be set in the vicinity of the edge (end) of the reference image I1, and a non-image area (outside image area) outside the image area is included in the template TP. There is a risk of mishandling. That is, when the corresponding point search is performed for the reference image I2 shifted by N pixels in the horizontal direction with respect to the standard image I1, the corresponding points of the reference image I2 corresponding to the attention point A of the standard image I1 are as shown in FIG. Originally, a point A ′ that is a position shifted by N pixels should be obtained. However, when an outside image area is included in the template TP set with respect to the target point A of the reference image I1, an edge that does not exist as an image is caused by the difference between the brightness value of the image area and the brightness value of the outside image area. This occurs at the boundary between the region and the region outside the image. Therefore, the corresponding point search is performed in a state including this edge. As a result, due to the influence of the edge, that is, the influence of the discontinuity of the image in the image area and the non-image area, the corresponding point of the reference image I2 corresponding to the attention point A of the reference image I1 becomes the reference image in the reference image I2. There is a case where the similarity (correlation) becomes high at the point A ″ at the same position as I1, which causes an erroneous correspondence.

  Further, when the two images for which the corresponding point search is performed are time-series images, the following incorrect correspondence may also occur. When the image one frame before is the reference image I1 and the current frame image is the reference image I2, for example, as shown in FIG. 18, when the moving body is moving straight forward, the correspondence in the reference image I2 The window WD serving as a point search calculation area may be set near the edge (end) of the reference image I2, and an out-of-image area is included in the window WD, which may cause a correspondence error. That is, when the attention point A of the standard image I1 moves to the vicinity of the edge (end) of the reference image I2 by the straight forward movement of the moving body, as shown in FIG. 18, it corresponds to the attention point A of the standard image I1. The corresponding point of the reference image I2 to be obtained should be a point A ′ that is originally a position corresponding to the moving speed of the straight forward movement. However, if an outside image area is included in the window WD set in the reference image I2, an edge that is not originally an image has an image area and an outside image area due to the difference between the luminance value of the image area and the luminance value of the outside image area. Will occur at the boundary. Therefore, the corresponding point search is performed in a state including this edge. As a result, due to the influence of the edge, that is, the influence of the discontinuity of the image in the image area and the non-image area, the corresponding point of the reference image I2 corresponding to the attention point A of the reference image I1 becomes the reference image in the reference image I2. There is a case where the similarity (correlation) becomes high at the point A ″ at the same position as I1, which causes an erroneous correspondence.

  As a case similar to such a case, there is a case where the target object projected in the standard image is not projected in the reference image. In such a case, if this target object is a feature point, the corresponding point is searched. It becomes difficult. Therefore, for example, in the technique disclosed in Patent Document 1, when evaluating the correlation between the correlation source image area and the correlation destination candidate image area, it is determined whether or not occlusion has occurred, and the occlusion occurs. In the area where the occurrence occurs, the weighting coefficient set in association with the pixels in the image area is made smaller than that in the non-occlusion area. As a result, the technique disclosed in Patent Document 1 causes an occlusion in which a certain object is hidden behind another object and the object projected on one image is not projected on the other image. Even in the case of a captured image, the certainty of matching is achieved. In Patent Document 1, the 4 × 4 pixel region is widened to the periphery to make the 8 × 8 pixel region the calculation target block. However, in the [0026] paragraph, “at the edges of the four sides of the captured image, the calculation target A situation may occur in which the peripheral area of the block protrudes from the effective pixel area, in which case the calculation target block is defined using pixels outside the effective pixel area, or the pixel block of 4 × 4 pixels is correlated. Is evaluated. " Thus, in patent document 1, it sets so that the area outside an image may not be included in a template. For this reason, the technique disclosed in Patent Document 1 does not take into account the erroneous correspondence due to the influence of the edge that occurs between the image region and the non-image region, and cannot deal with such erroneous correspondence.

  In addition, as a corresponding point search method in such a case, when a template is set around an attention point and this template includes a region outside the image, a corresponding point search is performed for other templates that do not include the outside of the image region. There is a method for obtaining a corresponding point of this attention point from the result of performing the above. However, with this method, since a corresponding point of this attention point is not actually obtained when a template is set around the attention point, there is a risk of erroneous correspondence.

As a corresponding point search method in such a case, the image area is folded back to the image outside area at the boundary between the image area and the image outside area, and mirror processing is performed to fill the image outside area with the data of the image area. There is a technique. This technique eliminates the discontinuity between the image area and the non-image area. However, in this method, since the pattern is naturally reversed in the area outside the image that is filled with the data of the image area by mirror processing, an opposite phase appears. For this reason, as a result of corresponding points appearing on the opposite side, there is a risk of erroneous correspondence.
JP 2004-102323 A

  The present invention has been made in view of the above-described circumstances, and the object thereof is to reduce miscorrespondence even when the calculation area of the corresponding point search is set over the image area and the non-image area. It is to provide a corresponding point search device and a corresponding point search method.

As a result of various studies, the present inventor has found that the above object is achieved by the present invention described below. In the corresponding point searching apparatus and the corresponding point searching method according to the present invention, a predicted image is generated, a synthesized image is generated based on the predicted image, and a corresponding point search is performed by using the synthesized image. That is, in one aspect according to the present invention, in a corresponding point search device for searching for points corresponding to each other between two images, one of the two images is a reference image and the other is a reference image. A corresponding point search unit that searches for a corresponding point of the reference image corresponding to a point of interest of the reference image, and an image at a prediction time point different from the predetermined time point for the reference image or the reference image acquired at a predetermined time point An image prediction unit for generating a prediction image, and generating a combined image at the prediction time by replacing a region corresponding to the image acquired at the prediction time in the prediction image with an image acquired at the prediction time And the corresponding point search unit uses the synthesized image synthesized by the image synthesizing unit as one of the base image and the reference image. And wherein the Rukoto. In another aspect of the present invention, in a corresponding point search method for searching for points corresponding to each other between two images, one of the two images is used as a reference image and the other is a reference image. And a corresponding point search step of searching for a corresponding point of the reference image corresponding to the target point of the reference image, and a prediction time point different from the predetermined time point for the reference image or the reference image acquired at a predetermined time point an image prediction step of generating an image as a prediction image in the area corresponding to the image acquired in said predicted time in the prediction image, the synthesis image after the predicted time by replacing the obtained image to the predicted time And the corresponding point search step includes a step of combining the synthesized image in one of the base image and the reference image in the image synthesis step. Characterized by using the post-image.

  In the corresponding point search device and the corresponding point search method configured as described above, an image at a prediction time point different from the predetermined time point is generated as a prediction image based on an image acquired at a predetermined time point, and the prediction image and the prediction A composite image is generated by combining the image obtained at the time. When one of the two images is a standard image and the other is a reference image, the corresponding point of the reference image corresponding to the target point of the standard image is searched for. If there is a composite standard image of the composite image to be used, the composite standard image is used instead of the standard image, and if there is a composite reference image of the composite image corresponding to the reference image, the composite reference image is used instead of the reference image. Used. Therefore, when the corresponding point search is performed, it is possible to generate a composite image using the predicted image so that the calculation region does not include the non-image region. For this reason, even when the calculation area of the corresponding point search is set over the image area and the non-image area, it is possible to reduce erroneous correspondence.

  In the above-described corresponding point search device, the image prediction unit uses affine transformation for generation of the predicted image.

  In the corresponding point search apparatus having such a configuration, affine transformation is used for generating a predicted image, and accordingly, enlargement / reduction (scaling), rotational movement, and parallel movement are appropriately performed on an image acquired at a predetermined time point. It is possible to generate an optimal prediction image at a prediction time point different from the predetermined time point.

  In the corresponding point search apparatus, the affine coefficient of the affine transformation is based on an optical flow obtained based on an image acquired at the predetermined time point and an image acquired at a time point different from the predetermined time point. It is characterized by being required.

  In the corresponding point search device having such a configuration, the affine coefficient of the affine transformation is obtained based on the optical flow obtained based on the two images having different acquisition points, so that only the image is used without using another external device. Thus, the affine coefficient of the affine transformation can be obtained.

  Further, in the corresponding point search device described above, the image is acquired by a photographing camera attached to a moving body, and the affine coefficient of the affine transformation relates to steering information relating to a rotation angle by steering of the moving body and a self-running speed. It is calculated | required based on the optical flow calculated | required based on speed information. Preferably, the steering information is acquired by a steering angle sensor that detects a rotation angle by steering of the moving body attached to the moving body, and the speed information is acquired by the moving body attached to the moving body. It is acquired by a speed sensor that detects the speed of.

  In the corresponding point search device having such a configuration, since the steering information and the speed information are acquired from other external devices such as a steering angle sensor and a speed sensor, the calculation time can be shortened and the arithmetic processing device can be simplified. In addition, the affine coefficient of the affine transformation can be obtained with high accuracy.

In the corresponding point search apparatus described above, the two images are stereo images including a set of images taken at the prediction time point by a stereo camera.

  According to this configuration, it is possible to provide a stereo image corresponding point search apparatus that can perform a corresponding point search to the edge (end) of the image region with high accuracy.

  In the corresponding point search apparatus described above, the image composition unit generates each composite image for each of a set of images in the stereo image.

  In the corresponding point search device having such a configuration, a composite image is generated for both images of a stereo image, and therefore it is possible to perform corresponding point search more accurately up to the edge (end) of the image region.

  In the corresponding point search apparatus described above, the two images are images acquired at different points in time of the time-series images.

  According to this configuration, it is possible to provide a corresponding point search device for time-series images that can perform a corresponding point search with high accuracy up to the edge (end) of the image region.

  Further, in the corresponding point search device described above, the image synthesis unit determines the predicted image and an image acquired at the predicted time according to an optical flow obtained based on the time-series image. To do.

  In the corresponding point search device having such a configuration, the predicted image and the image acquired at the predicted time point are determined according to the optical flow obtained based on the time-series image. Therefore, the synthesis target of the synthesized image can be changed as appropriate, and the corresponding point search can be performed effectively.

Further, in the corresponding point search device described above, the image prediction unit, than the first time point to generate a prediction image of the first point on the basis of the image acquired in the past time, the image synthesizing unit, the When the optical flow obtained based on the time-series image extends radially from the vanishing point , combining the predicted image at the first time point with the image at the first time point acquired at the first time point Features.

  In the corresponding point search device having such a configuration, a predicted image at the first time point is generated based on an image acquired at a time point earlier than the first time point, and is acquired at the predicted image at the first time point and the first time point. The first time point image is synthesized. For this reason, when the image on which the corresponding point search is performed is an image obtained by the photographing camera attached to the moving body, the corresponding point search can be performed more accurately when the moving body is moving forward. It becomes possible.

Further, in the corresponding point search device described above, the image prediction unit, based on said acquired image to the first point than the first time point to generate a predicted image at a past time, the image synthesizing unit, the When the optical flow obtained based on the time series image extends toward the vanishing point, the predicted image at the past time point and the image at the past time point acquired at the past time point are combined. Features.

  In the corresponding point search device having such a configuration, a predicted image at a time point earlier than the first time point is generated based on the image acquired at the first time point, and the past predicted image and the past acquired at the past time point are generated. And the image at the time of. For this reason, when the image on which the corresponding point search is performed is an image obtained by a photographing camera attached to the moving body, the corresponding point search is performed more accurately when the moving body is moving backward (backward). Can be done.

  In the above-described corresponding point search device, the corresponding point search unit is performed by a correlation calculation between the standard image and the reference image.

  According to this configuration, there is provided a corresponding point search device in which corresponding point search is performed by correlation calculation between the standard image and the reference image.

  In the above-described corresponding point search device, the corresponding point search unit calculates a corresponding position based on the similarity of signals in which the pattern in the calculation region is frequency-resolved and the amplitude component is suppressed.

  In the corresponding point search apparatus having such a configuration, the pattern in the calculation area is frequency-resolved, and the corresponding position is calculated based on the similarity of the signals with the amplitude component suppressed. For this reason, it is possible to perform a robust corresponding point search that is not easily affected by a luminance difference or noise between two images.

  In the corresponding point search apparatus described above, the frequency decomposition method is any one of fast Fourier transform, discrete Fourier transform, discrete cosine transform, discrete sine transform, wavelet transform, and Hadamard transform. To do.

  In the corresponding point search apparatus having such a configuration, fast Fourier transform (FFT), discrete Fourier transform (DFT), discrete cosine transform (DCT), discrete sine transform (DST), wavelet transform, and Hadamard are used as frequency resolution techniques. Any of the transformations are used. Therefore, since an already established method is used, it is possible to reliably perform frequency decomposition.

  In the corresponding point search apparatus described above, the calculation processing of the corresponding position in the corresponding point search unit is a phase-only correlation method.

  In the corresponding point search apparatus having such a configuration, the phase-only correlation method (POC) is used for the calculation processing of the corresponding position. For this reason, it is possible to search for corresponding points with higher accuracy.

  In the corresponding point search apparatus described above, the corresponding point search unit uses a multi-resolution strategy.

  In the corresponding point search using the multiplexed resolution strategy, there are many cases where the calculation area includes an area outside the image. Therefore, according to this configuration, it is possible to provide a multi-resolution strategy corresponding point search device capable of performing corresponding point search with higher accuracy.

  With the corresponding point search device and the corresponding point search method according to the present invention, it is possible to reduce miscorrespondence even when the calculation area of the corresponding point search is set over the image area and the non-image area.

DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, an embodiment of the invention will be described with reference to the drawings. In addition, the structure which attached | subjected the same code | symbol in each figure shows that it is the same structure, The description is abbreviate | omitted.
(First embodiment)
FIG. 1 is a diagram showing a schematic configuration of a corresponding point search system according to the first embodiment. FIG. 2 is a block diagram illustrating a configuration of a corresponding point search apparatus in the corresponding point search system according to the embodiment.

  The corresponding point search system S is an apparatus for capturing two images and searching for corresponding points between the two images. For example, as shown in FIG. And the corresponding point search device 1. In the example shown in FIG. 1, the corresponding point search system S is a stereo camera composed of a pair (two) of photographing cameras 2-1 and 2-2 to obtain a stereo image composed of a pair (one pair) of images. It has.

  In the present specification, when referring generically, it is indicated by a reference symbol without a suffix, and when referring to an individual configuration, it is indicated by a reference symbol with a suffix.

  The photographing camera 2 is a device that acquires an image for searching for corresponding points. The corresponding point search method is applied to apparatuses for various uses as described above in the background art, and the photographing camera 2 is appropriately arranged according to the use of the apparatus. For example, as shown in FIG. 1, when the photographing camera 2 is a stereo camera mounted on a moving body such as a vehicle or a robot in order to recognize the preceding object Ob and measure the distance, the photographing camera 2-1, 2-2 is disposed on the moving body so as to be separated by a predetermined baseline interval and so that their optical axes are parallel to each other. As described above, the photographing cameras 2-1 and 2-2 are arranged so that the preceding object Ob can be photographed from different positions. In the example shown in FIG. 1, automobiles Ob-1 and Ob-2 are shown in FIG. 1 as the preceding object Ob that is a subject.

  When the imaging camera 2 captures the preceding object Ob and acquires an image, the imaging camera 2 transmits the captured image (image data) to the corresponding point search device 1 via the transmission path. The communication method may be a wired method or a wireless method. As shown in FIG. 1, when the photographing camera 2 is a stereo camera, the photographing cameras 2-1 and 2-2 are controlled by the corresponding point search device 1 or the photographing cameras 2-1 and 2-2. With the control in between, the preceding object Ob is photographed at the same timing (synchronously), and a pair of left and right (a pair of left and right) images (stereo images) photographed at the same timing is output to the corresponding point search device 1. One of the left and right set of images is used as a reference image for searching for corresponding points, and the other is used as the reference image. For example, an image photographed by the photographing camera 2-1 is used as a reference image, and an image photographed by the photographing camera 2-2 is used as a reference image.

  Note that the number of the photographing cameras 2 may be three or more, or may be one. When there are three or more photographic cameras 2, an image photographed by one of the photographic cameras 2 is used as a reference image, and an image photographed by another photographic camera 2 is used as a reference image. In addition, when there is one photographing camera 2, an image photographed at a predetermined timing (time) among time-series images photographed by the photographing camera 2 is used as a reference image, and at a timing different from the predetermined timing. The taken image is used as a reference image. For example, an image taken at a timing after a predetermined time has elapsed from the predetermined timing is set as a reference image. Further, for example, an image taken at a timing before a predetermined time has elapsed from the predetermined timing is set as a reference image.

  The corresponding point searching device 1 is a device that searches for corresponding points between two images acquired by the photographing camera 2. For example, in the present embodiment, the corresponding point search apparatus 1 is realized by an information processing apparatus such as a computer by installing a corresponding point search program according to an embodiment of the present invention, as shown in FIG. The corresponding point search program is created by programming the corresponding point searching method according to the embodiment of the present invention. The corresponding point search device 1 may be a dedicated device manufactured for the corresponding point search system S having various functions to be described later. As illustrated in FIG. 2, the corresponding point search device 1 includes, for example, a central processing unit (CPU) 12, a storage unit 13, a display unit 15 such as a liquid crystal display or an organic EL display, and a startup, for example. An input unit 16 for inputting instructions and data and a communication unit 17 such as an interface card are configured.

  The storage unit 13 includes a plurality of storage media such as a hard disk drive (HDD) 13a and a semiconductor memory 13b such as a RAM (Random Access Memory) and a ROM (Read Only Memory). Further, as indicated by a broken line, the corresponding point search device 1 may include a media drive 14 as necessary. The media drive 14 stores information (data) recorded in a portable recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD (Digital Versatile Disk), a flexible disk, and a memory card. Can be read. The information supplied to the corresponding point search unit 1 is not limited to the case of being supplied via the recording medium, but via a network such as a local area network (LAN) or the Internet. It may be supplied. Therefore, in the above description, an image photographed by the photographing camera 2 connected to the corresponding point searching device 1 via a transmission path (including a network) is supplied to the corresponding point searching device 1, but was photographed by the photographing camera. An image may be stored in a recording medium or the like, and the image may be supplied from the recording medium to the corresponding point search device 1.

  In the example shown in FIGS. 1 and 2, the photographing camera 2 is connected to the communication unit 17 so as to be communicable, and an image (image data) photographed by the photographing camera 2 is input to the CPU 12 via the communication unit 17. Is done.

  The CPU 12 controls the entire corresponding point search apparatus 10 by controlling the storage unit 13, the display unit 15, the input unit 16, and the communication unit 17 according to the function. As shown in FIG. 2, the CPU 12 is functionally configured to include an image prediction unit 21, an image synthesis unit 22, and a corresponding point search unit 23.

  The image predicting unit 21 generates an image at a prediction time point different from the predetermined time point as a predicted image based on an image acquired at the predetermined time point. In the present embodiment, the data acquisition unit 211, the coefficient calculation unit 212, and the predicted image generation unit 213 are configured. The data acquisition unit 211 acquires data used by the coefficient calculation unit 212 from an image captured by the imaging camera 2. The coefficient calculation unit 212 calculates an affine coefficient for affine transformation using the data acquired by the data acquisition unit 211. The prediction image generation unit 213 predicts an image at a prediction time point different from the predetermined time point based on an image acquired at a predetermined time point by using an affine transformation having an affine coefficient calculated by the coefficient calculation unit 212, It is generated as a predicted image.

  The image synthesizing unit 22 generates a synthesized image by synthesizing the predicted image generated by the image predicting unit 21 and the image acquired at the prediction time point. More specifically, in the present embodiment, the image compositing unit 22 is configured to display an image at the first time point captured by the camera 2-1 at the first time point (for example, the current time point) and a time point earlier than the first time point. The predicted image at the first time point predicted by the image prediction unit 21 is synthesized based on the image at the past time point captured by the photographing camera 2-1. Then, the image composition unit 22 captures the image at the first time point captured by the photographing camera 2-2 at the first time point (for example, the current time point) and the image capturing camera 2-2 at a time point before the first time point. Based on the image at the past time point, the predicted image at the first time point predicted by the image prediction unit 21 is synthesized. As described above, in the present embodiment, the image composition unit 22 generates each composite image for each of a set of images in a stereo image photographed by each of the photographing cameras 2-1 and 2-2.

  In the present embodiment, the image synthesizing unit 22 synthesizes the first time point image and the first time point predicted image in this manner. Therefore, more specifically, the predicted image generation unit 213 predicts an image at the first time point by using affine transformation on an image acquired at a time point earlier than the first time point, and the predicted image at the first time point. Is generated. Therefore, more specifically, the data acquisition unit 211 acquires data used by the coefficient calculation unit 212 from images captured by the imaging camera 2 at a time point earlier than the first time point.

  The corresponding point search unit 23 selects the corresponding point of the reference image corresponding to the attention point of the standard image when one of the two images captured by the imaging camera 2 is the standard image and the other is the reference image. To explore. Then, in the corresponding point search, if there is a combined reference image generated by the image predicting unit 21 and the image combining unit 22 corresponding to the reference image in the corresponding point search, the corresponding point searching unit 23 replaces the reference image with the combined reference image. When the reference image is used and there is a combined reference image generated by the image predicting unit 21 and the image combining unit 22 corresponding to the reference image, the combined reference image is used instead of the reference image.

  Next, the operation of this embodiment will be described. FIG. 3 is a flowchart showing the operation of the corresponding point search device in the corresponding point search system of the first embodiment.

  When the corresponding point search system S is activated and its operation is started, the preceding objects Ob are sequentially photographed at predetermined time intervals by the photographing cameras 2-1 and 2-2 at the same timing (synchronously). The Therefore, a time-series image (time-series image data) of the preceding object Ob sequentially captured at a predetermined frame period by the imaging camera 2-1 is acquired, and sequentially at a predetermined frame period by the imaging camera 2-2. A time-series image (time-series image data) of the captured preceding object Ob is acquired. Each captured image (each image data) is sequentially input from the communication unit 17 to the corresponding point search device 1, notified to the CPU 12 of the corresponding point search device 1, and the CPU 12 corresponds to the corresponding point between the images. The search process is started.

In the corresponding point search process of the stereo image, first, as shown in FIG. 3, in step S11, the data acquisition unit 211 of the image prediction unit 21 uses each image captured by the imaging cameras 2-1 and 2-2. Data used in the coefficient calculation unit 212 is acquired. Since the predicted image generation unit 213 generates a predicted image at the first time point by applying affine transformation to an image acquired at a time point earlier than the first time point, the coefficient calculation unit 212 has an affine coefficient of this affine transformation. Need to be calculated. Therefore, more specifically, the data acquisition unit 211 searches for at least three corresponding points between two images acquired at a time point earlier than the first time point. That is, when the image captured by the imaging camera 2-1 at time t is I ¹ _t and the image captured by the imaging camera 2-2 at time t is I ² _t , the data acquisition unit 211 captures the image. the image ^{I 1} of the camera 2-1, for example, current image ^I ₁ ^t-1 of the (in the current frame time) t0 than one frame before the time t-1 and two frames before the time ^t-2, _{I 1 t-} performing corresponding point search between ₂ and the corresponding point searching results oF ¹ comprising at least three corresponding points corresponding point searching results oF 1 ^{t ₍₂₁₎ (at} least three of the target point corresponding to at least three corresponding points _{t (21))} to acquire the (S11-1), the image ^{I 2} of the photographic camera 2-2, for example, current time (in the current frame time) t0 than one frame before the time t- And performing corresponding point search in two frames before the image ^I ₂ ^t-1 time t-2 ^of, _{I 2 t-2} between the corresponding point searching results ^OF _{2 t (21)} comprising at least three corresponding points (at least Corresponding point search result OF ² _{t (21)} ) including at least three corresponding points respectively corresponding to the three attention points is acquired (S11-2). ^{Incidentally,} _{OF 1 t (21)} is located at the corresponding point search result between captured frame preceding image ^I _{1 t-1} and the two frames before the image ^I _{1 t-2} by the photographing camera 2-1 , OF ² _{t (21)} is a corresponding point search result between the image I ² _t-1 two frames before and the image I ² _t-2 two frames before captured by the camera 2-2.

Here, one of the image I _{t-1 one} frame before and the image I _t-2 two frames before, for example, the image I _t-2 two frames before is set as the reference image I1, and one frame before When the image It _-1 is used as the reference image I2, the corresponding point search for the corresponding point of the reference image I2 corresponding to the target point of the standard image I1 is executed as follows. First, in the standard image I1, a template TP having a size of P pixels × Q pixels is set in the vertical and horizontal directions around the target point, and similarly, a window WD having the same size is also set in the reference image I2. Is done. In this case, the window WD starts from the same position as the template TP in the reference image I1 in the reference image I2, and is in a certain range (0 <p <) in a predetermined direction (for example, the baseline length direction (epipolar line direction in a stereo image)). pmax)), the correlation calculation is performed at each position while changing the position. For the correlation calculation, for example, SAD, SSD, NCC, and the like can be used. In the conversion from the correlation calculation result to the similarity, the similarity is higher as the distance between the patterns is shorter. For example, the conversion formula of (similarity) = 1 / ((result of correlation calculation) +1) is used. . In general, the template TP set in the reference image I1 may also be referred to as a window. Here, in order to perform the corresponding point search, the calculation region set in the reference image I1 is referred to as a template TP, and is referred to as a reference image. The calculation area set to I2 is called a window.

  As shown in Equation 1, SAD (Sum of Absolute Difference) is the luminance value of the pixel in the template TP of the standard image I1 and the pixel in the window WD of the reference image I2 at the position corresponding to the pixel position of the template TP. This is a technique for performing a correlation calculation by obtaining an absolute value of a difference between the luminance value. As shown in Equation 1, the SAD calculates the absolute value of the difference between the luminance values of pixels corresponding to the same position in the template TP and the window WD, and adds them over all the pixels in the window WD. Therefore, the amount of calculation is small, and therefore the corresponding point search can be performed in a short time.

Here, M _L (i, j) is the luminance value of the pixel position (i, j) in the template TP of the standard image I1, and M _R (i, j) is the pixel position in the window WD of the reference image I2. The luminance value of (i, j). P and Q represent the sizes (area sizes) of the template TP and the window WD, P is the number of vertical pixels, and Q is the number of horizontal pixels.

  Such processing is performed while shifting one pixel at a time, and it is determined that there is an image equal to the template TP at the window position with the highest similarity.

  In addition, SSD (Sum of Squared Intensity Difference), as shown in Equation 2, is a pixel intensity value in the template TP of the reference image I1 and a position corresponding to the pixel position of the template TP. This is a technique for performing a correlation calculation by obtaining the square of the difference between the luminance value of the pixel in the window WD of the reference image I2. Since the SSD squares the difference in luminance value between pixels as described above, the correlation between the template TP and the window WD is more clearly expressed even when the template TP and the window WD having a relatively small size are used. be able to.

  Further, as shown in Equation 3, NCC (Normalized Cross Correlation) is an average of luminance values from luminance values at each point in the template TP of the standard image I1 and the window WD of the reference image I2. This is a method of subtracting values and performing correlation calculation with the similarity of variance values. In NCC, the influence of a linear brightness change, that is, the influence of noise and the linear change of the luminance value and contrast of an image is reduced. The correlation value is a value in the range of −1 to +1, and the larger the value, the more similar the template TP and the window WD are. For this reason, in NCC, it is the process which calculates | requires the maximum value of Formula 3.

Here, μM _L is an average value in the template TP of the standard image I1, and μM _R is an average value in the window WD of the reference image I2.

  As a robust corresponding point search method, a correlation method in which an amplitude component is suppressed is known. In this correlation method, the template TP and the pattern in the window WD set for the standard image I1 and the reference image I2 are frequency-divided, and the similarity calculation is performed using only the phase component signal whose amplitude component is suppressed from the frequency resolution signal. It is a technique to do. For this reason, this corresponding point search method is less susceptible to luminance fluctuations and noise, and can search for corresponding points between images with high accuracy.

  As a method for calculating the frequency-resolved signal of this pattern, Fast Fourier Transform (FFT), Discrete Fourier Transform (DFT), Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Wavelet Transform, Hadamard Transform, etc. are common. Widely used in Since these methods have already been established, frequency decomposition can be reliably performed, which is preferable. Among these, the phase-only correlation method (POC) using Fourier transform for conversion performs correlation calculation of only the phase component in which the amplitude component of the Fourier series is suppressed. In the DCT code-only correlation method, discrete cosine transform is used for conversion, and correlation calculation is performed only for the code synthesis with the amplitude component of the cosine transform result suppressed.

  Details will be described below by taking the phase-only correlation method as an example. FIG. 4 is a block diagram of the phase only correlation method. In this phase-only correlation method, first, a pair of a template TP and a window WD (candidate region) is set for the base image I1 and the reference image I2, and a correlation between them is calculated. It is determined from the degree of similarity whether the pair is a correct region. In FIG. 4, an apparatus for searching for corresponding points by the phase-only correlation method normalizes Fourier transform units 31 and 32 that perform Fourier transform on an image, and amplitude components of Fourier series obtained by the Fourier transform units 31 and 32. Normalization units 33 and 34, a synthesis unit 35 that synthesizes each standard result obtained by each normalization unit 33 and 34, and an inverse Fourier that performs inverse Fourier transform on the synthesis result obtained by the synthesis unit 35 And a conversion unit 36. In the apparatus having such a configuration, the template TP of the standard image I1 and the window WD of the reference image I2 are input to the Fourier transform unit 31 and the Fourier transform unit 32, respectively, and subjected to Fourier transform. The Fourier transformed standard image I1 template TP and reference image I2 window WD are input to the normalization unit 33 and the normalization unit 34, respectively, and are normalized. The standardized template TP of the standard image I1 and the window WD of the reference image I2 are respectively input to the synthesis unit 35, synthesized, and subjected to inverse Fourier transform in the inverse Fourier transform unit 36. It is as follows when expressed in mathematical formulas.

  As described above, the phase only correlation method is a technique in which two Fourier-transformed images are normalized for each spectrum and then synthesized and inverse Fourier transformed.

  As shown in FIG. 5, the POC value obtained by this phase-only correlation method has a steep similarity peak in the coordinates of the amount of movement between images (the template TP of the standard image I1 and the window WD of the reference image I2). Is known and has high robustness in image matching. For this reason, it is possible to search for corresponding points with higher accuracy.

Returning to FIG. 3, when data is acquired, the data acquisition unit 211 notifies the coefficient calculation unit 212 of the data. Next, in step S <b> 12, the coefficient calculation unit 212 of the image prediction unit 21 calculates the affine coefficient of the affine transformation using the data acquired by the data acquisition unit 211. That is, the coefficient calculation unit 212 performs the corresponding point search result OF ¹ _{t (21} including at least three corresponding points acquired by performing the corresponding point search on the image I ¹ of the photographing camera 2-1 by the data acquisition unit 211. ₎ Is used to calculate the affine coefficient P ¹ _{A (21)} of the affine transformation (S12-1), and the image I ² of the photographing camera 2-2 is acquired by searching for corresponding points in the data acquisition unit 211. Then, the affine coefficient P ² _{A (21)} of the affine transformation is calculated using the corresponding point search result OF ² _{t (21)} including at least three corresponding points (S12-2). ^{Incidentally,} _{P 1 A (21),} the affine coefficients calculated on the basis of the previous frame image ^I _{1 t-1} and 2-frame and the previous image ^I _{1 t-2} photographed by the photographing camera 2-1 ( an affine _{^parameter), P 2 a (21)} is calculated based on the image ^I _{2 t-2} of the previous frame preceding image ^I _{2 t-1} and 2 frames taken by the photographing camera 2-2 The affine coefficient (affine parameter).

More specifically, the coefficient calculation unit 212 performs calculation processing as follows, and each affine coefficient P ¹ _{A (21)} , from each corresponding point search result OF ¹ _{t (21)} , OF ² _{t (21)} , P ² _{A (21)} is obtained respectively. In the following description, calculation processing for obtaining the affine coefficient P ¹ _{A (21)} from the corresponding point search result OF ¹ _{t (21)} will be described, but the affine coefficient P ² from the corresponding point search result OF ² _{t (21)} will be described. _The calculation process for obtaining _{A (21)} can be similarly described.

Affine transformation refers to coordinate transformation for transforming from one coordinate system to another, and scaling (rotation), rotational movement, and parallel movement are performed when moving from one coordinate system to another. . Here, the coordinates of the target point in the reference image I1 to as (x, y), the coordinates of the corresponding point of the reference image I2 corresponding to the point of interest and (X, Y), affine coefficients P _A and a, b, When c, d, e, and f are used, the affine transformation is expressed by Equation 5-1 and Equation 5-2.

  As described above, in the coordinate conversion, enlargement / reduction (scaling), rotational movement, and parallel movement are performed, and each of these conversions is expressed by Equation 6 when expressed in homogeneous coordinates. The relationship between enlargement / reduction, rotational movement, and parallel movement is expressed as Expression 7, Expression 8, and Expression 9, respectively.

Here, S _x and S _y are the magnification of x and the magnification of y in enlargement / reduction. θ is a rotation angle in the rotational movement. X ₀ and y ₀ are a movement amount in the x direction and a movement amount in the y direction in the parallel movement.

  The point [x, y] is expressed as [x, y, 1] in the homogeneous coordinates, while the point [X, Y, W] in the homogeneous coordinates is [X / W, Y / W]. W means a magnification, and W = 0 expresses an infinite point.

Accordingly, the relationship between at least three sets of attention points and corresponding points in the corresponding point search result OF ¹ _{t (21)} is substituted into Equation 6 and solved for a, b, c, d, e, and f to solve the affine coefficient P. ¹ _{A (21)} (a, b, c, d, e, f) is determined.

When there are three or more corresponding point search results, for example, the least square method is used. Corresponding to three or more attention points (x ₁ , y ₁ ), (x ₂ , y ₂ ),..., (X _n , y _n ) of the reference image I1 in the corresponding point search result OF ¹ _{t (21)} . Assuming that three or more corresponding points of the reference image I2 are (X ₁ , Y ₁ ), (X ₂ , Y ₂ ),..., (X _n , Y _n ), the affine coefficient P ¹ _{A ( 21)} a, b, c, d, e, and f of (a, b, c, d, e, f) are obtained by calculating Equations 10-1 and 10-2.

Note that the at least three sets of the target point and the corresponding point for obtaining the affine coefficients P _A, without erroneous corresponding, to obtain a stable corresponding points a trusted, corresponding points for the area not including the image area outside It is preferable to perform a search.

When the affine coefficients P ¹ _{A (21)} and P ² _{A (21)} are calculated, the coefficient calculation unit 212 calculates the affine coefficients P ¹ _{A (21)} and P ² _{A (21)} to the predicted image generation unit 213. Notice. Next, in step S13, the prediction image generation unit 213 of the image prediction unit 21 calculates an image at a prediction time point different from the predetermined time point based on the image acquired at the predetermined time point by the coefficient calculation unit 212. It predicted by using the affine transformation having coefficients P _a, and generates a prediction image. FIG. 6 is a diagram for explaining a predicted image generation method. That is, the predicted image generation unit 213 uses the coefficient calculation unit 212 to convert the image I ¹ _t-1 (FIG. 6A) acquired one frame before with respect to the image I ¹ of the photographing camera 2-1. By applying an affine transformation having the calculated affine coefficient P ¹ _{A (21)} , a predicted image I ^pre1 _t-1 (FIG. ^6B ) at the present time (current frame) is generated (S13-1), The affine transformation having the affine coefficient P ² _{A (21)} calculated by the coefficient calculation unit 212 is applied to the image I ² _t−1 acquired one frame before the image I ² of the photographing camera 2-2. By doing so, the prediction image ^Ipre2 _t-1 at the present time (current frame) is generated (S13-2).

Note that the predicted image I ^pre1 _t-1 is an area corresponding to an image area photographed by the photographing camera 2-1, as illustrated in FIG. 6C, in order to easily perform image composition by the image composition unit 22. The DM may be extracted. For example, by setting the luminance value in the region DM to 0, the region DM corresponding to the image region photographed by the photographing camera 2 is extracted. Similarly, the predicted image I ^pre2 _t-1 may be obtained by extracting an area corresponding to an image area photographed by the photographing camera 2-2 in order to easily perform image composition by the image composition unit 22. good.

When a predicted image I ^pre _t−1 (predicted image I ^pre1 _t−1 , predicted image I ^pre2 _t−1 ) is generated, the predicted image generating unit 213 of the image predicting unit 21 generates the predicted image I ^pre _t−1. Is sent to the image composition unit 22. Next, in step S <b> 14, the image synthesis unit 22 and the predicted image I ^pre _t−1 generated by the image prediction unit 21 and the image I _t0 (image I ¹ _t0 , image I ² _t0 ) acquired at this prediction time point. _Are combined to generate a composite image I ′ _t0 . That is, for the image I ¹ of the photographing camera 2-1, the image composition unit 22 generates the current time (current frame) generated by applying affine transformation to the image I ¹ _t−1 one frame before acquired one frame before. of the predicted image ^{I pre1} _t-1, by fitting the image ^I _{1 t0} of time, combines the predicted image ^{I pre1} _t-1 and the image ^I _{1 t0} of the current time, the composite image ^{I 1} _'t0 generated thereby (S14-1), the image I ² of the photographic camera 2-2, the moment of the predicted image generated by the action of the affine transformation to one frame image I ^{2 _t-1} of one frame before the previously obtained the I ^pre2 _t-1, by fitting the image ^I _{2 t0} of time, combines the predicted image ^{I pre2} _t-1 and the image ^I _{2 t0} of the current time, if Generating an image ^{I 2} _'t0 (S14-2). Process fitting the predicted image ^{I pre1} _t-1 picture _{I t0} the current in the current, the pixel value of the region corresponding to the image _{I t0} the current in the prediction image ^{I pre1} _t-1 of the current (luminance value), the present time This is executed by replacing the pixel value (luminance value) in the image _It0 . That is, the fitting process is shown in FIG. 6 (C), in the region DM corresponding to the image area to be captured by the photographing camera 2-1 in the prediction image ^{I pre1} _t-1 at the present time, fit the image _{I t0} the moment To be executed.

When the composite image I ′ _t0 (composite image I ¹ ′ _t0 , composite image I ² ′ _t0 ) is generated, the image composition unit 22 notifies the corresponding point search unit 23 of the composite image I ′ _t0 . In step S15, the corresponding point search unit 23 sets one of the two images captured by the imaging cameras 2-1 and 2-2 as the standard image I1 and the other as the reference image I2. The corresponding point of the reference image I2 corresponding to the attention point of the reference image I1 is searched. Then, in the corresponding point search, if there is a combined reference image I1 ′ generated by the image predicting unit 21 and the image combining unit 22 corresponding to the reference image I1, the corresponding point searching unit 23 performs the reference image I1. Is used instead of the reference image I2 ′, and if there is a composite reference image I2 ′ generated by the image prediction unit 21 and the image synthesis unit 22 corresponding to the reference image I2, the composite image is replaced with the reference image I2. A reference image I2 ′ is used. That is, in this embodiment, 'with _t0 is generated, the photographing camera 2-2 composite image ^{I 2'} composite image ^{I 1} for photographing camera 2-1 since _t0 are generated, the corresponding point searching unit 23 For example, the composite image I ¹ ′ _t0 is set as the base image I1 and the composite image I ² ′ _t0 is set as the reference image I2, and the corresponding point of the reference image I2 corresponding to the attention point of the base image I1 is searched. Here, the corresponding point search unit 23 executes the corresponding point search by using any one of the corresponding point searching methods described in step S11. Then, the corresponding point search unit 23 calculates a predetermined value according to the use of the corresponding point search system S, for example, a parallax d _t0 between stereo images.

  And a process is returned from step S15 to step S11, and the above-mentioned process is performed about each flame | frame by repeatedly performing each process from step S11 to step S15.

As described above, the corresponding point search system S and the corresponding point search device 1 according to the present embodiment uses one of the two images as the reference image I1 (for example, the image I ¹ photographed by the photographing camera 2-1). when the other reference image I2 (e.g. photographing camera 2-2 image I ² captured ^by) the reference image I ¹ than the first time point t0 obtaining the time in the past t-1 as well as, t-2 ^Generating the prediction standard image I ^pre1 _t-1 at the first time point t0 predicted based on the _two images I ¹ _t-1 and I ¹ _t-2 acquired at the same time, and acquiring the reference image I ² Two images I ² _t-1 and I ² _t acquired at time points t-1 and t-2 that are past the second time point (in this embodiment, the second time point is the same as the first time point because it is a stereo image) second time predicted based _-2 (= The first time point) combined with the image prediction unit 21 for generating, the predictive reference image ^{I pre1} _t-1 and the image ^I _{1 t0} of the first time of the first time point t0 the prediction reference image ^{I pre2} _t-1 of t0 In this way, the composite reference image I ¹ ′ _t0 is generated, and the predicted reference image I ^pre2 _t−1 at the second time point (= first time point) t0 and the image I ² _t0 at the second time point (= first time point) t0 are generated. corresponding point searching 'and the image synthesis unit 22 for generating _t0, synthetic reference image I ^1' synthetic reference image I ² corresponding points of the synthetic reference image I ² _'t0 corresponding to the current point of _t0 by synthesizing bets And a search unit 23. Since the corresponding point search system S and the corresponding point search device 1 according to the present embodiment operate as described above, when the corresponding point search is performed, for example, as illustrated in FIG. ) Can generate the synthesized image I ′ by the predicted image I ^pre so that the region outside the image is not included. For this reason, even when the calculation area of the corresponding point search is set over the image area and the non-image area, it is possible to reduce erroneous correspondence. In FIG. 7, a region indicated by diagonal lines indicates a portion of the predicted image I ^pre generated by the predicted image generation unit 213 of the image prediction unit 21.

Here, as can be seen from FIG. 7, the size of the predicted image I ^pre searched for corresponding points corresponds to the size of the calculation area (template TP, window WD) around the image I shot by the shooting camera 2. Exceeding the size in the case where the regions are arranged will include the case where the calculation region (TP, WD) is set only in the region outside the image (such a calculation region is not set in the actual corresponding point search). Thus, there is a useless region of the predicted image I ^pre . Therefore, the size of the predicted image I ^pre is preferably set according to the size of the calculation area. The size of the predicted image I ^pre is preferably set to be equal to or smaller than the size when the region corresponding to the size of the calculation region is arranged around the image I photographed by the photographing camera 2.

Further, in the corresponding point search system S and the corresponding point search device 1 in the present embodiment, the image prediction unit 21 uses affine transformation to generate the predicted image I ^pre . Therefore, enlargement / reduction (scaling), rotational movement, and parallel movement can be appropriately performed on an image acquired at a predetermined time point, and an optimal predicted image I ^pre at the prediction time point can be generated.

Further, in the corresponding point search system S and the corresponding point search device 1 in the present embodiment, the two images to be subjected to the corresponding point search are a set of images I ¹ and I taken by the photographing camera 2 that is a stereo camera. It is a stereo image consisting of ^two . For this reason, it is possible to provide a corresponding point search system and a corresponding point search apparatus 1 for a stereo image that can perform corresponding point search with high accuracy up to the edge (end) of the image region.

Further, in the corresponding point search system S and the corresponding point search device 1 in the present embodiment, the image composition unit 22 performs each composite image I ¹ ′ _t0 for each of the pair of images I ¹ _t0 and I ² _{t0 in} the stereo image. , I ² ′ _t0 is generated. For this reason, composite images I ¹ ′ _t0 and I ² ′ _t0 are generated for both images I ¹ _t0 and I ² _{t0 of} the stereo image. For example, as shown in FIG. The composite image I ¹ ′ _t0 can be generated by the predicted image I ^pre1 _t−1 so that the template TP does not include the region outside the image, and is a calculation region as shown in FIG. 7B. It is also possible to generate the synthesized image I ² ′ _t0 by the predicted image I ^pre2 _t−1 so that the window WD does not include the region outside the image. Therefore, it is possible to perform the corresponding point search more accurately up to the edge (end) of the image area.

In the present embodiment, the composite images I ¹ ′ _t0 and I ² ′ _t0 are generated with respect to both the stereo images I ¹ _t0 and I ² _t0 as the standard image I1 and the reference image I2, as described above. Only the composite image I ¹ ′ _t0 may be generated with respect to the image I ¹ _{t0 serving as the} reference image I1. Alternatively, only the composite image I ² ′ _t0 may be generated for the image I ² _t0 to be the reference image I2. With this configuration, information processing is omitted, and it is possible to shorten the calculation time and simplify the calculation processing device.

  Further, in the corresponding point search system S and the corresponding point search device 1 in the present embodiment, the corresponding point search unit 23 is performed by a correlation calculation between the standard image I1 and the reference image I2. For this reason, according to this configuration, it is possible to provide the corresponding point search system S and the corresponding point search device in which the corresponding point search is performed by the correlation calculation between the reference image and the reference image.

  In the corresponding point search system S and the corresponding point search device 1 according to the present embodiment, the corresponding point search unit 23 frequency-decomposes the pattern in the calculation area (template TP and window WD), and suppresses the amplitude component. The corresponding position is calculated based on the similarity. For this reason, it is possible to perform a robust corresponding point search that is not easily affected by a luminance difference or noise between two images.

  Further, in the corresponding point search system S and the corresponding point search device 1 in the present embodiment, the frequency decomposition method is fast Fourier transform, discrete Fourier transform, discrete cosine transform, discrete sine transform, wavelet transform, and Hadamard transform. One of them. Therefore, since an already established method is used, it is possible to reliably perform frequency decomposition.

  In the corresponding point search system S and the corresponding point searching apparatus 1 according to the present embodiment, the corresponding point calculation processing in the corresponding point search unit 23 is a phase-only correlation method (POC). For this reason, it is possible to search for corresponding points with higher accuracy.

Further, in the corresponding point search system S and the corresponding point search device 1 in the present embodiment, as described above, each process from step S11 to step S15 is repeatedly executed. Accordingly, in the above embodiment, the first composite image I _'t0, obtained from the image _{I t-2} and 1-frame preceding image _{I t-1} two frames before the image _{I t-1} of one frame before While the predicted image I ^pre _t−1 generated by applying the affine transformation and the current frame image I _t0 are combined, the next combined image I ′ _{t + 1} is one frame in the combined image I ′ _t0. The prediction image I ^pre _{t + 1} generated by applying the affine transformation obtained from the previous image I _t-1 and the current image (current frame) I _t0 and the current frame image I _{t + 1} may be combined. For example, as shown in FIG. 8 (A), the first composite image I _'t-3, the three frames before the image _{I t-3} to 4 frames before the image _{I t-4} and 3 frame preceding image I _The predicted image I ^pre _t-3 generated by applying the affine transformation PA ₍₄₃₎ obtained from _t-3 and the image I _t-2 two frames before are combined, and the next combined image I ′ _{t -2} is an affine transformation P _{A (A)} obtained from the image I _{t-3 three} frames before and the image It _-2 of two frames before the synthesized image I ′ _t-3 , as shown in FIG. ₃₂₎ is synthesized from the predicted image I ^pre _t-2 generated by applying the above and the image I _t-1 of one frame, and the next synthesized image I ′ _t-1 is shown in FIG. 8C. as such, our composite image I _'t-2 two frames before the image _{I t-2} Is synthesized from the predicted image ^I _{pre t-1} and the current frame image _{I t0} Metropolitan generated by the action of affine transformation _{P A (21)} obtained from the image _{I t-1} beauty preceding frame.

Next, another embodiment will be described.
(Second Embodiment)
In the first embodiment, the corresponding point search system S and the corresponding point search apparatus 1 perform corresponding point search on a stereo image composed of a pair of images I ¹ and I ² photographed by a stereo camera. In the embodiment, the corresponding point search system S and the corresponding point search apparatus 1 are images acquired at different time points among time-series images sequentially captured at predetermined time intervals (frame periods) by a photographing camera. The corresponding points are searched for.

  For this reason, in the corresponding point search system S and the corresponding point search device 1 in the second embodiment, the photographing camera 2 is configured by one photographing camera, for example, the photographing camera 2-1, and is functionally configured by the CPU 12. In the first embodiment, except that the image prediction unit 21 (the data acquisition unit 211, the coefficient calculation unit 212, and the predicted image generation unit 213), the image synthesis unit 22, and the corresponding point search unit 23 function as described later. This is the same as the corresponding point search system S and the corresponding point search device 1.

  In the present embodiment, the corresponding point search is executed for the time-series image captured by the imaging camera 2-1, but the time-series image captured by the imaging camera 2-2 is also provided. Alternatively, the corresponding point search in this embodiment may be executed.

  FIG. 9 is a flowchart showing the operation of the corresponding point search device in the corresponding point search system of the second embodiment.

  When the corresponding point search system S is activated and its operation is started, a time-series image (time-series image data) of the preceding object Ob sequentially captured at a predetermined frame period by one imaging camera 2 is acquired. The Then, each time-series image (each time-series image data) is sequentially input from the communication unit 17 to the corresponding point search device 1 and notified to the CPU 12 of the corresponding point search device 1, and the CPU 12 determines between the time-series images. Corresponding point search processing is started.

In the corresponding point search process between the time series images, as shown in FIG. 9, first, in step S21, the data acquisition unit 211 of the image prediction unit 21 calculates coefficients from the time series images taken by the photographing camera 2. Data used by the calculation unit 212 is acquired. Since the prediction image generation unit 213 generates an image at a prediction time point different from the predetermined time point based on an image acquired at a predetermined time point by using affine transformation, the coefficient calculation unit 212 generates the prediction image. It is necessary to calculate the affine coefficient of the affine transformation. Therefore, more specifically, the data acquisition unit 211 searches for at least three corresponding points between two images of the time-series images. In the present embodiment, for example, an image photographed by the photographing camera 2 at time t in the case of a I _t, the data acquisition unit 211, for example, current time (in the current frame time) t0 than one frame before the time t- Corresponding point search between images I _t-1 and I _t-2 at time t-2 one and two frames before is performed, and corresponding point search result OF _{t (21)} including at least three corresponding points (at least three Corresponding point search result OF _{t (21)} ) including at least three corresponding points respectively corresponding to the attention point is acquired. OF _{t (21)} is a corresponding point search result between the image It _{-1 of one} frame before and the image It _-2 of two frames before taken by the photographing camera 2. The data acquisition unit 211 uses any one of the corresponding point search methods described in step S11 in the first embodiment for the corresponding point search between the two images It _-1 and It _-2. Execute by.

When data is acquired, the data acquisition unit 211 notifies the coefficient calculation unit 212 of the data. Next, in step S22, the coefficient calculation unit 212 of the image prediction unit 21 uses the data acquired by the data acquisition unit 211 to perform the affine coefficient of affine transformation by the same processing as step S12 in the first embodiment. calculate. That is, the coefficient calculation unit 212 uses the corresponding point search result OF _{t (21)} including at least three corresponding points acquired by performing the corresponding point search in the data acquisition unit 211, and uses the affine coefficient PA _{( 21)} is calculated. Note that PA ₍₂₁₎ is an affine coefficient (affine parameter) calculated based on the image I _{t-1 one} frame before and the image It _t-2 two frames before captured by the photographing camera 2. .

When the affine coefficient PA ₍₂₁₎ is calculated, the coefficient calculation unit 212 notifies the prediction image generation unit 213 of the affine coefficient PA ₍₂₁₎ . Next, in step S23, the prediction image generation unit 213 of the image prediction unit 21 calculates an image at a prediction time point different from the predetermined time point based on the image acquired at the predetermined time point by the coefficient calculation unit 212. Prediction is performed by using an affine transformation having PA _(21), and a prediction image is generated.

More specifically, first, when the prediction time is the current time, the predicted image generation unit 213 predicts the current image by affine transformation with respect to images acquired at a time earlier than the current time. A prediction image is generated. For example, the prediction image generator 213, current I by exerting an affine transformation of current (current frame) image was acquired at time t-1 of one frame before t0 I _t-1 to the affine coefficients P _{A (21)} A predicted image I ^pre _t-1 of t0 is generated (S23-1).

Second, when the prediction time point is a past time point, the prediction image generation unit 213 generates a prediction image by predicting an image at a past time point by affine transformation with respect to the image acquired at the current time point. . For example, the predicted image generation unit 213 _applies the affine transformation of the affine coefficient P ⁻¹ _{A (21)} to the image It0 acquired at the current time (current frame) t0 to thereby predict the predicted image at the time t−1 one frame before. I ^pre _t0 is generated (S23-2). The affine coefficient PA ₍₂₁₎ obtained by the process of step S22 is obtained by using the image It _-2 that is two frames before as the standard image I1 and the image It _-1 that is one frame before as the reference image I2. Is. Therefore, affine coefficients in the case of step S23-2 to generate a prediction image ^I _{pre t0} of past time t-1 from the image _{I t0} the moment t0, the affine transformation is the inverse matrix of the affine coefficients _{P A (21)} P ^- _{1A (21)} is required.

When the predicted images I ^pre _t−1 and I ^pre _t0 are generated, the predicted image generating unit 213 of the image predicting unit 21 notifies the image combining unit 22 of the predicted images I ^pre _t−1 and I ^pre _t0 . Next, in step S24, the image synthesis unit 22 predicts the images I ^pre _t-1 and I ^pre _t0 generated by the image prediction unit 21 and the images I corresponding to the prediction images I ^pre _t-1 and I ^pre _{t0. t0, I t-1} and the combined image I by the synthesizing _'t0, I' to generate a _t-1. That is, since the optical flow extends radially from the vanishing point in the predicted image I ^pret ₋₁ , the image composition unit 22 fits the current image _It0 in the current predicted image I ^pret _−1. by writing, by combining the predicted image ^I _{pre t-1} and the current image _{I t0} of time, to generate a composite image I _'t0 (S24-1), the image combining unit 22, the predicted image ^I _{pre t0} Then, since the optical flow extends toward the vanishing point, the predicted image I one frame before is inserted by inserting the image I _t-1 one frame before in the predicted image I ^pre _t0 one frame before. ^Pre _t0 and the image I _{t-1 of the} previous frame are combined to generate a combined image I ′ _t-1 (S24-2).

Here, in this step S24, the image composition unit 22 determines the predicted image I ^pre to be synthesized and the prediction time point according to the mode of relative motion between the photographing camera 2 and the subject (for example, the preceding object Ob in the present embodiment). The acquired image I may be determined. By being configured in this way, the composition target of the composite image can be changed as appropriate, and the corresponding point search can be performed effectively.

  FIG. 10 is a diagram for explaining generation of a predicted image and generation of a composite image when the optical flow extends radially from the vanishing point. FIG. 10A shows a mode of generating a predicted image, and FIG. 10B shows an optical flow. FIG. 11 is a diagram for explaining generation of a predicted image and generation of a composite image when the optical flow extends toward the vanishing point. FIG. 11A shows a manner of generating a predicted image, and FIG. 11B shows an optical flow.

The mode of relative motion between the photographing camera 2 and the subject can be obtained, for example, by an optical flow obtained from a time-series image. The optical flow is a distribution of a two-dimensional velocity vector on an image generated by a three-dimensional movement of an object. For example, an image at a time point earlier than the first time point is set as a reference image I1, and an image at the first time point is changed. In the case of the reference image I2, the corresponding point of the reference image I2 corresponding to the target point of the standard image I1 is searched by the corresponding point search to obtain a vector from the target point to the corresponding point. More specifically, for example, when the image It _-1 of the previous frame is set as the reference image I1 and the current image (current frame) _It0 is set as the reference image I2, the optical flow of the reference image I1 is determined. By searching the corresponding point of the reference image I2 corresponding to the target point by the corresponding point search, it is obtained as a vector from the target point to the corresponding point.

When the optical flow obtained in this way extends radially from the vanishing point NP as shown in FIG. 10B, for example, the photographing camera 2 moves so as to approach the subject, that is, moves forward. Therefore, as shown in FIG. 10A, the predicted image generation unit 213 predicts the current t0 by applying the affine transformation of the affine coefficient PA ₍₂₁₎ to the image I _t-1 one frame before. generating an image _{^{I pre t-1 (S23-1)}} , the image synthesizing unit 22 synthesizes the image _{I t0} predicted image ^I _{pre t-1} and the current t0 the current t0, generating a composite image I _'t0 (S24-1). Image synthesizing unit 22, the predicted image ^I _{pre t-1} of the current t0, the region DM corresponding to the image area to be captured by the photographing camera 2, both images by fitting the images _{I t0} the moment t0 ^{I pre} _t-1 and _It0 are synthesized. The vanishing point NP is given as an intersection of optical flows, for example.

On the other hand, when the obtained optical flow extends toward the vanishing point NP as shown in FIG. 11B, for example, the photographing camera 2 moves away from the subject, that is, moves backward. Therefore, as shown in FIG. 11A, the predicted image generation unit 213 predicts the current t0 by applying the affine transformation of the affine coefficient PA ₍₂₁₎ to the image I _t-1 one frame before. generating an image _{^{I pre t-1 (S23-1)}} , the image synthesizing unit 22 synthesizes the image _{I t0} predicted image ^I _{pre t-1} and the current t0 the current t0, generating a composite image I _'t0 (S24-1). Image synthesizing unit 22, the predicted image ^I _{pre t-1} of the current t0, the region DM corresponding to the image area to be captured by the photographing camera 2, both images by fitting the images _{I t0} the moment t0 ^{I pre} _t-1 and _It0 are synthesized. However, in this case (during backward movement), since the image _It0 at the current time _t0 is larger than the current predicted image ^Ipret _-1 , the combined predicted image ^Ipret _-1 is hidden under the current time t0. would be, becomes an image _{I t0} at the present time t0 is substantially.

  In the above description, a predicted image is created based on a past image, and a composite image is created by fitting the current image therein. However, as described in FIG. 9, the past image is created based on the current image. A predicted image may be created, and a composite image may be created by inserting a past image therein.

In the above description, when the photographing camera 2 is moving forward, as shown in FIG. 10, the affine transformation of the affine coefficient PA ₍₂₁₎ is applied to the image It _-1 before one frame as shown in FIG. A prediction image I ^pre _{t-1 at t0} is generated, and a composite image I ′ _t0 is generated by combining the prediction image I ^pre _t-1 at the current time t0 and the image I _t0 at the current time t0. as shown, when the photographing camera 2 is forward movement, the synthetic image I 'not only _t0 is generated, further, an affine transformation of the affine coefficients P ^{-1 a ₍₂₁₎} in the image I _t0 the moment As a result, a predicted image I ^pre _t0 one frame before is also generated, and the predicted image I ^pre _t0 one frame before and the image I _t-1 acquired one frame before are combined. The composite image I ′ _t−1 may also be generated. Similarly, in the above description, in the case where photographing camera 2 is backward movement, as shown in FIG. 11, one frame I by exerting an affine transformation of the affine coefficients P ^{-1 A ₍₂₁₎} in the image I _t0 the moment A previous predicted image I ^pre _t0 is generated, and a synthesized image I ′ _t-1 is generated by synthesizing the predicted image I ^pre _t0 one frame before and the image I _t−1 acquired one frame before However, as shown in FIG. 13, when the photographing camera 2 is moving backward, not only the composite image I ′ _t−1 is generated, but also the affine coefficient is added to the image I _t−1 one frame before. By applying the affine transformation of PA ₍₂₁₎ , a predicted image I ^pre _t-1 at the current time t0 is also generated, and the predicted image I ^pre _t-1 at the current time t0 and the image I _t0 at the current time t0 are synthesized. Yo Thus, the composite image I ′ _t0 may also be generated. By being configured in this way, when the motion direction is changed, it is not necessary to switch by determining the motion direction, and it is possible to perform a corresponding point search with a predicted image from the time when the motion direction is changed. It becomes.

Note that, in the generation of the composite image I ′, when the size of the predicted image I ^pre is smaller than the region DM corresponding to the image region photographed by the photographing camera 2, the predicted image I ^pre below the measured image I. May be hidden and the composite image I ′ may be substantially the actual measurement image I.

When the composite images I ′ _t0 and I ′ _t−1 are generated, the image composition unit 22 notifies the corresponding point search unit 23 of the composite images I ′ _t0 and I ′ _t−1 . In step S25, the corresponding point search unit 23 sets the attention point of the standard image I1 when one of the two images captured by the shadow camera 2 is the standard image I1 and the other is the reference image I2. The corresponding point of the reference image I2 corresponding to is searched. Then, in the corresponding point search, if there is a combined reference image I1 ′ generated by the image predicting unit 21 and the image combining unit 22 corresponding to the reference image I1, the corresponding point searching unit 23 performs the reference image I1. Is used instead of the reference image I2 ′, and if there is a composite reference image I2 ′ generated by the image prediction unit 21 and the image synthesis unit 22 corresponding to the reference image I2, the composite image is replaced with the reference image I2. A reference image I2 ′ is used. That is, in the present embodiment, since these composite images I ′ _t0 and I ′ _t−1 are generated, the corresponding point search unit 23 uses, for example, the composite image I ′ _t−1 as the reference image I1 and the composite image. Using I ′ _t0 as the reference image I2, the corresponding point of the reference image I2 corresponding to the target point of the base image I1 is searched. Here, the corresponding point search unit 23 executes the corresponding point search by using any one of the corresponding point searching methods described in step S11. Then, the corresponding point search unit 23 calculates a predetermined value according to the use of the corresponding point search system S.

  Then, the process returns from step S25 to step S22, and the processes described above are executed for each frame by repeatedly executing the processes from step S21 to step S25.

Since the corresponding point search system S and the corresponding point search device 1 according to this embodiment perform the corresponding point search, for example, as illustrated in FIG. 14, the calculation area (TP, WD) is an image. The synthesized image I ′ can be generated by the predicted image I ^pre so as not to include the outer region. For this reason, even when the calculation area of the corresponding point search is set over the image area and the non-image area, it is possible to reduce erroneous correspondence.

Further, in the corresponding point search system S and the corresponding point search device 1 in the present embodiment, the image synthesis unit 22 is acquired at the predicted image I ^pre and the predicted time according to the optical flow obtained based on the time-series image. Image I is determined. For this reason, the synthesis target of the synthesized image can be changed as appropriate, and the corresponding point search can be performed effectively.

Further, in the corresponding point search system S and the corresponding point search device 1 in the present embodiment, the image prediction unit 21 is based on the image It _-1 obtained one frame before the current time t0, and the current predicted image ^Ipre. It generates a _t-1, the image synthesizing unit 22 synthesizes the image _{I t0} predicted image ^I _{pre t-1} and the current t0 obtained in the present time of the current. For this reason, when the image on which the corresponding point search is performed is an image obtained by the photographing camera attached to the moving body, the corresponding point search can be performed more accurately when the moving body is moving forward. It becomes possible.

Further, in the corresponding point search system S and the corresponding point search device 1 in the present embodiment, the image prediction unit 21 predicts the predicted image I ^pre _t0 one frame before the current t0 based on the image I _t0 acquired at the current time _t0. The image composition unit 22 synthesizes the predicted image I ^pre _t0 one frame before and the image I _t-1 one frame before acquired one frame before. For this reason, when the image on which the corresponding point search is performed is an image obtained by a photographing camera attached to the moving body, the corresponding point search is performed more accurately when the moving body is moving backward (backward). Can be done.

FIG. 14 is a diagram for explaining the relationship between the position of the photographing camera and the photographing conditions. In the first and second embodiments described above, the affine coefficients P _A of the affine transformation was obtained by performing the corresponding point search between the two images different from each other acquisition times, at least three corresponding points Although it was calculated | required based on the corresponding point search result OF containing, it may be calculated | required based on the optical flow calculated | required based on two images from which acquisition time differs mutually. For example, the target point and the corresponding point in the corresponding point search results OF, by using the start and end points of a vector representing the optical flow, respectively, affine coefficients P _A of the affine transformation is determined. According to this configuration, it is possible to obtain the affine coefficient of the affine transformation from only the image without using another external device.

On the other hand, in this case, the image is acquired by the photographing camera 2 mounted on the movable body, affine coefficients P _A of the affine transformation, based on the steering information and velocity information about the free-running speed relating to rotation angle according to the steering of the mobile May be obtained based on the optical flow obtained in the above. The steering information is preferably acquired by a rudder angle sensor that detects a rotation angle by steering of the moving body attached to the moving body, and the speed information indicates the speed of the moving body attached to the moving body. It is preferably acquired by a speed sensor to detect. According to this configuration, since the steering information and the speed information are acquired from other external devices such as a rudder angle sensor and a speed sensor, the calculation time can be shortened and the arithmetic processing device can be simplified. it is possible to obtain the affine coefficients P _a accurately affine transformation.

Here, as shown in FIG. 14, the photographing camera 2 having a focal length f and an angle of view θ is attached to the moving body at a height h from the road surface with an inclination φ in the pitch direction with respect to the forward direction Z. It is assumed that the inclination in the pan direction and the roll direction is substantially zero. In such a case, the horizontal component Δx and the vertical component Δy of the optical flow are expressed by Expression 11-1 and Expression 11-2.
Δx = (vt (y + αφ) x / (αh)) − (α + vt (y + αφ) / (2h)) σt
... (11-1)
Δy = (vt (y + αφ) ² / (αh)) (11-2)
Here, v is the speed of the moving body, t is the shooting time interval between the two images, and y is a point on the image representing the point A on the road surface away from the shooting camera 2 in the Z direction. X is a horizontal component of a point on the image representing a point A on the road surface away from the photographing camera 2 in the Z direction, and α is a pixel pitch p and the number of pixels is N. α = f / p = N / (2θ), and σt is a rotation angle by steering (σ is an angular velocity).

FIG. 15 is a diagram for explaining another method for generating a predicted image. In the first and second embodiments described above, the predicted image I ^pre _t-1 generated by the predicted image generation unit 213 is generated by collectively handling the original image It _-1 before prediction. However, as shown in FIG. 15, the original image It _-1 is divided into a plurality of images (FIG. 15 (A)), and the affine coefficient P ⁿ _{A (21)} (n is divided into each of these divided divided images. for each of the divided divided images I ⁿ _t−1 , and the predicted image I ^pren _t−1 is obtained for each divided divided image I ⁿ _t−1 , and these are connected to maintain the positional relationship before and after the prediction. The predicted image I ^pre _t-1 may be generated by combining them together (FIG. 15B). Each divided image I ^{n _t-1} size (size) may be square, also may be rectangular, or may be different not need more necessarily identical. In this way, by dividing the original image I _t-1 and then generating the predicted image I ^pre _t-1 , it is possible to divide the region where the movement amount varies greatly and the region where the movement amount varies slightly during one frame. Therefore, the predicted image I ^pre _t-1 can be generated more accurately.

Also in this case, as in the above-described embodiment, the predicted image I ^pre _t-1 is obtained by the camera 2 as shown in FIG. The area DM corresponding to the image area to be photographed may be extracted.

  FIG. 16 is a diagram for explaining the multi-resolution strategy. In the first and second embodiments described above, the corresponding point search unit 23 may execute a corresponding point search using a multi-resolution strategy. In the corresponding point search using the multi-resolution strategy, as shown in FIG. 16, the resolution of the base image I1 and the reference image I2 is reduced, and the corresponding point search is performed using the low-resolution base image I1 and the reference image I2. This is a technique for sequentially increasing the resolution to the original resolution while propagating the solution (corresponding point search result). More specifically, first, as the first process, each reduced image of the standard image I1 and the reference image I2 is created by, for example, a thinning process for thinning out pixels (reduction in resolution). Next, as a second process, a corresponding point search is performed using the reduced images of the standard image I1 and the reference image I2. Next, as a third process, the reduced image of the reference image I2 is enlarged at a predetermined magnification in a predetermined range including the corresponding points searched by the corresponding point search, and the reference image is used in the predetermined range including the attention point. The reduced image of the image I1 is enlarged at a predetermined magnification (propagation of the solution). Next, as a fourth process, a corresponding point search is performed using the enlarged standard image I1 and reference image I2. Next, as the fifth process, the third process and the fourth process are repeated until the original size (size, resolution) is reached.

  In the corresponding point search using the multiplexed resolution strategy, there are many cases where the calculation area includes an area outside the image. In one embodiment of the present invention, as described above, a predicted image is generated, a synthesized image is generated based on the predicted image, and a corresponding point search is performed by using the synthesized image. The point search system S and the corresponding point search device 1 can perform the corresponding point search with higher accuracy when performing the corresponding point search by the multi-resolution strategy.

  In order to express the present invention, the present invention has been properly and fully described through the embodiments with reference to the drawings. However, those skilled in the art can easily change and / or improve the above-described embodiments. It should be recognized that this is possible. Therefore, unless the modifications or improvements implemented by those skilled in the art are at a level that departs from the scope of the claims recited in the claims, the modifications or improvements are not covered by the claims. To be construed as inclusive.

It is a figure which shows schematic structure of the corresponding point search system in 1st Embodiment. It is a block diagram which shows the structure of the corresponding point search apparatus in the corresponding point search system of embodiment. It is a flowchart which shows operation | movement of the corresponding point search apparatus in the corresponding point search system of 1st Embodiment. It is a block diagram of a phase only correlation method. It is a graph which shows an example of a POC value. It is a figure for demonstrating the production | generation method of an estimated image. It is a figure for demonstrating the magnitude | size of an estimated image. (A) is a case of a standard image, and (B) is a case of a reference image. It is a figure for demonstrating the production | generation method of a synthesized image. It is a flowchart which shows operation | movement of the corresponding point search apparatus in the corresponding point search system of 2nd Embodiment. It is a figure for demonstrating the production | generation of a prediction image and the production | generation of a synthesized image in case an optical flow is extended radially from a vanishing point. It is a figure for demonstrating the production | generation of a prediction image and the production | generation of a synthesized image in case an optical flow is extended toward a vanishing point. It is a figure for demonstrating the production | generation of the other estimated image in the case where an optical flow is extended radially from a vanishing point, and the production | generation of a synthesized image. It is a figure for demonstrating the production | generation of another prediction image and the production | generation of a synthesized image in case an optical flow is extended toward a vanishing point. It is a figure for demonstrating the production | generation method of another composite image. It is a figure for demonstrating the relationship between the position of an imaging camera, and imaging conditions. It is a figure for demonstrating the other production | generation method of an estimated image. It is a figure for demonstrating a multi-resolution strategy. It is a figure for demonstrating the miscorrespondence at the time of performing a corresponding point search between two images. It is a figure for demonstrating the miscorrespondence at the time of performing a corresponding point search between two images of a time series image.

Explanation of symbols

S Corresponding Point Search System 1 Corresponding Point Search Device 2 Shooting Camera 12 CPU
21 image prediction unit 22 image synthesis unit 23 corresponding point search unit 211 data acquisition unit 212 coefficient calculation unit 213 prediction image generation unit

Claims (16)

In a corresponding point search device for searching for points corresponding to each other between two images,
A corresponding point search unit that searches one of the two images as a reference image and the other as a reference image, and searching for a corresponding point in the reference image corresponding to a point of interest in the reference image;
An image prediction unit that generates, as a predicted image, an image at a prediction time point different from the predetermined time point for the standard image or the reference image acquired at a predetermined time point;
An image synthesizing unit that generates a synthesized image at the prediction time point by replacing a region corresponding to the image obtained at the prediction time point in the prediction image with an image obtained at the prediction time point;
The corresponding point searching unit uses the synthesized image synthesized by the image synthesizing unit as one of the standard image or the reference image. The corresponding point search apparatus according to claim 1, wherein the image prediction unit uses affine transformation to generate the predicted image. The affine coefficient of the affine transformation is obtained based on an optical flow obtained based on an image obtained at the predetermined time point and an image obtained at a time point different from the predetermined time point. 2. The corresponding point search device according to 2. The image is acquired by a photographing camera attached to a moving body,
The affine coefficient of the affine transformation is obtained based on an optical flow obtained based on steering information related to a rotation angle by steering of the moving body and speed information related to a self-running speed. Corresponding point search device. The steering information is acquired by a steering angle sensor that detects a rotation angle by steering of the mobile body attached to the mobile body,
The corresponding point search apparatus according to claim 4, wherein the speed information is acquired by a speed sensor that detects a speed of the moving body attached to the moving body. The corresponding point search device according to claim 1, wherein the two images are stereo images including a set of images taken at the prediction time point by a stereo camera. The corresponding point search apparatus according to claim 6, wherein the image composition unit generates each composite image for each of a set of images in the stereo image. The corresponding point search device according to claim 1, wherein the two images are images acquired at different points in time in the time-series images. The image prediction unit generates a predicted image at the first time point based on images acquired at a time point earlier than the first time point,
When the optical flow obtained based on the time-series image extends radially from the vanishing point, the image composition unit and the predicted image at the first time point and the first time point obtained at the first time point The corresponding point search apparatus according to claim 8, wherein the corresponding point is synthesized with an image. The image prediction unit generates a predicted image at a time point earlier than the first time point based on the image acquired at the first time point,
When the optical flow obtained based on the time-series image extends toward the vanishing point, the image composition unit is configured to predict the predicted image at the past time point and the past time point acquired at the past time point. The corresponding point search apparatus according to claim 8, wherein the corresponding point is synthesized with an image. The corresponding point search apparatus according to claim 1, wherein the corresponding point search unit is performed by a correlation calculation between the base image and the reference image. The corresponding point search device according to claim 1, wherein the corresponding point search unit calculates a corresponding position based on a similarity of signals in which a pattern in a calculation region is frequency-resolved and an amplitude component is suppressed. The corresponding point search according to claim 12, wherein the frequency decomposition method is any one of fast Fourier transform, discrete Fourier transform, discrete cosine transform, discrete sine transform, wavelet transform, and Hadamard transform. apparatus. The corresponding point search apparatus according to claim 13, wherein the corresponding point calculation processing in the corresponding point search unit is a phase-only correlation method. The corresponding point search apparatus according to claim 14, wherein the corresponding point search unit uses a multi-resolution strategy. In a corresponding point search method for searching for points corresponding to each other between two images,
A corresponding point search step of searching for a corresponding point of the reference image corresponding to a target point of the reference image, with one of the two images being a reference image and the other being a reference image;
An image prediction step of generating an image at a prediction time point different from the predetermined time point as a prediction image for the standard image or the reference image acquired at a predetermined time point;
An image synthesis step for generating a post-combination image at the prediction time point by replacing a region corresponding to the image obtained at the prediction time point in the prediction image with an image obtained at the prediction time point,
In the corresponding point searching step, the combined image synthesized in the image synthesizing step is used for one of the base image or the reference image.