JP2016152027A - Image processing device, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, image processing method and program capable of reducing a load of image processing.SOLUTION: The image processing device includes: extraction means for extracting a feature point of a subject in a frame of a video image picked up by at least one imaging means in two imaging means; matching means for performing a matching operation for retrieving a feature point that coincides with a feature point in the next frame of a frame with the feature point reflected therein; determination means for determining whether the feature point disappears in the next frame on the basis of a result of the matching operation; prediction means for acquiring a prediction point at which the position of the disappeared feature point is predicted on the basis of an operation of a replacement point that replaces the disappeared feature point in the case that the determination means determines that the feature point disappears; and return means for returning a point based on the prediction point as a feature point being a tracking object in the case that the point based on the prediction point coincides with the feature point before disappearance.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  2. Description of the Related Art As a technique for creating a map necessary for autonomous driving of a car or a robot in recent years, a technique for autonomously constructing a map and estimating its own position (SLAM: Simulatively Localization Mapping) is known in an unknown environment. It has been. In SLAM, feature points are extracted from sensor information such as camera images or laser radar, feature points are tracked in time series, posture estimation and 3D reconstruction are performed based on the results, map construction and self-location Estimate

  However, for example, when a camera image is used, the feature point disappears when an object moving in front of the camera passes, or in the case of a stereo camera or the like, the feature point is visible on one camera due to parallax. Occlusion that the feature point cannot be seen by the other camera may occur, and tracking of the feature point may fail. In order to cope with such failure of tracking of feature points, when occlusion occurs, by changing the parameters, the tracking range of the feature points is reduced and the feature points are tracked again. A technique for reducing point tracking failures has been proposed (see Patent Document 1).

  However, in the technique described in Patent Document 1, it is necessary to use a plurality of past images in which feature points cannot be tracked in a portion where occlusion occurs in order to reduce failure in tracking feature points. In addition, since it is necessary to perform tracking again, there is a problem that it leads to a load of image processing.

  The present invention has been made in view of the above, and an object thereof is to provide an image processing apparatus, an image processing method, and a program capable of reducing the load of image processing.

  In order to solve the above-described problems and achieve the object, the present invention provides an extraction unit that extracts a feature point of a subject in a frame of an image captured by at least one of two imaging units, and the feature Matching means for performing a matching operation for searching for a feature point matching the feature point in a frame next to the frame in which the point is shown, and based on the result of the matching operation, the feature point is lost in the next frame. A determination unit that determines whether or not the feature point has disappeared, and when the determination unit determines that the feature point has disappeared, the position of the disappeared feature point based on the movement of the alternative point that replaces the lost feature point When the prediction means for obtaining the prediction point predicting the point, the point based on the prediction point, and the feature point before disappearing match, the point based on the prediction point is the tracking target And restoration means for restoring the feature points, and further comprising a.

  According to the present invention, it is possible to reduce the load of image processing.

FIG. 1 is an overview of the image processing apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example in which the image processing apparatus according to the first embodiment is mounted on a vehicle. FIG. 3 is a diagram illustrating an example of a hardware configuration of the image processing apparatus according to the first embodiment. FIG. 4 is a diagram illustrating an example of a block configuration of the image processing apparatus according to the first embodiment. FIG. 5 is a diagram illustrating an example of a block configuration of a distance measurement unit of the image processing apparatus according to the first embodiment. FIG. 6 is a diagram illustrating an example of a block configuration of a feature point tracking unit of the image processing apparatus according to the first embodiment. FIG. 7 is a diagram for explaining the principle of deriving the distance from the imaging device to the object. FIG. 8 is a conceptual diagram showing a reference image, a high-density parallax image, and an edge parallax image. FIG. 9 is an explanatory diagram for obtaining a corresponding pixel in a comparison image corresponding to a reference pixel in a reference image. FIG. 10 is a diagram illustrating an example of a graph indicating the relationship between the shift amount and the cost value. FIG. 11 is a conceptual diagram for deriving the synthesis cost. FIG. 12 is a diagram illustrating an example of a graph indicating the relationship between the shift amount and the combined cost value. FIG. 13 is a diagram illustrating an example of a feature point tracking failure and success state. FIG. 14 is a diagram for explaining the block matching operation. FIG. 15 is a diagram for explaining processing in the search range in the block matching operation. FIG. 16 is a diagram for explaining an occlusion determination operation. FIG. 17 is a diagram illustrating a grouping operation for estimating a prediction point in the image processing apparatus according to the first embodiment. FIG. 18 is a diagram illustrating a grouping operation for estimating a prediction point in the image processing apparatus according to the first embodiment. FIG. 19 is a diagram for explaining a prediction point estimation operation when occlusion occurs in the image processing apparatus according to the first embodiment. FIG. 20 is a diagram for explaining the tracking point return operation in the image processing apparatus according to the first embodiment. FIG. 21 is a diagram illustrating an example of an operation flow of image processing of the image processing apparatus according to the first embodiment. FIG. 22 is a diagram for explaining the tracking point return operation in the image processing apparatus according to the modification of the first embodiment. FIG. 23 is a diagram illustrating an example of a block configuration of the image processing apparatus according to the second embodiment. FIG. 24 is a diagram illustrating an example of a block configuration of a feature point tracking unit of the image processing apparatus according to the second embodiment. FIG. 25 is a diagram for explaining a prediction point estimation operation when occlusion occurs in the image processing apparatus according to the second embodiment. FIG. 26 is a diagram for explaining the tracking point return operation in the image processing apparatus according to the modification of the third embodiment.

[First Embodiment]
(Overview of image processing device and installation example)
FIG. 1 is an overview of the image processing apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example in which the image processing apparatus according to the first embodiment is mounted on a vehicle. An overview and an installation example of the image processing apparatus 1 will be described with reference to FIGS.

  As shown in FIG. 1, the image processing apparatus 1 according to the present embodiment includes a main body 2 and a pair of cylindrical imaging devices 10 a and 10 b fixed to the main body 2. The image processing apparatus 1 is installed so that the imaging devices 10a and 10b can capture a scene in front of the traveling direction of the automobile, for example. For convenience of explanation, the imaging device 10a may be referred to as a “right camera” and the imaging device 10b may be referred to as a “left camera”.

  For example, as illustrated in FIG. 2, the image processing apparatus 1 is installed in the vicinity of a rearview mirror inside a windshield of a living room space of a vehicle 100 such as an automobile. 2A is a schematic side view of the vehicle 100 on which the image processing apparatus 1 is mounted, and FIG. 2B is a front view of the vehicle 100.

  The image processing apparatus 1 may be mounted not only on an automobile as an example of a vehicle but also on a motorcycle, a bicycle, a wheelchair, an agricultural cultivator, or the like as another example of a vehicle.

(Hardware configuration of image processing device)
FIG. 3 is a diagram illustrating an example of a hardware configuration of the image processing apparatus according to the first embodiment. The hardware configuration of the image processing apparatus 1 will be described with reference to FIG.

  As shown in FIG. 3, the image processing apparatus 1 includes a main body 2, an imaging apparatus 10a (imaging means), an imaging apparatus 10b (imaging means), a signal conversion apparatus 20a, a signal conversion apparatus 20b, and a tracking process. Device 30.

  The main body 2 constitutes the housing of the image processing apparatus 1, fixes and supports the imaging devices 10 a and 10 b, and incorporates the signal conversion devices 20 a and 20 b and the tracking processing device 30.

  The imaging device 10a is a device that images an object in front and generates an analog image signal. The imaging device 10a includes an imaging lens 11a, a diaphragm 12a, and an image sensor 13a.

  The imaging lens 11a is an optical element for refracting incident light to form an object image. The diaphragm 12a is a member that adjusts the amount of light reaching the image sensor 13a by blocking part of the light that has passed through the imaging lens 11a. The image sensor 13a is a solid-state image sensor that converts light that has passed through the imaging lens 11a and the diaphragm 12a into an electrical analog image signal. The image sensor 13a is realized by, for example, a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxide Semiconductor).

  The imaging device 10b is a device that images an object in front and generates an analog image signal. The imaging device 10b includes an imaging lens 11b, a diaphragm 12b, and an image sensor 13b. The functions of the imaging lens 11b, the diaphragm 12b, and the image sensor 13b are the same as the functions of the imaging lens 11a, the diaphragm 12a, and the image sensor 13a, respectively. Further, the imaging lens 11a and the imaging lens 11b are installed such that their lens surfaces are on the same plane.

  The signal conversion device 20a is a device that converts the analog image signal generated by the imaging device 10a into digital image data. The signal conversion device 20a includes a CDS (Correlated Double Sampling) 21a, an AGC (Auto Gain Control) 22a, an ADC (Analog Digital Converter) 23a, and a frame memory 24a.

  The CDS 21a removes noise from the analog image signal generated by the image sensor 13a by correlated double sampling, a horizontal differential filter, a vertical smoothing filter, and the like. The AGC 22a performs gain control for controlling the intensity of the analog image signal from which noise has been removed by the CDS 21a. The ADC 23a converts an analog image signal whose gain is controlled by the AGC 22a into digital image data. The frame memory 24a stores the image data converted by the ADC 23a.

  The signal conversion device 20b is a device that converts an analog image signal generated by the imaging device 10b into digital image data. The signal conversion device 20b includes a CDS 21b, an AGC 22b, an ADC 23b, and a frame memory 24b. The functions of the CDS 21b, AGC 22b, ADC 23b, and frame memory 24b are the same as the functions of the CDS 21a, AGC 22a, ADC 23a, and frame memory 24a, respectively.

  The tracking processing device 30 executes image processing for tracking feature points described later (hereinafter sometimes referred to as “tracking image processing”) on the image data converted by the signal conversion device 20a and the signal conversion device 20b. It is a device to do. The tracking processing device 30 includes an FPGA (Field Programmable Gate Array) 31, a CPU (Central Processing Unit) 32, a ROM (Read Only Memory) 33, a RAM (Random Access Memory) 34, and an I / F (Inter) (35). And a speaker 36 and a bus line 39.

  The FPGA 31 is an integrated circuit that performs processing for calculating a parallax value Δ in an image based on image data, tracking image processing, and the like. The CPU 32 controls each function of the image processing apparatus 1. The ROM 33 stores an image processing program executed by the CPU 32 to control each function of the image processing apparatus 1. The RAM 34 is used as a work area for the CPU 32. The I / F 35 is an interface for communicating with an external device or the like, and is an interface such as a CAN (Controller Area Network), for example. The speaker 36 is a device that outputs a sound such as a warning sound according to a command from the FPGA 31 or the CPU 32. As shown in FIG. 3, the bus line 39 is an address bus and a data bus that electrically connect the FPGA 31, the CPU 32, the ROM 33, the RAM 34, the I / F 35, and the speaker 36 to each other.

  Further, the above-described image processing program may be recorded in a computer-readable recording medium and distributed as a file in an installable format or an executable format. This recording medium is a CD-ROM (Compact Disc Read Only Memory) or an SD memory card (Secure Digital memory card).

(Block configuration of image processing apparatus)
FIG. 4 is a diagram illustrating an example of a block configuration of the image processing apparatus according to the first embodiment. The block configuration of the image processing apparatus 1 will be described with reference to FIG.

  As shown in FIG. 4, the image processing apparatus 1 includes an image acquisition unit 110, a correction unit 120, a distance measurement unit 130, a feature point extraction unit 140 (extraction means), a grouping unit 150, and a feature point tracking unit. 160 and a warning unit 170.

  The image acquisition unit 110 is a functional unit that captures a subject in front by two cameras on the left and right sides, generates an analog image signal, and obtains two luminance images that are images based on the image signals. The image acquisition unit 110 is realized by the imaging devices 10a and 10b illustrated in FIG.

  The correction unit 120 has a function of removing noise and distortion from the image data of the two luminance images obtained by the image acquisition unit 110 by correction using a filter process or the like, converting the image data into digital image data, and outputting the image data. Part. Here, image data captured by the right camera (imaging device 10a) of the image acquisition unit 110 among the image data of the two luminance images output from the correction unit 120 (hereinafter simply referred to as “luminance image”). The image data of the reference image Ia (hereinafter simply referred to as “reference image Ia”), and the image data captured by the left camera (imaging device 10b) is the image data of the comparison image Ib (hereinafter simply referred to as “comparison image Ib”). ). That is, the correction unit 120 outputs the reference image Ia and the comparison image Ib based on the two luminance images output from the image acquisition unit 110. The correction unit 120 is realized by the signal conversion devices 20a and 20b illustrated in FIG.

  The distance measurement unit 130 performs an image for each subject (object) included in the comparison image Ib (or the reference image Ia) based on the parallax based on the reference image Ia and the comparison image Ib obtained from the correction unit 120 by the stereo matching operation. It is a functional unit that measures the distance from the processing apparatus 1 to each subject. In principle, the distance measuring operation by the distance measuring unit 130 is always performed while the tracking image processing by the image processing apparatus 1 is being performed. Details of the configuration and operation of the distance measuring unit 130 will be described later. The distance measuring unit 130 is realized by the FPGA 31 illustrated in FIG.

  The feature point extraction unit 140 is a functional unit that extracts points (feature points) on an image that are remarkably recognized, such as the corners of an object imaged by the imaging devices 10a and 10b. In principle, the feature point extraction operation by the feature point extraction unit 140 is always performed while the tracking image processing by the image processing apparatus 1 is being performed. This is because, in the vehicle 100 or the like that is a moving body on which the image processing device 1 is mounted, new feature points always appear in images captured by the imaging devices 10a and 10b. As a feature point extraction method by the feature point extraction unit 140, for example, a Harris corner detection algorithm can be applied. The feature point extraction unit 140 is realized by the FPGA 31 illustrated in FIG.

  In the feature point tracking operation by the feature point tracking unit 160, the grouping unit 150 includes a plurality of points in order to identify a point to be replaced with the lost feature point (hereinafter referred to as a replacement point) when the feature point is lost. It is a functional unit that groups feature points. As described above, since new feature points always appear in the images captured by the imaging devices 10a and 10b, the grouping operation by the grouping unit 150 is performed by tracking image processing by the image processing device 1. In principle, it needs to be executed at all times. The grouping operation by the grouping unit 150 will be described later. The grouping unit 150 is realized by the FPGA 31 illustrated in FIG.

  Based on the distance information measured by the distance measuring unit 130, the feature point coordinates extracted by the feature point extracting unit 140, and the feature point grouping information by the grouping unit 150, the feature point tracking unit 160 It is a functional unit that executes a tracking operation. Details of the configuration and operation of the feature point tracking unit 160 will be described later. The feature point tracking unit 160 is realized by the FPGA 31 shown in FIG.

  The warning unit 170 is a functional unit that issues a warning when the feature point tracking unit 160 receives tracking information indicating failure of tracking a feature point. Specifically, the warning unit 170 outputs a warning sound as a warning when, for example, tracking information indicating failure to track a feature point is received. The warning unit 170 is realized by the speaker 36 shown in FIG. The warning unit 170 is not limited to being realized by the speaker 36, and may be realized by, for example, a display device such as a warning lamp or a warning message.

  Hereinafter, the feature point extraction operation by the feature point extraction unit 140, the grouping operation by the grouping unit 150, and the feature point tracking operation by the feature point tracking unit 160 are imaged by the left camera (imaging device 10b). The description will be made assuming that the processing is performed on the frame of the video data that has been processed. However, it goes without saying that it can also be executed on a frame of video data imaged by the right camera (imaging device 10a).

  FIG. 5 is a diagram illustrating an example of a block configuration of a distance measurement unit of the image processing apparatus according to the first embodiment. A block configuration of the distance measuring unit 130 of the image processing apparatus 1 will be described with reference to FIG.

  As illustrated in FIG. 5, the distance measurement unit 130 includes a cost calculation unit 131, a cost synthesis unit 132, and a distance derivation unit 133.

  The cost calculator 131 calculates the luminance value of the reference pixel p (x, y) (x and y are coordinates on the image) in the reference image Ia and the epipolar line in the comparison image Ib based on the reference pixel p (x, y). Thus, based on each luminance value of the candidate pixel q (x + d, y), which is a candidate for the corresponding pixel, specified by shifting by the shift amount d from the pixel corresponding to the position of the reference pixel p (x, y). , A functional unit that calculates a cost value C (p, d) indicating the dissimilarity of each candidate pixel q (x + d, y). As the cost value C calculated by the cost calculation unit 131, for example, SAD (Sum of Absolute Differences), SSD (Sum of Squared Differences), or the like can be used. Details of the operation of the cost calculation unit 131 will be described later.

  The cost calculation unit 131 uses the cost calculation unit 131 to calculate the cost value C of the pixel in the comparison image Ib for the reference pixel when the pixel around the reference pixel p (x, y) in the reference image Ia is the reference pixel. This is a functional unit that aggregates the calculated cost value C (p + d, y) of the candidate pixel q (x + d, y) and calculates the combined cost value Ls (p, d) of the candidate pixel q (x + d, y). . Details of the operation of the cost composition unit 132 will be described later.

  The distance deriving unit 133 specifies the shift amount d corresponding to the minimum value of the combined cost value Ls of the pixels in the comparison image Ib for the reference pixel in the reference image Ia, calculated by the cost combining unit 132, as the parallax value Δ. Part. Furthermore, the distance deriving unit 133 derives a distance from the specified parallax value Δ. Details of the operation of the distance deriving unit 133 will be described later. Note that the parallax value Δ identified as described above is a value in units of pixels, but the parallax value Δ may be obtained in units smaller than pixels (subpixel units) by subpixel estimation, for example. As the subpixel estimation, a parabola fitting method or a least square method can be applied.

  FIG. 6 is a diagram illustrating an example of a block configuration of a feature point tracking unit of the image processing apparatus according to the first embodiment. The block configuration of the feature point tracking unit 160 of the image processing apparatus 1 will be described with reference to FIG.

  As shown in FIG. 6, the feature point tracking unit 160 includes a matching unit 161 (matching unit), a tracking determination unit 162 (determination unit), a coordinate prediction unit 163 (prediction unit), and a return unit 164 (return unit). And having.

  The matching unit 161 has a predetermined size (N pixels) centered on the coordinates of the feature points extracted in a specific frame (hereinafter sometimes simply referred to as “nth frame”) by the feature point extraction unit 140. A predetermined search range (W pixels × H pixels) in the next frame (hereinafter, sometimes simply referred to as “(n + 1) th frame”) is an image that matches the template that is an image in the range of × N pixels. ) Is a functional unit that executes a block matching operation to be searched. Here, the frame indicates each image that constitutes video data captured by the imaging devices 10a and 10b. Details of the operation of the matching unit 161 will be described later. The template is N pixels × N pixels, but is not limited to this, and may be an image having a different number of pixels in the vertical and horizontal directions, or may be a template such as a circle instead of a rectangle. Further, the matching operation by the matching unit 161 may be executed using an image obtained by differentiating the pixel value of the image vertically and horizontally.

  As a result of the block matching operation by the matching unit 161, the tracking determination unit 162 generates (n + 1) templates including feature points to be tracked in the nth frame (hereinafter sometimes referred to as “tracking points”). This is a functional unit that determines whether or not the search has been performed within the search range in the eye frame, that is, whether or not the feature point has been tracked in the (n + 1) th frame. The tracking determination unit 162 determines whether or not the tracking point has been tracked by determining whether or not occlusion has occurred for the tracking point, for example. Details of the operation of the tracking determination unit 162 will be described later.

  When the tracking determination unit 162 determines that the tracking point cannot be tracked, that is, when it is determined that the tracking point has disappeared, the coordinate prediction unit 163 is the same as the feature point that is the tracking point that has disappeared by the grouping unit 150 This is a functional unit that identifies other feature points grouped in a group as alternative points, and uses the alternative points to predict the movement of a tracked point that has disappeared. Specifically, the coordinate predicting unit 163 predicts a point where the disappeared tracking point exists using a motion vector (optical flow) of the specified alternative point. Details of the operation of the coordinate prediction unit 163 will be described later.

  The return unit 164 forms a search area around a point (predicted point) where the tracking point disappeared by the coordinate predicting unit 163 is predicted, and forms a template of the tracking point before the tracking point disappears. This is a functional unit that performs tracking point return operation by comparing the search area. Details of the operation of the return unit 164 will be described later.

  4 to 6 conceptually show functions, and are not limited to such a configuration. For example, a plurality of functional units illustrated as independent functional units in FIGS. 4 to 6 may be configured as one functional unit. On the other hand, it is also possible to divide the functions of one functional unit shown in FIGS.

  The functional units shown in FIGS. 4 to 6 are not limited to being configured by hardware circuits such as the FPGA 31, but at least a part of the functional units is executed by the CPU 32 shown in FIG. It is good also as what is realized by.

(Object distance measurement operation)
The distance measuring operation using the SGM (Semi-Global Matching) method by the distance measuring unit 130 will be described with reference to FIGS.

<Principles of ranging>
FIG. 7 is a diagram for explaining the principle of deriving the distance from the imaging device to the object. With reference to FIG. 7, the principle of deriving a parallax with respect to an object from a stereo camera by a stereo matching operation and measuring the distance from the stereo camera to the object with a parallax value indicating the parallax will be described. In the following, in order to simplify the description, an example of matching in units of pixels will be described instead of matching a predetermined region composed of a plurality of pixels. In addition, when processing is performed in units of a predetermined area including a plurality of pixels instead of a single pixel, the predetermined area including the reference pixel is indicated as a reference area, and the predetermined area including the corresponding pixel is indicated as a corresponding area. Further, this reference area includes the case of only the reference pixel, and the corresponding area includes the case of only the corresponding pixel.

  As shown in FIG. 7, the imaging devices 10a and 10b are installed in parallel equiposition. In FIG. 7, the point S on the object E in the three-dimensional space is mapped to a position on a straight line parallel to a straight line connecting the imaging lens 11a and the imaging lens 11b in each of the reference image Ia and the comparison image Ib. Here, the point S mapped to each image is a point Sa (x, y) in the reference image Ia and a point Sb (X, y) in the comparison image Ib. At this time, the parallax value Δ is expressed by the following (Expression 1) using the coordinates of the point Sa (x, y) on the reference image Ia and the coordinates of the point Sb (X, y) on the comparison image Ib. It is expressed as follows.

  Δ = X−x (Formula 1)

  In FIG. 7, the distance between the point Sa (x, y) in the reference image Ia and the intersection of the perpendicular line taken from the imaging lens 11a on the imaging surface is Δa, and the point Sb (X, y) in the comparative image Ib is When the distance from the intersection of the perpendicular line taken from the imaging lens 11b on the imaging surface is Δb, the parallax value Δ is expressed as Δ = Δa + Δb.

  Next, the distance Z between the imaging devices 10a and 10b and the object E is derived by using the parallax value Δ. Here, the distance Z is a distance from a straight line connecting the focal position of the imaging lens 11a and the focal position of the imaging lens 11b to the point S on the object E. As shown in FIG. 7, using the focal length f of the imaging lens 11a and the imaging lens 11b, the baseline length B that is the length between the imaging lens 11a and the imaging lens 11b, and the parallax value Δ, According to 2), the distance Z can be calculated.

  Z = (B × f) / Δ (Expression 2)

  As shown in (Equation 2), the larger the parallax value Δ, the smaller the distance Z, and the smaller the parallax value Δ, the larger the distance Z.

  FIG. 8 is a conceptual diagram showing a reference image, a high-density parallax image, and an edge parallax image. 8A shows a reference image, FIG. 8B shows a dense parallax image with respect to the reference image shown in FIG. 8A, and FIG. 8C shows FIG. 8A. It is a conceptual diagram which shows the edge parallax image with respect to the reference | standard image shown in FIG. Here, the reference image corresponds to the reference image Ia shown in FIG. 7, and is an image in which the captured subject is represented by a luminance value. Further, the high-density parallax image indicates an image in which each pixel of the reference image is represented by a parallax value corresponding to each pixel in the reference image derived by the SGM method. And an edge parallax image shall show the image which represented each pixel of the reference | standard image by the parallax value corresponding to each pixel in the reference | standard image derived | led-out by the block matching method. However, as will be described later, the disparity value that can be derived by the block matching method is a relatively textured portion such as an edge portion in the reference image, and when the disparity value cannot be derived such as a weak texture portion, for example, The image is configured with a parallax value of “0”.

  The SGM method is a method for appropriately deriving a parallax value even for a portion having a weak texture in an image. Based on the reference image shown in FIG. 8A, the high-density parallax image shown in FIG. It is a method of deriving. The block matching method is a method for deriving the edge parallax image shown in FIG. 8C based on the reference image shown in FIG. According to the SGM method, as can be seen by comparing the inside of the dashed ellipse in FIGS. 8B and 8C, the high-density parallax image is detailed even on roads and the like where the texture is weaker than the edge parallax image. Since the distance based on the parallax value can be expressed, more detailed distance measurement can be performed.

  In the SGM method, a cost value that is a dissimilarity on a comparison image with respect to a reference image is not calculated and a parallax value is not derived immediately. After calculating a cost value, a disparity value is calculated by calculating a combined cost value. It is a method of deriving. The SGM method finally derives a parallax image (here, a high-density parallax image) represented by parallax values corresponding to almost all pixels in the reference image.

  On the other hand, in the case of the block matching method, the cost value is calculated in the same way as the SGM method. However, as in the SGM method, a comparatively strong texture such as an edge portion is calculated without calculating the synthesis cost value. Only the parallax value is derived.

  As described above, the image captured by the right camera (imaging device 10a) is the reference image Ia, and the image captured by the left camera (imaging device 10b) is the comparison image Ib. However, the present invention is not limited to this. Is not to be done. That is, an image captured by the left camera may be used as the comparison image Ib, and an image captured by the right camera may be used as the reference image Ia.

<Calculation of cost value>
FIG. 9 is an explanatory diagram for obtaining a corresponding pixel in a comparison image corresponding to a reference pixel in a reference image. FIG. 10 is a diagram illustrating an example of a graph indicating the relationship between the shift amount and the cost value. A method for calculating the cost value C (p, d) will be described with reference to FIGS. In the following description, C (p, d) is described as representing C (x, y, d).

  9A is a conceptual diagram showing reference pixels in the reference image, and FIG. 9B is a corresponding pixel candidate in the comparison image corresponding to the reference pixel shown in FIG. 9A. It is a conceptual diagram at the time of calculating a cost value while shifting (shifting) sequentially. Here, the corresponding pixel indicates a pixel in the comparison image that is most similar to the reference pixel in the reference image.

  As shown in FIG. 9A, the cost calculation unit 131 of the distance measurement unit 130 includes the reference pixel p (x, y) in the reference image Ia and the epipolar in the comparison image Ib with respect to the reference pixel p (x, y). The cost of the candidate pixel q (x + d, y) of the corresponding pixel with respect to the reference pixel p (x, y) based on each luminance value of the candidate pixel q (x + d, y) that is a candidate pixel of the corresponding pixel on the line EL A value C (p, d) is calculated. d is a shift amount (shift amount) between the reference pixel p and the candidate pixel q, and the shift amount d is shifted in units of pixels. That is, in FIG. 9, the candidate pixel q (x + d, y) and the reference pixel p are sequentially shifted by one pixel within a predetermined range (for example, 0 <d <25). A cost value C (p, d) that is a dissimilarity with (x, y) is calculated by the cost calculation unit 131.

  As described above, since the imaging devices 10a and 10b are arranged in parallel equiposition, the reference image Ia and the comparison image Ib are also in parallel equivalence relations. Therefore, the corresponding pixel in the comparison image Ib corresponding to the reference pixel p in the reference image Ia exists on the epipolar line EL shown as the horizontal line in FIG. 9, and the corresponding pixel in the comparison image Ib is obtained. For this purpose, a pixel on the epipolar line EL of the comparison image Ib may be searched.

  The cost value C (p, d) calculated in this way is represented by the graph shown in FIG. 10 in relation to the shift amount d in pixel units. In the example of FIG. 10, the cost value C is “0” when the shift amount d = 5, 12, and 19, and thus the minimum value cannot be obtained. Thus, for example, when there is a portion where the texture is weak in the image, it is difficult to obtain the minimum value of the cost value C in this way.

<Calculation of synthesis cost value and derivation of distance>
FIG. 11 is a conceptual diagram for deriving the synthesis cost. FIG. 12 is a diagram illustrating an example of a graph indicating the relationship between the shift amount and the combined cost value. A method for calculating the combined cost value Ls (p, d) will be described with reference to FIGS.

  The calculation method of the synthesis cost value according to the present embodiment not only calculates the cost value C (p, d), but also calculates the cost value when pixels around the reference pixel p (x, y) are the reference pixels. The combined cost value Ls (p, d) is calculated by consolidating the cost value C (p, d) in the reference pixel p (x, y). In order to calculate the combined cost value Ls (p, d), the cost combining unit 132 of the distance measuring unit 130 first calculates the route cost value Lr (p, d). The cost synthesizing unit 132 calculates the route cost value Lr (p, d) by the following (Formula 3).

Lr (p, d) = C (p, d) + min (Lr (p−r, k) + P (d, k))
... (Formula 3)
(P = 0 (when d = k))
P = P1 (when | d−k | = 1),
P = P2 (> P1) (when | dk |> 1))

  As shown in (Formula 3), the route cost value Lr is obtained recursively. Here, r indicates a direction vector in the aggregation direction, and has two components in the x direction and the y direction. min () is a function for obtaining the minimum value. Lr (p−r, k) is the respective path when the shift amount is changed (the shift amount in this case is k) for the pixel having the coordinate shifted by one pixel in the r direction from the coordinate of the reference pixel p. The cost value Lr is shown. Based on the relationship between d, which is the amount of shift of the route cost value Lr (p, d) in the route cost value Lr (pr, k), and the shift amount k, the following (1) The value P (d, k) is obtained as in (3), and Lr (pr, k) + P (d, k) is calculated.

(1) When d = k, P = 0. That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k).
(2) When | d−k | = 1, P = P1. That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k) + P1.
(3) When | d−k |> 1, P = P2 (> P1). That is, Lr (p−r, k) + P (d, k) = Lr (p−r, k) + P2.

  Then, min (Lr (p−r, k) + P (d, k)) is Lr (p−r) calculated in the above (1) to (3) when k is changed to various values. , K) + P (d, k) is a value obtained by extracting the minimum value. That is, the value P1 or the value P1 is exceeded when the pixel located at the coordinate position of the reference pixel p in the comparison image Ib is away from the pixel shifted by the shift amount d from the pixel (pr) adjacent in the r direction. By adding the large value P2, the shift amount d of the pixel away from the coordinate of the reference pixel p in the comparison image Ib is prevented from being influenced by the discontinuous path cost value Lr. Further, the value P1 and the value P2 are fixed parameters determined in advance by experiments, and are parameters such that the parallax values of the reference pixels adjacent on the path are likely to be continuous. For example, P1 = 48 and P2 = 96. As described above, in order to obtain the path cost value Lr for each pixel in the r direction in the comparison image Ib, the cost composition unit 132 first determines the extreme end in the r direction from the coordinates of the reference pixel p (x, y). The route cost value Lr is obtained from the pixel, and the route cost value Lr is obtained along the r direction.

Then, as shown in FIG. 11, the cost composition unit 132 has Lr which is a route cost value Lr in eight directions (r ₀ , r ₄₅ , r ₉₀ , r ₁₃₅ , r ₁₈₀ , r ₂₂₅ , r ₂₇₀ and r ₃₁₅ ). ₀ , Lr ₄₅ , Lr ₉₀ , Lr ₁₃₅ , Lr ₁₈₀ , Lr ₂₂₅ , Lr ₂₇₀ and Lr ₃₁₅ are obtained, and finally a synthesis cost value Ls (p, d) is obtained based on (Equation 4) below. .
As described above, the synthesis cost value Ls (p, d) calculated by the cost synthesis unit 132 can be represented by the graph shown in FIG. 12 in relation to the shift amount d in pixel units.

  Then, the distance deriving unit 133 of the distance measuring unit 130 calculates the shift amount d corresponding to the minimum value of the combined cost value Ls of the pixels in the comparison image Ib for the reference pixel in the reference image Ia calculated by the cost combining unit 132. It is specified as a parallax value Δ. In the example of FIG. 12, the combined cost value Ls is derived as a parallax value Δ = 3 because the minimum value is obtained when the shift amount d = 3. Further, the distance deriving unit 133 calculates the distance Z by the above-described equation (2) using the specified parallax value Δ.

  Note that the distance measurement by the distance measurement unit 130 is not performed on the entire image, but on the pixels corresponding to the coordinates of the feature points extracted by the feature point extraction unit 140, and only the distances of the feature points are measured. It is good also as what measures.

  In the above description, the number in the r direction is eight, but the present invention is not limited to this. For example, the eight directions may be further divided into two to be 16 directions, or may be divided into three to be 24 directions. Or it is good also as what calculates | requires the path | route cost value Lr in any of eight directions, and calculates the synthetic | combination cost value Ls.

  Further, although the cost value C is a value indicating the dissimilarity, the cost value C is not limited thereto, and may be represented by an evaluation value of the similarity. In this case, the shift amount d at which the combined cost value Ls is not the minimum value but the maximum value is derived as the parallax value Δ. Further, both dissimilarity and similarity may be referred to as “matching degree”.

  In addition, as described above, the distance measurement unit 130 performs distance measurement using the SGM method. However, the distance measurement unit 130 is not limited to this, and performs distance measurement by a conventional block matching method. It may be a thing.

(Details of tracking image processing)
Details of the tracking image processing by the image processing apparatus 1 according to the present embodiment will be described with reference to FIGS.

<About success or failure of feature point tracking>
FIG. 13 is a diagram illustrating an example of a feature point tracking failure and success state. With reference to FIG. 13, an example of the success or failure of the tracking of feature points extracted by the feature point extraction unit 140 will be described.

  As shown in FIG. 13A, the feature point 300 extracted by the feature point extraction unit 140 is shown in the frame 200 which is the nth frame. The feature point 300 is also reflected in the frame 201 which is the frame after the frame 200 (the (n + 1) th frame). In this case, since the feature point 300 is shown from the frame 200 to the frame 201, the feature point tracking unit 160 can track the feature point 300 in principle.

  On the other hand, as shown in FIG. 13B, the feature point 300 extracted by the feature point extraction unit 140 is shown in the frame 200, but in the frame 201 a that is the next frame, the wiper 900 has the feature point 300. The feature point 300 is not captured by blocking. In this case, since the feature point 300 disappears from the frame 200 to the frame 201a, the feature point tracking unit 160 cannot temporarily track the feature point 300.

<Block matching operation and tracking point disappearance determination operation>
FIG. 14 is a diagram for explaining the block matching operation. FIG. 15 is a diagram for explaining processing in the search range in the block matching operation. FIG. 16 is a diagram for explaining an occlusion determination operation. Details of the block matching operation by the matching unit 161 of the feature point tracking unit 160 and the tracking point disappearance determination operation by the tracking determination unit 162 will be described with reference to FIGS.

  As shown in FIG. 14A, the feature point extraction unit 140 has a predetermined size centered on the coordinates of the extracted feature point (for example, the feature point 310) in the frame 210, which is the nth frame. A template (template 610 shown in FIG. 14B) (first template image) in the range of N pixels × N pixels is specified, and the image of the template 610 is temporarily stored in the storage unit (for example, the RAM 34).

  Next, as illustrated in FIG. 14C, the matching unit 161 corresponds to the coordinates of the feature point 310 of the frame 210 in the frame 211 that is the next frame (the (n + 1) th frame) of the frame 210. A predetermined search range 810 (W pixels × H pixels) (first search range) is formed around the corresponding point 410 that is a point. Then, as shown in FIG. 15A, the matching unit 161 raster-scans the search area 710 having the same size as the template 610 temporarily stored in the storage unit in the search range 810, and the template 610 and the search area A block matching operation with the image 710 is performed. Specifically, the matching unit 161 raster scans the search area 710 in the search range 810 while calculating the degree of coincidence between the template 610 and the image of the search area 710. As the degree of coincidence calculated by the matching unit 161, for example, SAD or SSD can be used.

  Then, the tracking determination unit 162 performs threshold determination on the degree of coincidence calculated by the matching unit 161. For example, if the degree of coincidence is a value indicating dissimilarity such as SAD, the tracking determination unit 162, when the degree of coincidence is a predetermined threshold value or less, the image of the search area 710 for which the degree of coincidence is calculated, It is determined that the template 610 matches and it is determined that tracking is successful. On the other hand, when the matching degree is larger than a predetermined threshold in the entire search range 810, the tracking determination unit 162 does not match the image of the search region 710 and the template 610, and the feature point 310 cannot be tracked. That is, it is determined that the feature point 310 has disappeared in the frame 211.

  When the tracking determination unit 162 succeeds in tracking, tracking is a result of tracking the feature point 310 at the center point of the image of the search region 711 that is the search region 710 that matches the template 610 shown in FIG. The point 311 is specified. Then, the matching unit 161 and the tracking determination unit 162 further repeat tracking in the subsequent frames with respect to the tracking point 311 (feature point 310) specified in the frame 211 that is the (n + 1) th frame. Further, the tracking determination unit 162 temporarily stores an image of the search area 711 centered on the coordinates of the tracking point 311 as a new template in a storage unit (for example, the RAM 34). Then, the tracking determination unit 162 stores information of the optical flow 510 that is a vector from the corresponding point 410 to the tracking point 311 in a storage unit (for example, an external storage device such as a HDD (Hard Disk Drive)).

  In addition, as an example of the cause that the tracking determination unit 162 cannot track the feature point, that is, determines that the feature point has disappeared, occlusion occurs due to the feature point being blocked by the object. For example, as illustrated in FIG. 16A, it is assumed that feature points 320 and 321 are extracted as feature points of the object 902 that is the subject in the frame 220 by the feature point extraction unit 140. Then, as shown in FIG. 16B, in the frame 221 that is the next frame of the frame 220, the occlusion caused by the feature point 320 being blocked by the wiper 900 moving on the windshield 901 of the vehicle 100 (see FIG. 1). Suppose that occurs. In this case, as a result of the block matching operation by the matching unit 161, the tracking determination unit 162 has a feature in a search range 820 (first search range) centered on a corresponding point 420 that is a corresponding point in the frame 221 of the feature point 320. An image in the search area that matches the template of the point 320 cannot be searched, and it is determined that the feature point 320 has disappeared. Further, for example, the tracking determination unit 162 uses the distance measurement unit 130 to calculate distance information of the subject included in the search range 820 centered on the corresponding point 420 that is a point corresponding to the coordinates of the feature point 320 of the frame 220 in the frame 221. If the search range 820 determines that an object (for example, the wiper 900) having a distance shorter than a predetermined distance is included in the search range 820 based on the distance information, it is determined that occlusion has occurred, and the frame 221 is characterized. The point 320 may be determined as having disappeared.

  Note that, in the matching operation by the matching unit 161, the size of the search area 710 may change according to the state of the vehicle 100. For example, the size of the search area 710 may be adjusted based on information such as the traveling speed of the vehicle 100, the steering angle of the vehicle 100, or the brightness of the external environment.

<Grouping operation>
17 and 18 are diagrams for explaining the grouping operation for estimating the prediction point in the image processing apparatus according to the first embodiment. Details of the grouping operation of the feature point extraction unit 140 of the image processing apparatus 1 will be described with reference to FIGS. 17 and 18.

  As described above, the grouping unit 150 specifies an alternative point used for obtaining a predicted point of a lost feature point when the feature point disappears in the feature point tracking operation by the feature point tracking unit 160. It is necessary to group a plurality of feature points that are assumed to move along a similar locus. As a grouping method, the grouping unit 150 groups, for example, feature points having the same distance from the image processing apparatus 1 among the feature points extracted by the feature point extraction unit 140. In this case, the grouping unit 150 determines whether or not the distance from the image processing apparatus 1 to the feature point is the same based on the distance information received from the distance measurement unit 130. Here, the same distance does not assume the case where the distances are completely the same, but includes the case where the distances are approximate to the extent that they can be regarded as the same. In this case, for example, the grouping unit 150 determines whether the special distance from the image processing apparatus 1 is the same for the two feature points, whether the difference in distance to each feature point is equal to or less than a predetermined threshold value. It may be determined based on whether or not.

  For example, in FIG. 17A, an image captured by the left camera is a frame 225, and an image captured by the right camera is a frame 226. As shown in FIGS. 17A and 17B, the feature point extraction unit 140 extracts feature points 320 and 321 in the object 902 shown in the frame 225. The distance measuring unit 130 measures the distance between the feature points 320 and 321 based on the parallax value Δ1 shown in the image 227 obtained by superimposing the frame 225 and the frame 226 and sends the distance information to the grouping unit 150. Then, when the grouping unit 150 determines that both are the same distance from the distance information of the feature points 320 and 321 received from the distance measurement unit 130, the grouping unit 150 groups them into the same group. The grouping unit 150 sends information (group information) obtained by grouping the feature points extracted by the feature point extraction unit 140 to the feature point tracking unit 160.

  In the example illustrated in FIG. 18, the grouping method by the grouping unit 150 is performed as follows, for example. As shown in FIG. 18A, the left and right cameras of the image processing apparatus 1 respectively capture the objects 910 and 911, and the frame 230 illustrated in FIG. 18B is captured by the left camera (imaging apparatus 10b). Frame. First, it is assumed that the feature point extraction unit 140 extracts the feature points 330 and 331 of the object 910 and the feature points 332 and 333 of the object 911 shown in the frame 230 as shown in FIG. Further, the distance Z1 to the object 910 measured by the distance measuring unit 130 (that is, the distance to the feature points 330 and 331) and the distance Z2 to the object 911 (that is, the distance to the feature points 332 and 333) are Are the same. Here, when the vehicle 100 on which the image processing apparatus 1 is mounted passes between the object 910 and the object 911, the feature points 330 and 331 are on the left side from the center of the image in the image captured by the imaging device 10b. In the direction toward, the feature points 332 and 333 are directed in the direction from the center of the video toward the right side.

  In this case, as in the example of FIG. 17, when feature points having the same distance from the image processing device 1 are grouped, the feature points 330 to 333 are all the same distance from the image processing device 1 by the grouping unit 150. It will be grouped into the same group. However, as described above, since the feature points 330 and 331 and the feature points 332 and 333 move along different trajectories, the feature point tracking operation performed by the feature point tracking unit 160 obtains a predicted point of the lost feature point. Therefore, there is a possibility that a feature point having a different locus is specified as an alternative point used for the purpose, and the estimation operation of the prediction point by the coordinate prediction unit 163 and the return operation of the tracking point by the return unit 164 may fail. . Therefore, for example, the grouping unit 150 may perform grouping by dividing the frame 230 into a left part and a right part from the center. Specifically, in FIG. 18B, the grouping unit 150 groups the feature points 330 and feature points 331 on the left side from the center of the frame 230 into the same group, and the feature points 332 and feature points on the right side. 333 are grouped into a group different from the group of feature points 330 and 331.

  Note that the grouping method by the grouping unit 150 is not limited to the method described above with reference to FIGS. 17 and 18, and grouping a plurality of feature points that are assumed to move along a similar locus by other methods. It is good. In the example of FIG. 18, the frame is divided into a left part and a right part from the center of the frame. For example, the frame is divided into the upper left, lower left, upper right, and lower right from the center of the frame. The feature points may be grouped. Further, for example, when the vehicle 100 does not travel forward but bends at a curve or an intersection, all feature points in the frame are assumed to move to the right or left. It is good also as what groups the feature point with the same distance in the whole flame | frame, without dividing.

<Prediction point estimation operation>
FIG. 19 is a diagram for explaining a prediction point estimation operation when occlusion occurs in the image processing apparatus according to the first embodiment. With reference to FIG. 19, a prediction point estimation operation by the coordinate prediction unit 163 of the feature point tracking unit 160 will be described.

  As shown in FIG. 19A, first, it is assumed that feature points 340 and 341 in the object 940 are extracted by the feature point extraction unit 140 in the frame 240 that is the nth frame. Further, it is assumed that the feature point extraction unit 140 groups the feature points 340 and the feature points 341 into the same group. In addition, the wiper 900 that rotates and moves from left to right is shown in the frame 240.

  Then, as shown in FIG. 19B, in the frame 241 that is the next frame ((n + 1) th frame) of the frame 240, the feature point that is tracked by the wiper 900 that has been rotated from right to left. Assume that occlusion occurs due to 340 being blocked. In this case, as described above, the tracking determination unit 162 cannot track the feature point 340 and determines that the feature point 340 has disappeared. As described above, when the tracking determination unit 162 determines that the feature point 340 has disappeared, the coordinate prediction unit 163 predicts the movement of the feature point 340 that has disappeared. The closest feature point (feature point 341 in the example of FIG. 19B) is specified as an alternative point.

  The coordinate predicting unit 163 then erases the optical flow 541 that is a vector from the corresponding point 441 corresponding to the coordinate of the feature point 341 of the frame 240 to the coordinate of the feature point 341 in the frame 241. It is used as an optical flow 340 (optical flow 540). Specifically, the coordinate predicting unit 163 places the lost feature point 340 in the position indicated by the optical flow 540 from the corresponding point 440 that corresponds to the coordinate of the feature point 340 of the frame 240 in the frame 241. Predict that it is moving. That is, the coordinate predicting unit 163 obtains the coordinates of the point indicated by the optical flow 540 from the corresponding point 440 in the frame 241 as the coordinates of the predicted point 450 estimated that the missing feature point 340 exists.

  Further, as illustrated in FIG. 19C, the coordinate predicting unit 163 corresponds to the coordinates of the feature point 341 of the frame 241 in the frame 242 that is the next frame ((n + 2) th frame) of the frame 241. The optical flow 551 that is a vector from the corresponding point 442 that is a point to the coordinates of the feature point 341 is used as the next optical flow (optical flow 550) of the lost feature point 340. Specifically, the coordinate predicting unit 163 newly moves the lost feature point 340 from the prediction point 450 of the frame 241 (prediction point of the feature point 340) to the position indicated by the optical flow 550 in the frame 242. Predict that you are. That is, the coordinate predicting unit 163 obtains the coordinates of the point indicated by the optical flow 550 from the corresponding point 450 as the coordinates of the predicted point 451 newly estimated as the missing feature point 340 exists in the frame 242.

<Tracking point return operation>
FIG. 20 is a diagram for explaining the tracking point return operation in the image processing apparatus according to the first embodiment. The tracking point return operation by the return unit 164 of the feature point tracking unit 160 will be described with reference to FIG.

  FIG. 20A is the same as the state shown in FIG. 19A, and shows a state in which feature points 340 and 341 in the object 940 are extracted by the feature point extraction unit 140 in the frame 240, respectively. As described above, the feature point extraction unit 140 further specifies a template 640 having a predetermined size around the coordinates of the extracted feature point 340 in the frame 240, and temporarily stores the image of the template 640 in the storage unit. Let The template specifying and storing operations are performed in the same manner for the feature points 341.

  FIG. 20B is the same as the state shown in FIG. 19B, and in the frame 241, occlusion occurs when the feature point 340 to be tracked is blocked by the wiper 900 that has rotated from the right to the left. Shows the state. In this case, the tracking determination unit 162 cannot track the feature point 340 and determines that the feature point 340 has disappeared.

  After it is determined by the tracking determination unit 162 that the feature point 340 to be tracked has disappeared, as shown in FIG. 20C, the return unit 164 performs the frame 242 (next to the frame 241 in which the feature point 340 has disappeared). In the frame), a search area 740 having the same size as the template 640 is formed around the prediction point 451 estimated by the coordinate prediction unit 163. Then, the restoration unit 164 compares the template 640 stored in the storage unit with the image in the search area 740 and calculates the degree of coincidence thereof. As the degree of coincidence calculated by the return unit 164, for example, SAD or SSD can be used in the same manner as the matching unit 161.

  Then, the return unit 164 performs threshold determination on the calculated degree of coincidence. For example, if the degree of coincidence is a value indicating dissimilarity such as SAD, the restoration unit 164 determines that the template 640 matches the image in the search area 740 if the degree of coincidence is equal to or less than a predetermined threshold. The prediction point 451 determined by the coordinate prediction unit 163 is returned as a tracking point for the feature point 340. On the other hand, the restoration unit 164 determines that the template 640 does not match the image in the search area 740 when the degree of matching is greater than a predetermined threshold. In this case, the prediction point estimation operation by the coordinate prediction unit 163 and the tracking point return operation by the return unit 164 are continued. However, for example, the return unit 164 determines that the tracking of the lost feature point 340 has failed if the return operation is successful even after being performed a predetermined number of times.

<Flow of feature point tracking>
FIG. 21 is a diagram illustrating an example of an operation flow of image processing of the image processing apparatus according to the first embodiment. With reference to FIG. 21, the flow of the feature point tracking operation by the feature point tracking unit 160 in the tracking image processing of the image processing apparatus 1 will be generally described. Note that the grouping unit 150 assumes that the feature points extracted by the feature point extraction unit 140 are grouped in advance by the method described above.

<< Step S11 >>
The feature point extraction unit 140 specifies a template having a predetermined size centered on the coordinates of the extracted feature points in the nth frame, and temporarily stores the template image in the storage unit (for example, the RAM 34). Let The matching unit 161 forms a predetermined search range around the corresponding point that is the point corresponding to the coordinates of the feature point to be tracked in the nth frame in the next frame ((n + 1) th frame). To do. The matching unit 161 raster-scans a search area having the same size as the template temporarily stored in the storage unit in the search range, and executes a block matching operation between the template and the image in the search area. Specifically, the matching unit 161 raster-scans the search area in the search range while calculating the degree of coincidence between the template and the image of the search area. Then, the process proceeds to step S12.

<< Step S12 >>
The tracking determination unit 162 performs threshold determination for the degree of coincidence calculated by the matching unit 161. For example, if the degree of coincidence is a value indicating dissimilarity such as SAD, the tracking determination unit 162, when the degree of coincidence is equal to or less than a predetermined threshold, the image of the search region in which the degree of coincidence is calculated, the template Are matched, and it is determined that the tracking is successful (step S12: Yes). If the tracking determination unit 162 succeeds in tracking, the tracking determination unit 162 identifies the center point of the image in the search area that matches the template as a tracking point that is a result of tracking the feature point. Then, the matching unit 161 and the tracking determination unit 162 further repeat tracking in the subsequent frames for the identified tracking points.

  On the other hand, when the matching degree is larger than a predetermined threshold in the entire search range, the tracking determination unit 162 does not match the search region image and the template, and the feature point cannot be tracked. Is determined to have disappeared (step S12: No), and the process proceeds to step S13.

<< Step S13 >>
In order to predict the movement of the lost feature point, the coordinate predicting unit 163 specifies a feature point closest to the lost feature point in the same group as the lost feature point as an alternative point. Then, the coordinate predicting unit 163 calculates an optical flow that is a vector from the corresponding point that is a point corresponding to the coordinates of the substitute point of the nth frame to the coordinates of the substitute point in the (n + 1) th frame, It is used as an optical flow for missing feature points. Specifically, the coordinate predicting unit 163 determines whether the lost feature point is a point corresponding to the coordinates of the lost feature point in the nth frame in the (n + 1) th frame. Is predicted to have moved to the position indicated by the optical flow. That is, in the (n + 1) th frame, the coordinate predicting unit 163 uses the coordinates of the point indicated by the optical flow from the corresponding point of the lost feature point as the coordinate of the predicted point where the lost feature point is estimated to exist. Ask. Then, the process proceeds to step S14.

<< Step S14 >>
When the estimation of the coordinates of the predicted point is successful by the coordinate predicting unit 163 (step S14: Yes), the process proceeds to step S15, and when it is not successful (step S14: No), the process proceeds to step S19. Here, as an example of the case where the estimation of the coordinates of the predicted point is not successful, the coordinate predicting unit 163 did not specify an alternative point due to a reason that no other feature point exists in the group to which the lost feature point belongs. Cases.

<< Step S15 >>
The return unit 164 forms a search area having the same size as the template around the prediction point estimated by the coordinate prediction unit 163 in the (n + 2) th frame. Next, the restoration unit 164 compares the template stored in the storage unit with the image in the search area, and calculates the degree of coincidence thereof. Then, the return unit 164 performs threshold determination on the calculated degree of coincidence. Then, the process proceeds to step S16.

<< Step S16 >>
For example, if the degree of coincidence is a value indicating dissimilarity such as SAD, the restoration unit 164 determines that the template and the image in the search area match if the degree of coincidence is equal to or less than a predetermined threshold. The tracking point return is determined to be successful (step S16: Yes). In this case, the return unit 164 returns the prediction point estimated by the coordinate prediction unit 163 as a tracking point for the feature point.

  On the other hand, when the matching degree is greater than the predetermined threshold, the restoration unit 164 determines that the template does not match the image in the search area, and determines that the tracking point restoration has failed (step S16: No). ). Then, the process proceeds to step S17.

<< Step S17 >>
The return unit 164 determines whether or not the tracking point return operation has reached a predetermined number of times. If the predetermined number of times has been reached (step S17: Yes), the process proceeds to step S18, and if not reached (step S17: No), the process returns to step S13, and the operation is repeated from the predicted point estimation operation by the coordinate prediction unit 163. . The return unit 164 is not limited to returning to step S13 when the tracking point return operation has not reached the predetermined number of times. For example, the return unit 164 determines that the tracking point return has failed. Then, after a lapse of a predetermined time, the process may return to step S13.

<< Step S18 >>
The warning unit 170 issues a warning when the feature point tracking unit 160 receives the tracking information indicating the tracking failure of the feature point.

<< Step S19 >>
The warning unit 170 issues a warning when the coordinate prediction unit 163 fails to estimate the coordinates of the predicted point.

  The feature point tracking operation by the feature point tracking unit 160 is executed by the flow of steps S11 to S19 described above.

  As described above, when the matching unit 161 and the tracking determination unit 162 track the extracted feature points and the tracking determination unit 162 determines that the feature points have disappeared, the coordinate prediction unit 163 The feature points of the same group as are specified as alternative points. Then, the coordinate predicting unit 163 uses a point indicated by the optical flow of the alternative point from the corresponding point of the lost feature point as a predicted point, and the returning unit 164 includes the template of the feature point before the disappearance includes the predicted point. If the image matches the image in the search area, the predicted point is returned as a tracking point for the lost feature point. As a result, the lost feature points can be restored again, and it is not necessary to use past images (frames) to continue tracking, and there is no need to re-track the feature points. The processing load can be reduced. Further, when at least one of the feature point extraction unit 140, the grouping unit 150, and the feature point tracking unit 160 is realized by a hardware circuit, as described above, the feature point extraction unit 140, the grouping unit 150, and the feature point tracking unit 160 can be restored again. It is not necessary to use a video (frame), and there is no need to re-track feature points, so that an increase in circuit scale can be suppressed.

  In addition, when the tracking determination unit 162 determines that the feature point has disappeared, the coordinate prediction unit 163 desirably specifies the feature point closest to the disappearing feature point in the same group as the disappeared feature point as an alternative point. . As a result, since the lost feature point is assumed to move similar to the specified substitute point, there is a high possibility that the accuracy of the prediction of the prediction point by the coordinate prediction unit 163 is improved, and the tracking by the restoration unit 164 is performed. The point can be easily restored.

  In addition, even if the return by the return unit 164 fails, the estimation operation of the prediction point by the coordinate prediction unit 163 and the return operation of the tracking point by the return unit 164 are performed again. In this way, by repeating the estimation of the prediction point and the operation of returning the tracking point, the probability that the recovery of the tracking point is successful can be improved.

(Modification)
FIG. 22 is a diagram for explaining the tracking point return operation in the image processing apparatus according to the modification of the first embodiment. With reference to FIG. 22, the description will focus on differences from the tracking point return operation of the return unit 164 shown in FIG.

  FIGS. 22A and 22B show the same state as FIGS. 20A and 20B described above, respectively. In the frame 241, the wiper 900 that has been rotated from the right to the left is tracked. This shows a state in which occlusion occurs due to a certain feature point 340 being blocked. In this case, the tracking determination unit 162 cannot track the feature point 340 and determines that the feature point 340 has disappeared.

  After the tracking determination unit 162 determines that the feature point 340 to be tracked has disappeared, as shown in FIG. 22C, the restoration unit 164 performs the frame 242 (next to the frame 241 in which the feature point 340 has disappeared). Frame), a search range 840 (second search range) centered on the prediction point 451 is formed. Then, the restoration unit 164 performs a raster scan of the search area 740 having the same size as the template 640 (second template image) in the search range 840, and executes block matching between the template 640 and the image of the search area 740. Specifically, the restoration unit 164 performs a raster scan of the search area 740 in the search range 840 while calculating the degree of coincidence between the template 640 and the image of the search area 740.

  Then, the return unit 164 performs threshold determination on the calculated degree of coincidence. For example, if the degree of coincidence is a value indicating dissimilarity such as SAD, the restoration unit 164 determines that the template 640 matches the image in the search area 740 if the degree of coincidence is equal to or less than a predetermined threshold. The prediction point 451 determined by the coordinate prediction unit 163 is returned as a tracking point for the feature point 340. On the other hand, the restoration unit 164 determines that the template 640 does not match the image of the search area 740 when the matching degree is larger than a predetermined threshold in the entire search range 840. In this case, the prediction point estimation operation by the coordinate prediction unit 163 and the tracking point return operation by the return unit 164 are continued. However, for example, the return unit 164 determines that the tracking of the lost feature point 340 has failed if the return operation is successful even after being performed a predetermined number of times.

  As described above, the restoration unit 164 does not compare the template 640 with only the image of the search region 740 centered on the prediction point 451, but forms the search range 840 including the prediction point 451, Raster scanning is performed, and block matching between the template 640 and the image of the search area 740 is executed. As a result, an image that matches the template 640 is searched in the search range 840, so that there is a high possibility that an image that matches the template 640 will be found, and the probability of successful return of the tracking point can be improved.

[Second Embodiment]
The image processing apparatus 1a according to the present embodiment will be described focusing on differences from the image processing apparatus 1 according to the first embodiment. In the first embodiment, another feature point grouped by the grouping unit 150 is specified as an alternative point of the lost feature point. In this embodiment, the feature point tracking operation by the feature point tracking unit 160 is performed. The operation using the feature points in the video data of a camera different from the video data of the camera performing the above will be described.

(Block configuration of image processing apparatus)
FIG. 23 is a diagram illustrating an example of a block configuration of the image processing apparatus according to the second embodiment. With reference to FIG. 23, the block configuration of the image processing device 1 a will be described focusing on differences from the block configuration of the image processing device 1 according to the first embodiment.

  As shown in FIG. 23, the image processing apparatus 1a includes an image acquisition unit 110, a correction unit 120, a distance measurement unit 130, a feature point extraction unit 140, a specification unit 150a (specification means), and a feature point tracking unit. 160a and a warning unit 170. That is, the image processing device 1a includes a specifying unit 150a instead of the grouping unit 150 of the image processing device 1a according to the first embodiment. The operations of the image acquisition unit 110, the correction unit 120, the distance measurement unit 130, the feature point extraction unit 140, and the warning unit 170 are the same as those in the first embodiment. Similarly to the first embodiment, the feature point extraction operation by the feature point extraction unit 140 and the feature point tracking operation by the feature point tracking unit 160a are imaged by the left camera (imaging device 10b). A description will be given assuming that the process is performed on a frame of video data. However, it goes without saying that it can also be executed on a frame of video data imaged by the right camera (imaging device 10a).

  When the feature point disappears in the feature point tracking operation by the feature point tracking unit 160a, the specifying unit 150a uses the right camera (imaging device 10a) as an alternative point used to obtain a predicted point of the lost feature point. It is a functional part specified by the frame of the video data imaged by. The operation by the specifying unit 150a will be described later. The specifying unit 150a is realized by the FPGA 31 illustrated in FIG.

  The feature point tracking unit 160a is based on the distance information measured by the distance measuring unit 130, the coordinates of the feature point extracted by the feature point extracting unit 140, and the specific information about the alternative point specified by the specifying unit 150a. It is a functional unit that executes a feature point tracking operation. Details of the configuration and operation of the feature point tracking unit 160a will be described later. The feature point tracking unit 160a is realized by the FPGA 31 shown in FIG.

  FIG. 24 is a diagram illustrating an example of a block configuration of a feature point tracking unit of the image processing apparatus according to the second embodiment. Referring to FIG. 24, the block configuration of the feature point tracking unit 160a of the image processing device 1a is mainly different from the block configuration of the feature point tracking unit 160 of the image processing device 1 according to the first embodiment. explain.

  As shown in FIG. 24, the feature point tracking unit 160a includes a matching unit 161 (matching unit), a tracking determination unit 162 (determination unit), a coordinate prediction unit 163a (prediction unit), and a return unit 164 (return unit). And having. That is, the feature point tracking unit 160a has a coordinate prediction unit 163a instead of the coordinate prediction unit 163 of the feature point tracking unit 160 of the first embodiment. The operations of the matching unit 161, the tracking determination unit 162, and the return unit 164 are the same as those in the first embodiment.

  When the tracking determination unit 162 determines that the tracking point cannot be tracked, that is, when it is determined that the tracking point has disappeared, the coordinate prediction unit 163a specifies the feature point specified by the specifying unit 150a as an alternative point The function unit predicts the movement of the lost tracking point by using the alternative point. Specifically, the coordinate predicting unit 163a predicts a point where the disappeared tracking point exists by using a motion vector (optical flow) of the specified alternative point. Details of the operation of the coordinate predicting unit 163a will be described later.

  Each functional unit shown in FIGS. 23 and 24 conceptually shows a function, and is not limited to such a configuration. For example, a plurality of functional units illustrated as independent functional units in FIGS. 23 and 24 may be configured as one functional unit. On the other hand, it is also possible to divide the functions of one functional unit shown in FIGS. 23 and 24 into a plurality of functional units and configure them as a plurality of functional units.

(Identification of alternative points and estimation of prediction points)
FIG. 25 is a diagram for explaining a prediction point estimation operation when occlusion occurs in the image processing apparatus according to the second embodiment. With reference to FIG. 25, description will be made on the operation of specifying an alternative point of the specifying unit 150a of the image processing apparatus 1a and the operation of estimating the predicted point of the coordinate predicting unit 163a of the feature point tracking unit 160a.

  As shown in FIG. 25A, first, a feature point 360 in the object 960 is extracted by the feature point extraction unit 140 in a frame 260 that is the nth frame imaged by the left camera (imaging device 10b). It shall be. Similarly, as shown in FIG. 25B, in the frame 270 that is the n-th frame imaged by the right camera (imaging device 10a), the feature point 370 in the object 960 is obtained by the feature point extraction unit 140. Is extracted. As is clear from FIG. 25, the feature point 360 of the frame 260 is a point corresponding to the feature point 370 of the frame 270. The frames 260 and 270 show the wiper 900 that rotates and moves from left to right.

  As described above, when the feature point disappears in the feature point tracking operation by the feature point tracking unit 160a, the specifying unit 150a is similar as an alternative point used to obtain a predicted point of the lost feature point. It is necessary to identify feature points that are supposed to move along the trajectory. Specifically, the specifying unit 150a specifies a feature point in a frame imaged by the right camera (imaging device 10a) as an alternative point used for obtaining a predicted point of a lost feature point. In this case, the specifying unit 150a on the frame 270 corresponding to the feature point 360 to be tracked on the frame 260 based on the distance information received from the distance measuring unit 130 (disparity information about the feature point 360). The feature point 370 is specified, and the feature point 370 is specified as an alternative point used for obtaining a predicted point when the feature point 360 disappears.

  Then, as shown in FIG. 25A, in the frame 261 that is the next frame (the (n + 1) th frame) of the frame 260, the feature point that is the tracking target by the wiper 900 that has been rotated from right to left. It is assumed that occlusion occurs due to 360 being blocked. In this case, as described above, the tracking determination unit 162 cannot track the feature point 360 and determines that the feature point 360 has disappeared. As described above, when it is determined by the tracking determination unit 162 that the feature point 360 has disappeared, the coordinate prediction unit 163a is imaged by the specifying unit 150a with the right camera in order to predict the motion of the lost feature point 360. The feature point 370 specified as a point corresponding to the feature point 360 in the frame 271 that is the next frame (the (n + 1) th frame) after the frame 270 is specified as an alternative point.

  The coordinate predicting unit 163a then deletes the optical flow 570 that is a vector from the corresponding point 470 corresponding to the coordinate of the feature point 370 of the frame 270 to the coordinate of the feature point 370 in the frame 271. 360 is used as an optical flow (optical flow 560). Specifically, the coordinate predicting unit 163a sets the lost feature point 360 at a position indicated by the optical flow 560 from the corresponding point 460 corresponding to the coordinate of the feature point 360 of the frame 260 in the frame 261. Predict that it is moving. That is, the coordinate predicting unit 163a obtains the coordinates of the point indicated by the optical flow 560 from the corresponding point 460 in the frame 261 as the coordinates of the predicted point 480 where it is estimated that the lost feature point 360 exists.

  As described above, the coordinate predicting unit 163a sets the point indicated by the optical flow of the alternative point specified by the frame captured by the right camera from the corresponding point of the lost feature point as the predicted point. As a result, the lost feature points can be restored again, and it is not necessary to use past images (frames) to continue tracking, and there is no need to re-track the feature points. The processing load can be reduced.

  In addition, as shown in FIG. 25, the coordinate predicting unit 163a specifies a feature point 370 corresponding to the lost feature point 360 as an alternative point. As a result, since the lost feature point is assumed to move substantially the same as the specified substitute point, the accuracy of the prediction of the prediction point by the coordinate prediction unit 163a is likely to be improved, and the tracking by the restoration unit 164 is performed. Point return can be made easier.

  Note that it is naturally possible to apply the return operation of the return unit 164 according to the modification of the first embodiment to the image processing device 1a according to the second embodiment.

[Third Embodiment]
The image processing apparatus according to the present embodiment will be described focusing on differences from the image processing apparatus 1 according to the first embodiment. In the first embodiment, it is assumed that the prediction point estimated by the coordinate prediction unit 163 is matched with the lost feature point and returned, but in this embodiment, the prediction point is used as the prediction point. An operation of determining and returning whether the closest feature point matches the lost feature point will be described.

(Tracking point return operation)
FIG. 26 is a diagram for explaining the tracking point return operation in the image processing apparatus according to the modification of the third embodiment. With reference to FIG. 26, the tracking point return operation by the return unit 164 of the feature point tracking unit 160 in the present embodiment will be described.

  FIG. 26A shows a state in which feature points 340 and 341 in the object 940 are respectively extracted by the feature point extraction unit 140 in the frame 240, as in FIG. 20A of the first embodiment. As described above, the feature point extraction unit 140 further specifies a template 640 having a predetermined size around the coordinates of the extracted feature point 340 in the frame 240, and temporarily stores the image of the template 640 in the storage unit. Let The template specifying and storing operations are performed in the same manner for the feature points 341.

  FIG. 26B is the same as the state shown in FIG. 20B of the first embodiment. In the frame 241, the feature point 340 to be tracked by the wiper 900 rotated and moved from right to left is displayed. It shows the state where occlusion has occurred due to being blocked. In this case, the tracking determination unit 162 cannot track the feature point 340 and determines that the feature point 340 has disappeared.

  After the tracking determination unit 162 determines that the feature point 340 to be tracked has disappeared, as shown in FIG. 26C, the restoration unit 164 performs a frame 242a (next to the frame 241 in which the feature point 340 has disappeared). In the frame), not the predicted point 451 estimated by the coordinate predicting unit 163 but the feature point closest to the predicted point 451 among the feature points newly extracted by the feature point extracting unit 140 (in FIG. A search area 740a having the same size as the template 640 is formed around the feature point 340). Then, the restoration unit 164 compares the template 640 stored in the storage unit with the image in the search area 740a, and calculates the degree of coincidence thereof.

  Then, the return unit 164 performs threshold determination on the calculated degree of coincidence. For example, if the degree of coincidence is a value indicating dissimilarity such as SAD, the restoration unit 164 matches the template 640 and the image in the search area 740a when the degree of coincidence is a predetermined threshold value or less. The feature point closest to the prediction point 451 among the feature points newly extracted by the feature point extraction unit 140 is returned as a tracking point for the lost feature point 340. On the other hand, the restoration unit 164 determines that the template 640 does not match the image of the search area 740a when the degree of matching is greater than a predetermined threshold. In this case, the prediction point estimation operation by the coordinate prediction unit 163 and the tracking point return operation by the return unit 164 are continued. However, for example, the return unit 164 determines that the tracking of the lost feature point 340 has failed if the return operation is successful even after being performed a predetermined number of times.

  As described above, the return unit 164 according to the present embodiment sets the template 640 to the predicted point 451 among the feature points newly extracted by the feature point extracting unit 140 instead of the search region centered on the predicted point 451. Comparison is made with a search area centered on a nearby feature point. This increases the probability that it can be restored with the original feature point that has disappeared, rather than the predicted point that may have shifted from the true position of the missing feature point, improving the accuracy of tracking point restoration. be able to.

  Of course, the return operation of the return unit 164 according to the modification of the first embodiment can also be applied to the image processing apparatus according to the third embodiment.

  Note that at least one of the distance measurement unit 130, the feature point extraction unit 140, the grouping unit 150, the identification unit 150a, the matching unit 161, the tracking determination unit 162, the coordinate prediction units 163 and 163a, and the restoration unit 164 described above is a program. The program executed by the image processing apparatus 1 is a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk) in an installable or executable file. The program is recorded on a computer-readable recording medium such as a computer program product.

  In addition, the program executed by the image processing apparatus according to each of the above-described embodiments and modifications is stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. May be. The program executed by the image processing apparatus according to each of the above-described embodiments and modifications may be provided or distributed via a network such as the Internet. Further, the above program may be provided by being incorporated in advance in a ROM or the like.

  The program executed by the image processing apparatus according to each of the above-described embodiments and modifications includes the above-described distance measurement unit 130, feature point extraction unit 140, grouping unit 150, identification unit 150a, matching unit 161, tracking determination The module may include at least one of the unit 162, the coordinate prediction units 163 and 163a, and the return unit 164. As actual hardware, the CPU 32 reads and executes the program from the above-described storage medium. Thus, each functional unit described above is loaded and generated on the main storage device.

DESCRIPTION OF SYMBOLS 1, 1a Image processing apparatus 2 Main-body part 10a, 10b Imaging device 11a, 11b Imaging lens 12a, 12b Aperture 13a, 13b Image sensor 20a, 20b Signal conversion device 21a, 21b CDS
22a, 22b AGC
23a, 23b ADC
24a, 24b Frame memory 30 Tracking processor 31 FPGA
32 CPU
33 ROM
34 RAM
35 I / F
36 speaker 39 bus line 100 vehicle 110 image acquisition unit 120 correction unit 130 distance measurement unit 131 cost calculation unit 132 cost synthesis unit 133 distance derivation unit 140 feature point extraction unit 150 grouping unit 150a identification unit 160, 160a feature point tracking unit 161 matching Unit 162 tracking determination unit 163, 163a coordinate prediction unit 164 return unit 170 warning unit 200, 201, 201a frame 210, 211 frame 220, 221, 225, 226 frame 227 image 230 frame 240-242, 242a frame 260, 261, 270 , 271 Frame 300, 310 Feature point 311 Tracking point 320, 321 Feature point 330-333 Feature point 340, 341 Feature point 360, 370 Feature point 410, 420 Corresponding point 440-4 42 Corresponding point 450, 451 Predicted point 460, 470 Corresponding point 480 Predicted point 510 Optical flow 540, 541 Optical flow 550, 551 Optical flow 560, 570 Optical flow 610, 640 Template 710, 711, 740, 740a Search region 810, 820 , 840 Search range 900 Wiper 901 Windshield 902, 910, 911 Object 940, 960 Object B Baseline length C Cost value d Shift amount E Object EL Epipolar line f Focal length Ia Reference image Ib Comparison image Lr Path cost value Ls Composite cost value S, Sa, Sb Points Z, Z1, Z2 Distance Δ, Δ1 Parallax value

Japanese Patent No. 4209647

Claims (15)

Extraction means for extracting feature points of a subject in a frame of an image captured by at least one of the two imaging means;
Matching means for performing a matching operation for searching for a feature point that matches the feature point in a frame next to the frame in which the feature point is shown;
Determination means for determining whether or not the feature point has disappeared in the next frame based on a result of the matching operation;
When it is determined by the determining means that the feature point has disappeared, based on the movement of an alternative point that replaces the lost feature point, a predicting unit that obtains a prediction point that predicts the position of the lost feature point;
When the point based on the predicted point matches the feature point before disappearing, a return means for returning the point based on the predicted point as the feature point to be tracked;
An image processing apparatus.   The image processing apparatus according to claim 1, wherein the prediction unit sets a position indicated by an optical flow of the alternative point from the position of the feature point before disappearance as the prediction point.   3. The image processing apparatus according to claim 1, wherein the matching unit performs the matching operation by searching for an image that matches the first template image including the feature point in a first search range in the next frame. .   The determination unit determines that the feature point has disappeared due to occlusion with respect to the feature point when it is determined that an object closer than a predetermined distance is included in the first search range in the next frame. Item 4. The image processing apparatus according to Item 3. Measuring means for measuring the distance to the subject included in the frame based on the parallax of each frame imaged by the two imaging means;
Classification means for classifying the plurality of feature points extracted by the extraction means into groups based on the distances to the plurality of feature points measured by the measurement means;
Further comprising
5. The prediction unit according to claim 1, wherein when the determination unit determines that the specific feature point has disappeared, the prediction unit uses any one of the feature points of the same group as the specific feature point that has disappeared as the substitute point. The image processing apparatus according to any one of claims.   The image processing apparatus according to claim 5, wherein the prediction unit sets a feature point closest to the specific feature point among the feature points of the same group as the specific feature point that has disappeared as the substitute point. The extraction means extracts feature points of the subject in a frame of a video imaged by both of the two imaging means;
A specifying unit that identifies a point corresponding to a feature point of a frame imaged by the one imaging unit based on a parallax of each frame imaged by the two imaging units using the frame imaged by the other imaging unit Further,
The said prediction means makes the point specified by the said specific means as a said alternative point as a point corresponding to the said specific feature point which lose | disappeared, when it determines with the specific feature point having disappeared by the said determination means. The image processing apparatus as described in any one of 1-4.   The restoration means searches for a matching image in the second search range including the prediction point and the second template image including the feature point before disappearing, and when the matching image can be searched, the prediction point The image processing apparatus according to claim 1, wherein a point based on the image is determined to match the feature point before disappearing.   The image processing apparatus according to claim 8, wherein when the returning unit cannot search for an image that matches the second template image, the searching unit expands the second search range and searches again.   When it is determined by the return means that the point based on the prediction point and the feature point before disappearing do not match, the operation for obtaining the prediction point by the prediction means, and the feature point by the return means The image processing apparatus according to claim 1, wherein the return operation is performed again.   The said return means returns this prediction point as the said feature point which is a tracking object, when the said prediction point and the said feature point before lose | disappearing correspond. Image processing apparatus.   The restoration unit is closest to the prediction point when the new feature point closest to the prediction point among the new feature points extracted by the extraction unit matches the feature point before disappearance. The image processing apparatus according to claim 1, wherein the new feature point is returned as the feature point to be tracked.   The notification means for notifying a warning when the point based on the prediction point cannot be returned as the feature point to be tracked by the return means. The image processing apparatus described. An extraction step of extracting feature points of a subject in a frame of an image captured by at least one of the two imaging units;
A matching step for performing a matching operation for searching for a feature point matching the feature point in a frame next to the frame in which the feature point is shown;
A determination step of determining whether the feature point has disappeared in the next frame based on a result of the matching operation;
When it is determined that the feature point has disappeared, a prediction step for obtaining a prediction point that predicts the position of the lost feature point based on the movement of the substitute point that replaces the lost feature point;
When the point based on the predicted point matches the feature point before disappearing, a return step for returning the point based on the predicted point as the feature point to be tracked;
An image processing method. Extraction means for extracting feature points of a subject in a frame of an image captured by at least one of the two imaging means;
Matching means for performing a matching operation for searching for a feature point that matches the feature point in a frame next to the frame in which the feature point is shown;
Determination means for determining whether or not the feature point has disappeared in the next frame based on a result of the matching operation;
When it is determined by the determining means that the feature point has disappeared, based on the movement of an alternative point that replaces the lost feature point, a predicting unit that obtains a prediction point that predicts the position of the lost feature point;
When the point based on the predicted point matches the feature point before disappearing, a return means for returning the point based on the predicted point as the feature point to be tracked;
A program to make a computer realize.