JP5152144B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus that processes an image captured by an imaging apparatus such as a video camera mounted on a moving body.

  For example, various apparatuses for detecting a moving object in an image by mounting a video camera on a moving body such as an automobile and processing an image changing from the video camera every moment have been proposed.

  For example, in the apparatus described in Patent Document 1, first, an optical flow that is a motion in units of local regions is extracted by analyzing a time change of a TV camera image included in an observation system that performs a straight-ahead motion. By converting this optical flow into a linear equation and solving these simultaneously, the FOE (Focus of Expansion) position, which is the projection point in the direction of the linear motion of the observation system, is obtained and stored. Then, the temporal position change of the stored FOE position with the current FOE position is analyzed, and when the temporal position change is equal to or greater than a predetermined change amount, it is determined that a moving object exists.

JP-A-8-194822

  In the apparatus of Patent Document 1 described above, it is assumed that there is no moving object when calculating the stored FOE position. However, for example, when a video camera is mounted on a moving body such as an automobile, a situation where a moving object does not always exist does not always appear.

  Here, the apparatus of Patent Document 1 simply extracts optical flows that are motions in units of local regions of an image, and calculates the FOE position that is the intersection of these optical flows by LU decomposition method, singular point decomposition method, or the like. It is only determined by the method. For this reason, when a moving object is included in an image or an error of an extracted optical flow is large even if it is a stationary object for some stationary objects, the FOE position can be obtained with high accuracy. Can not. Since the detection accuracy of the moving object depends on the accuracy of the FOE position, if the FOE position accuracy is not sufficient, the detection accuracy of the moving object also decreases.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus capable of obtaining the position of an FOE with high accuracy.

In order to achieve the above object, an image processing apparatus according to claim 1 is provided.
An image output means mounted on a moving body and periodically capturing and outputting an image;
A feature point extraction unit for extracting a plurality of feature points in the image captured by the image output means;
An optical flow calculator for calculating optical flows of a plurality of feature points from changes in the same feature points in a plurality of images output in time series from the image output means;
An extended focus calculation unit that calculates an extended focus that is an intersection obtained by extending a plurality of optical flows calculated by the optical flow calculation unit;
A determination unit that determines whether the feature point is a moving point or a stationary point by using the extended focus calculated by the extended focus calculation unit;
The extended focus calculation unit evaluates the sum of errors of each feature point with respect to a plurality of feature points, using an angle formed by a line segment from the extended focus to the coordinates of the feature point and the optical flow of the feature point as an error. A function is calculated and the extended focus is determined by using the evaluation function so that the sum of errors is reduced. When calculating the evaluation function from the error of each feature point, the extended focus is changed to the coordinate of the feature point. A weighting factor that reduces the degree of influence of the error of the feature point on the evaluation function is used as the angle formed by the line segment and the optical flow of the feature point increases.

  As described above, in the image processing apparatus according to claim 1, as the angle formed by the line segment from the extended focus to the feature point coordinates and the optical flow of the feature point increases, the error of the feature point with respect to the evaluation function decreases. An evaluation function is calculated from the error of each feature point using a weighting factor that reduces the degree of influence, and an extended focus is determined using the evaluation function calculated in this way so that the total error is reduced.

  If the optical flow at the moving point or the optical flow at the stationary point has a large error, the angle formed by the optical flow and the line segment from the extended focus to each feature point is relatively large. Regarding the feature point having such an optical flow, when the evaluation function is calculated by the above-described weighting factor, the influence degree of the error of the feature point is reduced. In other words, by calculating the evaluation function using the weighting factor, the error sum of the feature points having an optical flow with a small angle difference from the line segment from the extended focus to the feature point is more important and the sum of the errors is calculated. An evaluation function to be evaluated can be calculated.

  Therefore, even if the optical flow of the moving point is included in the image or the error of the optical flow of some stationary points is large, those effects are reduced, and the stationary The position of the extended focus can be determined from the optical flow of the points. As a result, the position of the extended focus can be obtained with high accuracy.

  As described in claim 2, when the determination unit determines whether each feature point is a moving point or a stationary point, the extended focus calculation unit uses only the feature point determined to be a stationary point. Thus, an evaluation function for evaluating the sum of errors of the feature points may be calculated. As described above, by calculating the evaluation function for only the stationary point as the feature point, it is possible to calculate the evaluation function by using only a relatively small error and emphasizing a smaller error among them. Therefore, the position of the extended focus can be determined with higher accuracy using this evaluation function.

  As described in claim 3, the extended focus calculation unit can use the evaluation function E represented by the following formula 1 to obtain the extended focus F at which the evaluation function E is minimized by the least square method.

  In Equation 1, | Vi∧ (Pi−F) | represents the value of the outer product of the optical flow Vi and the line segment (Pi−F) from the extended focus F to the feature point Pi. The value of the outer product increases as the angle between the optical flow Vi and the line segment (Pi-F) from the extended focus F toward the feature point Pi increases. Therefore, the evaluation function E is calculated using the weight coefficient Wi while placing importance on the value of the outer product with a small angle, and the position of the extended focus F is determined so that the evaluation function E becomes the minimum value. The position of the extended focal point F that matches the optical flow can be obtained.

  As described in claim 4, the extended focus calculation unit uses the outer product value Ri calculated using the extended focus Fprev calculated in the past as the initial value of the weighting factor Wi of each feature point Pi, When the extended focus F that minimizes the evaluation function E is determined using the initial value of the weighting factor Wi, as shown in Equation 2, the extended focus Fprev calculated in the past at the outer product value Ri is calculated. By replacing the focal point F, the weighting factor Wi of each feature point is updated, and the procedure of obtaining the extended focal point F that minimizes the evaluation function E using the updated weighting factor Wi is repeated, thereby expanding the focal point F. When the fluctuation in the range falls within a predetermined range, it is preferable that the extended focus F at that time is determined as the current extended focus F.

  It can be assumed that the position of the extended focus this time is in the vicinity of the extended focus Fprev calculated in the past (previous). Therefore, the initial value of the weighting factor Wi is determined using the outer product value Ri of the optical flow Vi and the line segment (Pi-Fprev) from the extended focus Fprev calculated in the past to the feature point Pi. Note that Ro is a constant determined in advance by experiments or the like. Thus, the weighting factor Wi can be determined so that the feature point Pi having a larger error has a smaller weight. On the contrary, the weighting factor Wi can be determined so that the feature point Pi with a smaller error has a larger weight with 1 being the upper limit. That is, when the weighting factor Wi is calculated by the above-described equation 2, the weighting factor Wi monotonously decreases as the error increases, with 1 being the maximum value when the error is zero and the outer product value Ri is zero. Accordingly, it is possible to calculate the weighting factor Wi that appropriately corresponds to the magnitude of the error of each feature point Pi.

  Furthermore, by calculating the weighting coefficient Wi using Equation 2, it is possible to suppress the influence of the distance difference from the extended focus F to each feature point Pi from being reflected in the evaluation function E. That is, the outer product value of the optical flow Vi and the line segment (Pi-F) from the extended focus F to the feature point Pi when calculating the evaluation function E is not limited to the angle formed by the optical flow Vi. And the length of the line segment (Pi-F) from the extended focus F to the feature point Pi. The length of the optical flow Vi and the length of the line segment (Pi-F) from the extended focus F to the feature point Pi differ depending on the distance between each feature point and the extension focus. Therefore, the initial value of the weighting factor Wi is expressed as in Equation 2 using the outer product value Ri of the optical flow Vi and the line segment (Pi-F) from the extended focus Fprev calculated in the past to the feature point Pi. By defining, for each feature point Pi, an evaluation function E for evaluating the total sum of errors with an angle θi formed by the optical flow Vi and a line segment (Pi−F) from the extended focus F to the feature point Pi as an error is defined. When calculating, the influence of the distance difference between each feature point Pi and the extended focus F can be reduced.

  When the extended focus F that minimizes the evaluation function E is obtained using the initial value of the weight coefficient Wi, the extended focus Fprev calculated in the past in the outer product value Ri for calculating the weight coefficient Wi is obtained. By replacing with the extended focus F, the weight coefficient Wi of each feature point is updated. Further, the procedure of obtaining the extended focus F that minimizes the evaluation function E using the updated weighting factor Wi is repeated until the variation of the extended focus F falls within a predetermined range. Thereby, the position of the extended focus can be obtained with higher accuracy.

  Note that the weighting factor Wi is not limited to the one calculated according to the above-described equation 2, and may be calculated according to the following equation 3, for example, as described in claim 5.

  According to Equation 3, the initial value of the weighting factor Wi is determined as the reciprocal of the outer product of the optical flow Vi and the line segment (Pi-Fprev) from the extended focus Fprev calculated in the past to the feature point Pi. Similarly to the case of Expression 2, when the extended focus F that minimizes the evaluation function E is obtained using the initial value of the weight coefficient Wi, the extended focus Fprev calculated in the past in the weight coefficient Wi of Expression 3 is obtained. Then, the weight coefficient Wi of each feature point is updated by replacing with the obtained extended focus F. Further, the procedure of obtaining the extended focus F that minimizes the evaluation function E using the updated weighting factor Wi is repeated until the variation of the extended focus F falls within a predetermined range.

  Even when the weighting factor Wi is calculated according to the above Equation 3, the weighting factor Wi can be determined such that the feature point having the larger error has a smaller weight. However, since the weighting factor Wi is maximized when the error becomes extremely small, there is a demerit that it is easy to cause instability of the extended focal position to be obtained. The calculation for calculating can be simplified.

  Note that when the weighting factor Wi is calculated according to Equation 3, in order to suppress the destabilization of the extended focal position to be obtained, the weighting factor Wi has a magnitude greater than or equal to a predetermined upper limit as described in claim 6. When this happens, the upper limit value is preferably used as the weighting factor Wi. Thereby, when the error becomes extremely small, it is possible to prevent the weight coefficient Wi from becoming excessively large.

  As described in claim 7, when the angle formed by the line segment from the extended focus to the coordinates of the feature point and the optical flow of the feature point is equal to or less than the first determination angle, the determination unit Can be determined to be a stationary point. Since the optical flow of the stationary point is projected onto the extended focus position, if the extended focus position with high accuracy can be obtained, the line segment from the extended focus to the coordinate of the feature point and the optical flow of the feature point are displayed. It is possible to determine that the point is a stationary point by a simple determination process in which the angle between and is equal to or less than the first determination angle.

  However, as described in claim 8, each time the extended focus is calculated by the extended focus calculation unit, the determination unit calculates a line segment from the extended focus to the feature point coordinates and the optical flow of the feature point. The angle formed may be compared with the first determination angle, and when the determination that the angle is equal to or less than the first determination angle is continuously made a plurality of times, the feature point may be determined as a stationary point. Thereby, the determination precision regarding a stationary point can be raised.

  According to a ninth aspect of the present invention, when the angle formed by the line segment from the extended focus to the coordinate of the feature point and the optical flow of the feature point is larger than the second determination angle, the determination unit It can be determined that the point is a moving point. The optical flow that is different from the direction of the line segment from the extended focus to the feature point coordinates is limited to the moving point, and if a highly accurate extended focus position can be obtained, the extension focus moves to the feature point coordinates. It can be determined that the point is a moving point by a simple determination process in which the angle formed by the line segment and the optical flow of the feature point is equal to or greater than the second determination angle.

  As described in claim 10, each time the extended focus is calculated by the extended focus calculator, the determination unit is an angle formed by a line segment from the extended focus to the coordinates of the feature point and the optical flow of the feature point. And the second determination angle may be compared, and if it is determined that the angle is larger than the second determination angle continuously for a plurality of times, the feature point may be determined as the moving point. Thereby, the determination precision regarding a moving point can be improved.

  As described in the eleventh aspect, the extended focus calculation unit may obtain the final extended focus by averaging the extended focus calculated this time and the extended focus calculated in the past. Accordingly, it is possible to obtain a stabilized extended focus position while suppressing fluctuations in the position of the extended focus.

It is a block diagram which shows the structure of the image processing apparatus by embodiment of this invention. It is a flowchart for demonstrating the various functions performed in an image processing apparatus. It is explanatory drawing for demonstrating the outline | summary of the detection method of a moving object in the image processing apparatus by this embodiment. It is the graph which showed the change of the weighting coefficient Wi with respect to the change of Ri / Ro. 5 is a graph showing an FOE position error calculated in the image processing apparatus according to the present embodiment and an FOE position error calculated without using the weighting coefficient Wi in the present embodiment.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described in detail with reference to the drawings. The image processing apparatus according to the present embodiment is mounted on a moving body and processes an image captured by an imaging apparatus that moves together with the moving body, thereby determining whether the moving object is included in the image and determining the moving object. Detection is performed. In the present embodiment, an example in which the image processing apparatus is mounted on an automobile will be described.

  As shown in FIG. 1, the image processing apparatus 100 includes a video camera 10 as an imaging apparatus and an image processing ECU 20 that processes an image taken by the video camera 10.

  The video camera 10 is constituted by, for example, a CCD camera, and is installed near the ceiling near the driver's seat of the automobile so as to photograph the front of the automobile. Note that the video camera 10 may be mounted in the vehicle interior so as to capture the rear of the vehicle, and a rear image of the vehicle may be captured when the vehicle moves backward. The video camera 10 periodically captures images and outputs the captured images to the image processing ECU 20.

  The image processing ECU 20 stores an A / D converter for converting an image output from the video camera 10 into a digital image, and a converted digital image, in addition to a microcomputer including a CPU, ROM, RAM, etc. (not shown). An image memory or the like is provided. Note that the image memory has a storage capacity capable of storing a plurality of images.

  The image processing ECU 20 executes various functions according to a program stored in advance. For example, an optical flow of each feature point included in the image is calculated from a change in the same feature point in a plurality of images output in time series from the video camera 10, and based on the calculated plurality of optical flows, Calculate the FOE position. Then, using the calculated FOE position, it is determined whether each feature point is a moving point or a stationary point. In this way, the image processing ECU 20 detects the moving object and outputs the detection result to the control device 30.

  Based on the detection result of the moving object by the image processing ECU 20, for example, the control device 30 notifies the driver that there is a moving object in front of the vehicle.

  Next, various functions executed in the image processing ECU 20 will be described in detail based on the flowchart of FIG. In FIG. 2, basically, various functions executed in the image processing ECU 20 are represented as functional blocks. However, in the image processing ECU 20, when image output is started by the video camera 20, the function to be executed changes according to the images output in time series. Therefore, FIG. 2 also shows changes over time of the functional blocks executed in the image processing ECU 20. Specifically, the same reference number is assigned to the same functional block, and different alphabets are attached to the reference number when the same functional block operates in different time zones.

  First, various functional blocks in FIG. 2 will be described. The image processing ECU 20 has a feature point calculation unit 41. The feature point calculation unit 41 includes a corner point extraction unit 42 and a neighborhood point deletion unit 43. When a new digital image is stored in the image memory 40, the corner point extraction unit 42 takes in the digital image and extracts edges, object corners, and the like as feature points in the digital image. Further, the neighboring point deletion unit 43 deletes the feature points existing in the vicinity of the coordinates of the feature points already extracted from the feature points such as the edges and the corners of the object extracted by the corner point extraction unit 42. There is a high possibility that a feature point existing in the vicinity of the coordinate of a feature point that has already been extracted is one in which the extracted feature point has moved. For this reason, by deleting such feature points by the neighboring point deletion unit 43, the feature point calculation unit 41 calculates only new feature points newly appearing in the digital image. Then, the new feature point calculated by the feature point calculation unit 41 is given to the first optical flow calculation unit 44.

  In the present embodiment, first to third optical flow calculation units 44 to 46 are provided as optical flow calculation units for calculating the optical flow of feature points.

  As described above, the first optical flow calculation unit 44 is already extracted based on the new feature points calculated by the feature point calculation unit 41, and other than the feature points that have already been determined as stationary point candidates or moving point candidates. The optical flow for a new feature point is calculated. For this reason, the first optical flow calculation unit 44 associates the feature points calculated by the feature point calculation unit 41 in the two digital images stored in the image memory 40 at different timings, thereby moving the motion vectors of the feature points. The optical flow corresponding to is calculated.

  The second optical flow calculation unit 45 calculates the optical flow of feature points that have been determined as still point candidates. The third optical flow calculation unit 46 calculates an optical flow of feature points that have been determined as moving point candidates. The method of calculating the optical flow in the second and third optical flow calculation units 45 and 46 is the same as that of the first optical flow calculation unit.

  In the above-described example, the feature point calculation unit 41 calculates feature points such as edges and corners of an object. However, the feature points can be calculated by other methods. For example, one digital image can be divided into small areas, and those small areas having characteristics in brightness distribution can be used as feature points. In this case, in another digital image, a window corresponding to the small area is set, and the position having the highest correlation with the brightness distribution of the small area is searched while slightly shifting the window. Even in this way, the optical flow of the feature points can be calculated.

  The FOE calculation unit 47 has already determined the feature point to be a still point candidate, and when the second optical flow calculation unit 45 has calculated the optical flow of the feature point of the still point candidate, Based on this, FOE coordinates are calculated. When the optical flow of the still point candidate is not calculated by the second optical flow calculation unit 45, the FOE coordinates are calculated based on the optical flow of the new feature point calculated by the first optical flow calculation unit 44. .

  The FOE calculation unit 47 may calculate the FOE coordinates based on all the optical flows calculated by the first to third optical flow calculation units 44 to 46, or may be calculated by the second optical flow calculation unit 45. The FOE coordinates may be calculated by combining the optical flow calculated by the first or third optical flow calculation unit 44 or 46 with the optical flow thus calculated. In this embodiment, as will be described later, an evaluation function for evaluating the sum of optical flow errors of each feature point is calculated, and the FOE coordinates are determined so that the sum of the errors is minimized. When calculating the evaluation function, a weighting factor Wi is used, and a small weighting factor Wi is set for an optical flow with a large error. For this reason, even if the optical flow for obtaining the FOE coordinates includes the optical flow of the moving point, the FOE coordinates can be obtained with high accuracy.

  Here, the calculation method of the FOE coordinate F in the FOE calculation part 47 is demonstrated. In this embodiment, as shown in FIG. 3, an angle θi formed by a line segment (Pi−F) from the FOE coordinate F to the coordinate of each feature point Pi and the optical flow Vi of the feature point Pi is defined as an error. An evaluation function E that evaluates the sum of errors of each feature point Pi is defined as in Equation 4 below. Then, the FOE coordinate F on the image can be obtained as a solution of the weighted least square problem defined by the evaluation function E.

  In Equation 4, | Vi∧ (Pi−F) | represents the value of the outer product of the optical flow Vi and the line segment (Pi−F) from the FOE coordinate F to the coordinate Pi of the feature point. The value of the outer product increases as the angle θi formed between the optical flow Vi and the line segment (Pi−F) from the FOE coordinate F toward the feature point coordinate Pi increases.

  Here, when the feature point is a stationary point, the angle θi formed by the line segment from the FOE coordinate F to the feature point coordinate Pi and the optical flow Vi of the feature point is usually zero or a value close to zero. . Therefore, the angle θi formed between the optical flow Vi and the line segment (Pi-F) from the FOE coordinate F to the feature point coordinate Pi increases when the feature point is a moving point or a stationary point. This is a case where the error of the calculated optical flow Vi is large. (However, as a real problem, it is difficult to completely separate the moving point and the stationary point.) Then, the influence of the optical flow Vi at such a moving point and the optical flow Vi at a stationary point with a large error as much as possible is considered. If it is excluded and the FOE coordinate F is obtained, the accuracy of the FOE position is improved.

  Therefore, in the present embodiment, the weight coefficient Wi is set to be small as will be described later for the optical flow Vi at the moving point and the optical flow Vi at the stationary point where the error is large. Thereby, when calculating the evaluation function E, the influence degree of the optical flow Vi of the moving point and the optical flow Vi of the stationary point having a large error is reduced. In other words, by calculating the evaluation function E using the weight coefficient Wi, the error of the feature point Pi having the optical flow Vi having a small angle θi with the line segment from the FOE coordinate F to the feature point coordinate Pi is more emphasized. Thus, the evaluation function E for evaluating the sum of errors can be calculated.

  Therefore, even if the optical flow Vi of the moving point is included in the image or the error of the optical flow Vi of a part of the stationary points is calculated largely, it is mainly not affected by the influence. The FOE coordinate F can be obtained with high accuracy from the optical flow Vi of the stationary point calculated with high accuracy.

  The weighting factor Wi is set to be smaller as the angle θi formed between the optical flow Vi and the line segment (Pi-F) from the FOE coordinate F to the feature point coordinate Pi is larger. Specifically, it can be set as shown in Equation 5 below.

  It can be assumed that the current position of the FOE coordinate F exists in the vicinity of the FOE coordinate Fprev calculated in the past (previous). Therefore, the initial value of the weighting coefficient Wi is determined using the outer product value Ri of the optical flow Vi and the line segment (Pi-Fprev) from the FOE coordinate Fprev calculated in the past to the feature point Pi.

  Specifically, 1 is added to the square value of the value obtained by dividing the outer product value Ri by a constant Ro determined in advance by experiments or the like, and the reciprocal of the square root is used as the weighting factor Wi. FIG. 4 is a graph showing changes in the weighting factor Wi with respect to changes in Ri / Ro, with Ri / Ro as the horizontal axis. Since Ro is a constant, the weighting factor Wi is a monotonically decreasing function that monotonously decreases as the outer product value Ri increases.

The data of the feature point Pi where Ri >> R0 is given a relatively small weight that is approximately the inverse of Ri / Ro, and the influence on the solution of the least squares method in the evaluation function E is reduced. Can be suppressed. On the other hand, for data of the feature point Pi satisfying Ri << R0, the weight is asymptotically approximated by 1−0.5 (Ri / Ro) ² , and as Ri / Ro approaches zero, a larger weight is set with 1 being the upper limit. Will take. As described above, the data of the feature point Pi having a very small error is handled in an (equivalent) manner equivalent to the solution of the normal least square method without weight.

  If the constant Ro is too small, the solution is not statistically stable, and if it is too large, the solution is susceptible to noise. Therefore, the constant Ro is set in advance to an appropriate value through experiments in an environment close to the usage environment.

  Thus, by calculating the weighting coefficient Wi in accordance with Equation 5, the weight of the feature point Pi with a larger error becomes smaller, and conversely, the data of the feature point Pi with a smaller error becomes larger with 1 as the upper limit. Thus, the weighting factor Wi can be determined. Accordingly, it is possible to calculate the weighting factor Wi that appropriately corresponds to the magnitude of the error of each feature point Pi.

  Further, by using the above-described weighting factor Wi, it is possible to suppress the influence of the distance difference from the FOE coordinate F to each feature point Pi from being reflected in the evaluation function E. That is, the outer product value of the optical flow Vi and the line segment (Pi−F) from the FOE coordinate F to the feature point Pi when calculating the evaluation function E is not dependent on only the angle θi formed by them, but the optical flow. It is influenced by the length of Vi and the length of the line segment (Pi-F) from the FOE coordinate F to the feature point Pi. The length of the optical flow Vi and the length of the line segment (Pi-F) from the FOE coordinate F to the feature point Pi differ depending on the distance between each feature point and the FOE coordinate F.

  In the present embodiment, the initial value of the weight coefficient Wi is set as the reciprocal number of the term including the outer product value Ri of the optical flow Vi and the line segment (Pi-F) from the FOE coordinate Fprev calculated in the past to the feature point Pi. Yes. For this reason, the initial value of the weight coefficient Wi changes in accordance with the change in the optical flow Vi or the line segment (Pi-F) so as to cancel the change. Thereby, when calculating the evaluation function E for evaluating the total sum of errors, the influence of the distance difference between each feature point Pi and the FOE coordinate F can be reduced.

  Then, as shown in Equation 6 below, the calculation of the FOE coordinate F and the update of the weighting factor Wi are repeatedly executed.

That is, when the FOE coordinate F ^(t) that minimizes the evaluation function E is obtained using the initial value of the weight coefficient Wi, the FOE coordinate Fprev calculated in the past in the outer product value Ri is obtained as the calculated FOE coordinate F ^{(t )} To update the outer product value Ri for each feature point Pi (Ri ^(t) → Ri ^{(t + 1)} ). Then, the weight coefficient Wi is updated using the updated outer product value Ri ^{(t + 1)} (Wi ^(t) → Wi ^{(t + 1)} ). Further, the FOE coordinate F ^{(t + 1)} that minimizes the evaluation function E is obtained using the updated weighting factor Wi ^{(t + 1)} . Such a procedure is repeated until the fluctuation of the FOE coordinate F falls within a predetermined range. Thereby, the position of the FOE coordinate F can be obtained with higher accuracy.

  FIG. 5 shows the accuracy of the FOE coordinates F obtained by the above-described method in comparison with the case where the FOE coordinates are obtained without using any weight. In the graph shown in FIG. 5, only the weight is changed, and the method for obtaining the FOE coordinates is the same as the method of the present embodiment. That is, FIG. 5 shows simulation results for obtaining FOE coordinates for cases with and without weight, and the true value of the FOE coordinates F is known. In the simulation, simulation data in which randomly generated errors were added to the optical flow of the feature point Pi was prepared, and the position of the FOE coordinate F was obtained by the least square method. The trial by this simulation was repeated 20 times, and the average of the difference between the calculated value of FOE and the true value was calculated with and without weight. In the calculation using the weight coefficient Wi, the weight coefficient Wi was updated five times.

  As shown in FIG. 5, the average of the deviation between the FOE coordinates and the true value obtained by the method according to the present embodiment is smaller than when the weighting factor Wi is not used, and is close to the true value according to the present embodiment. It can be seen that the FOE coordinates can be obtained.

  When the FOE calculation unit 47 calculates the FOE coordinate F as described above, the FOE coordinate F is given to the determination unit 48 shown in FIG. The determination unit 48 includes a θ calculation unit 49, a θ determination unit 50, a still point candidate determination unit 51, and a moving point candidate determination unit 52.

  The θ calculation unit 49 includes an optical flow Vi calculated by the first to third optical flow calculation units 44 to 46 and a line segment connecting the feature points Pi from the FOE coordinates F calculated by the FOE calculation unit 47. The formed angle θ is calculated for each feature point according to the following formula 7.

  The θ determination unit 50 compares the angle θ calculated for each feature point by the θ calculation unit 49 with a predetermined reference angle θth. When the angle θ is equal to or smaller than the predetermined reference angle θth, the feature point data is given to the stationary point candidate determination unit 51. On the other hand, when the angle θ is larger than the predetermined reference angle θth, the feature point data is given to the moving point candidate determination unit 52. Note that the θ determination unit 50 may change the reference angle used for determination of a stationary point candidate and determination of a moving point candidate. In this case, the stationary point candidate determination reference angle is set smaller than the moving point candidate determination reference angle. For feature points having an angle between the stationary point candidate determination reference angle and the moving point candidate determination reference angle, the determination of whether the point is a stationary point or a moving point is suspended until any reference angle is exceeded. The optical flow of the feature point for which this determination is suspended is calculated by a fourth optical flow calculation unit provided separately.

  Thus, in the present embodiment, as shown in FIG. 3, the angle θ formed by the line segment from the FOE coordinate F to the coordinate of the feature point Pi and the optical flow Vi of the feature point Pi is based on the predetermined reference angle θth. It can be determined whether the feature point is a moving point or a stationary point depending on whether or not it is larger. This is because the FOE coordinate F with high accuracy can be obtained according to the present embodiment.

  Conventionally, in order to separate the moving point and the stationary point, it is known to use a processing method such as the RANSAC method or the clustering method. However, these methods usually require a large amount of calculation. On the other hand, in the present embodiment, the high-precision FOE coordinate F can be obtained by repeating the least squares method several times, and by using such a high-precision FOE coordinate F, the moving point and Separation from the stationary point is sufficient only for the angle determination process. Accordingly, it is possible to calculate the FOE coordinate F that is robust against noise and to separate the moving point and the stationary point with a small amount of calculation compared to the conventional method.

  The still point candidate determining unit 51 determines the feature point given from the θ determining unit 50 as a still point candidate, and the moving point candidate determining unit 52 determines the feature point given from the θ determining unit 50 as a moving point candidate. To do. Then, the still point candidate determining unit 51 gives the feature point data determined as the still point candidate to the coordinate updating unit 53, and the moving point candidate determining unit 52 transmits the feature point data determined as the moving point candidate to the coordinate updating unit. 54.

  The coordinate updating units 53 and 54 update the coordinates of the stationary point candidate and the moving point candidate using each optical flow Vi included in the given feature point data. Then, the updated coordinates are given to the neighboring point deletion unit 43. As a result, the neighboring point deletion unit 43, as described above, among the feature points such as edges and object corners extracted by the corner point extraction unit 42, features that exist in the vicinity of the coordinates of feature points that have already been extracted. Points can be deleted.

  Further, the coordinate update unit 53 calculates the optical flow Vi of the feature point of the still point candidate when the image of the next frame is stored in the image memory 40 with respect to the feature point data that becomes the still point candidate whose coordinates are updated. To the second optical flow calculation unit 45. In addition, the coordinate update unit 54 provides the third optical flow calculation unit 46 with the data of feature points that are candidate movement points whose coordinates have been updated.

  Next, the change over time of the functional blocks executed in the image processing ECU 20 will be described with reference to FIG.

  In FIG. 2, it is assumed that an image of the first first frame is stored in the image memory 40a at the timing of time T1. At this time, the image processing ECU 20 has obtained only one image data. For this reason, functions other than the feature point calculation unit 41a cannot be executed. Since the feature point calculation unit 41a calculates the feature points of the image, the feature point calculation unit 41a is executed when the image is stored in the image memory 40a. Therefore, the image processing ECU 20 waits without executing any function other than the feature point calculation unit 41a until the image of the next frame is stored in the image memory 40b.

  When the image of the second frame is stored in the image memory 40b at the time T2, the image is given to the first optical flow calculation unit 44b. Accordingly, the first optical flow calculation unit 44 can obtain two images and calculate the optical flow Vi that is a motion vector of the feature point. However, at this point in time, the still point determination and the moving point determination are not performed for the feature points, and thus the optical flow Vi is not calculated by the second and third optical flow calculation units 45b and 46b.

  Further, since the optical flow of the stationary point is not calculated by the second optical flow calculation unit 45b, the FOE calculation unit 47b calculates the FOE coordinate F based on the optical flow calculated by the first optical calculation unit 44b. .

  When an image of the third frame is further stored in the image memory 40c at the time T3, the image is given to the first to third optical flow calculation units 44c to 46c. This is because, at this time, the stationary point determination or the moving point determination is performed on the feature points extracted from the two images acquired at time T1 and time T2. Thereafter, each time a new image is stored in the image memory 40, the first to third optical flow calculation units 44 to 46 respectively create new feature points and still images as in the case of the time T3. The optical flow of the point candidate and the moving point candidate is calculated.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

  For example, in the above-described embodiment, once the angle θi formed by the line segment (Pi−F) from the FOE coordinate F to the coordinate of the feature point Pi and the optical flow Vi of the feature point and the predetermined reference angle θth. According to the comparison result, it was determined whether the feature point Pi is a stationary point candidate or a moving point candidate.

  However, every time the FOE coordinate F is calculated by the FOE calculation unit 47, the angle θi formed by the line segment (Pi-F) from the FOE coordinate F to the feature point Pi and the optical flow Vi of the feature point and the reference angle When comparison with θth is made and it is determined that the angle θi is equal to or smaller than the reference angle θth continuously, the feature point Pi is determined to be a stationary point, or the angle θi is larger than the reference angle θth. Or the feature point Pi may be determined as the moving point. Thereby, the determination precision regarding a stationary point and a moving point can be raised.

  The FOE calculation unit 47 calculates the FOE coordinate F every time an image of a new frame is stored in the image memory 40. The FOE coordinate F calculated this time and one or more FOEs calculated in the past are calculated. The final FOE coordinates may be obtained by averaging the coordinates F with a predetermined weight. Thereby, the fluctuation | variation of the position of FOE coordinate can be suppressed and the stabilized FOE coordinate position can be calculated | required.

  In the above-described embodiment, the evaluation function E is defined using the outer product of the optical flow Vi and the line segment (Pi-F) from the FOE coordinate F to the feature point coordinate Pi. However, any angle θi formed between the optical flow Vi and the line segment (Pi-F) from the FOE coordinate F to the feature point coordinate Pi can be used as an evaluation function as long as the error can be evaluated. .

  Furthermore, in the above-described embodiment, the weighting factor Wi is calculated according to Formula 5, but the formula for calculating the weighting factor Wi is not limited to this. For example, the calculation formula for the weight coefficient Wi may be simplified as shown in the following formula 8.

  According to Equation 8, the initial value of the weighting factor Wi is determined as the reciprocal of the outer product of the optical flow Vi and the line segment (Pi-Fprev) from the extended focus Fprev calculated in the past to the feature point Pi. Similarly to the case of Equation 5, when the extended focus F that minimizes the evaluation function E is obtained using the initial value of the weighting factor Wi, the extended focus Fprev calculated in the past in the weighting factor Wi of Equation 8 is obtained. Then, the weighting factor Wi of each feature point Pi is updated by replacing with the obtained extended focus F. Further, the procedure of obtaining the FOE coordinate F that minimizes the evaluation function E using the updated weight coefficient Wi is repeated until the fluctuation of the FOE coordinate F falls within a predetermined range.

  Even when the weighting factor Wi is calculated in accordance with Equation 3 above, feature point data with a large angle between the optical flow Vi and the line segment (Pi-F) from the FOE coordinate F to the feature point coordinate Pi is large. As the weight becomes smaller, the weighting coefficient Wi can be determined. Furthermore, even when the weighting coefficient Wi calculated according to Expression 8 is used, the influence of the distance difference from the FOE coordinate F to each feature point Pi can be suppressed from being reflected in the evaluation function E.

  However, when the angle (error) between the optical flow Vi and the line segment (Pi-F) from the FOE coordinate F to the feature point coordinate Pi becomes extremely small, the weighting coefficient Wi calculated by Expression 8 is maximized. It will be. When the weighting factor Wi is maximized, there is a demerit that the FOE coordinates to be obtained are likely to be unstable.

  For this reason, when calculating the weighting factor Wi according to Equation 8, when the weighting factor Wi is greater than or equal to a predetermined upper limit value, the upper limit value is preferably used as the weighting factor Wi. Thereby, when the error becomes extremely small, it is possible to prevent the weight coefficient Wi from becoming excessively large. However, in this case, since the weight coefficients Wi of the feature points having an error smaller than the error corresponding to the upper limit value are all unified to the upper limit value, compared with the case where the weight coefficient Wi is calculated according to Equation 5, The accuracy of the FOE coordinate F is slightly reduced.

  Furthermore, the calculation formula of the weight coefficient Wi can be further simplified without using the outer product as shown in Formula 5 or Formula 8. For example, the weight coefficient Wi may be calculated according to the following formula 9.

  Also in this case, the weighting factor Wi is calculated so as to decrease as the angle θi between the optical flow Vi as an error and the line segment (Pi-F) from the FOE coordinate F to the feature point coordinate Pi increases. Can do.

  DESCRIPTION OF SYMBOLS 10 Video camera, 20 Image processing ECU, 40 Image memory, 41 Feature point calculation part, 44 1st optical flow calculation part, 45 2nd optical flow calculation part, 46 3rd optical flow calculation part, 47 FOE calculation part, 48 determination Unit, 53, 54 coordinate update unit, 100 image processing apparatus

Claims (11)

An image output means mounted on a moving body and periodically capturing and outputting an image;
A feature point extraction unit for extracting a plurality of feature points in the image captured by the image output means;
An optical flow calculation unit that calculates optical flows of a plurality of feature points from changes in the same feature points in a plurality of images output in time series from the image output unit;
An extended focus calculation unit that calculates an extended focus that is an intersection obtained by extending a plurality of optical flows calculated by the optical flow calculation unit;
A determination unit that determines whether the feature point is a moving point or a stationary point by using the extended focus calculated by the extended focus calculation unit;
The extended focus calculation unit evaluates the sum of errors of each feature point with respect to a plurality of feature points, using an angle formed by a line segment from the extended focus to the coordinate of the feature point and the optical flow of the feature point as an error. An evaluation function is calculated, and an extended focus is determined by using the evaluation function so that the sum of the errors becomes small. When calculating the evaluation function from an error of each feature point, the feature point is calculated from the extended focus. An image processing apparatus using a weighting coefficient that reduces a degree of influence of an error of a feature point on the evaluation function as an angle formed by a line segment toward the coordinates of the point and an optical flow of the feature point increases.   When the determination unit determines whether each feature point is a moving point or a stationary point, the extended focus calculation unit uses only the feature point determined to be a stationary point and uses the error of each feature point. The image processing apparatus according to claim 1, wherein an evaluation function for evaluating the sum of the image is calculated. 3. The extended focus calculation unit obtains an extended focus F at which the evaluation function E is minimum by a least square method using an evaluation function E represented by the following formula 1. An image processing apparatus according to 1.

The extended focus calculation unit determines an initial value of the weighting factor Wi of each feature point using an outer product value Ri calculated using the extended focus Fprev calculated in the past, as shown in Equation 2 below. When the extended focus F that minimizes the evaluation function E is obtained using the initial value of the weighting coefficient Wi, each of the extended focus Fprev calculated in the past in the outer product value Ri is replaced with the obtained extended focus F. By repeating the procedure of updating the outer product value Ri and the weighting factor Wi of the feature point and obtaining the extended focus F that minimizes the evaluation function E using the updated weighting factor Wi, the variation of the extended focus F is within a predetermined range. The image processing apparatus according to claim 3, wherein the extended focus F at that time is determined as the current extended focus F.

The extended focus calculation unit determines an initial value of the weighting factor Wi of each feature point using the extended focus Fprev calculated in the past as shown in Equation 3 below, and uses the initial value of the weighting factor Wi to perform the evaluation. When the extended focus F that minimizes the function E is obtained, the weighted coefficient Wi of each feature point is updated by replacing the previously calculated extended focus Fprev with the weighted coefficient Wi by the obtained extended focus F. By repeating the procedure of obtaining the extended focus F that minimizes the evaluation function E using the updated weight coefficient Wi, when the fluctuation of the extended focus F falls within a predetermined range, the extended focus at that time is set to the current focus. The image processing apparatus according to claim 3, wherein the image processing apparatus is defined as an extended focal point.

  The image processing apparatus according to claim 5, wherein when the weight coefficient Wi is greater than or equal to a predetermined upper limit value, the upper limit value is used as the weight coefficient Wi.   The determination unit determines that the feature point is a stationary point when an angle formed by a line segment from the extended focus to the coordinate of the feature point and an optical flow of the feature point is equal to or less than a first determination angle. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   Each time the extended focus is calculated by the extended focus calculation unit, the determination unit includes an angle formed by a line segment from the extended focus to the feature point coordinates and the optical flow of the feature point, and the first determination angle. The feature point is determined to be a stationary point when the determination that the angle is equal to or less than the first determination angle is made a plurality of times in succession. Processing equipment.   The determination unit determines that the feature point is a moving point when an angle formed by a line segment from the extended focus to the coordinate of the feature point and an optical flow of the feature point is larger than a second determination angle. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   Each time the extended focus is calculated by the extended focus calculation unit, the determination unit includes an angle formed by a line segment from the extended focus to the feature point coordinates and the optical flow of the feature point, and the second determination angle. The feature point is determined as a moving point when the determination that the angle is larger than the second determination angle is made a plurality of times in succession. Image processing device.   11. The extended focus calculation unit obtains a final extended focus by averaging the extended focus calculated this time and the extended focus calculated in the past. An image processing apparatus according to 1.

Images