JP7007649B2 - Optical flow estimator, optical flow estimation method, optical flow estimation system, and optical flow estimation program, as well as yaw rate estimator, yaw rate estimation method, yaw rate estimation system, and yaw rate estimation program. - Google Patents

Description

The present invention relates to an optical flow estimation device that estimates an optical flow based on data from a camera, an optical flow estimation method, an optical flow estimation system, and an optical flow estimation program, and a rate estimation device and a yaw rate estimation method using them. Regarding the yaw rate estimation system and the yaw rate estimation program.

In a system that automatically drives a vehicle and supports driving, it detects moving objects in front of the vehicle and estimates the course of the own vehicle. To this end, the system estimates an optical flow, which is a movement vector of pixel values, from an image of an in-vehicle camera. By estimating the optical flow of a stationary object in the image of the in-vehicle camera, it is possible to estimate the course of the own vehicle, and by estimating the optical flow of the moving object, it is possible to estimate the relative movement vector of the moving object with respect to the own vehicle.

However, in particular, when a moving object or a stationary object moves at a relatively high speed, or when the amount of light incident on the in-vehicle camera is insufficient (for example, when driving in the dark), or conversely, it is incident on the in-vehicle camera. If the amount of light emitted is too large (for example, when driving against backlight), an image that can effectively calculate the optical flow may not be obtained.

In view of the above problems, it has been proposed to adopt an event camera as an in-vehicle camera. While conventional cameras acquire brightness in frames, event cameras are cameras that imitate the human perception system and capture only changes in brightness. The output of the event camera is an asynchronous data string representing the time, the position of the pixel, and the polarity (whether the brightness is reduced or increased) when the brightness changes to a predetermined threshold value or more.

Therefore, the output of the event camera is spatially very sparse as compared with the output of the conventional camera, and the amount of data is very small. Further, the event camera has characteristics that the time resolution is very high (for example, in the order of micromilliseconds) and the dynamic range is high as compared with the conventional camera. That is, the event camera has the characteristics of a small amount of data, high time resolution, and high dynamic range (HDR) as compared with the conventional camera. These characteristics are very important for in-vehicle cameras that need to perform image processing in an instant or in real time to detect a moving object.

Non-Patent Document 1 discloses a method of estimating an optical flow from an output (event data) of an event camera. In the method of Non-Patent Document 1, an optical flow is estimated by defining a cost function related to event data and optimizing the cost function.

In this method, the brightness and the optical flow are estimated at the same time only from the event data within a certain time width. The cost function consists of sections related to the mechanism of the event camera for brightness and optical flow, the consistency between optical flow and brightness, and the spatial and temporal smoothness.

The method of Non-Patent Document 1 makes it possible to restore brightness and estimate optical flow in an instant or in real time even in a scene where a moving body moves quickly and has a high dynamic range.

P. Bardow, A. J. Davison, and S. Leutenegger. Simultaneous Optical Flow and Intensity Optimization From an Event Camera. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 884-892, 2016.

However, in the method of Non-Patent Document 1, since the brightness is restored from the sparse event data and the optical flow is calculated using the brightness, it is difficult to obtain an accurate optical flow and noise increases. Since the brightness is restored mainly at the place where the event occurred and the other parts are restored by smoothing, the optical flow often does not point in the correct direction, and the optical flow shows the correct direction. There is room for improvement in order to obtain.

An object of the present invention is to improve the accuracy of optical flow estimation based on data obtained by a camera fixed to a moving body.

One aspect of the present invention is an optical flow estimation device that estimates an optical flow based on data from a camera fixed to a moving body, and this optical flow estimation device is an initial estimation of the optical flow based on the data. The optical flow initial estimation unit that performs the initial estimation, the rotation parameter estimation unit that estimates the rotation parameter based on the initially estimated optical flow, and the FOE of the initially estimated optical flow based on the initially estimated rotation parameter. It has a configuration including an optical flow regularization unit that regularizes the initially estimated optical flow by modifying it so as to comply with.

With this configuration, the optical flow initially estimated by the conventional method is regularized so as to follow the FOE, so that the accuracy of the optical flow estimation can be improved.

In the above-mentioned optical flow estimation device, the optical flow regularization unit makes the above modification based on the optical flow obtained by removing the rotation component of the rotation parameter from the initially estimated optical flow and the optical flow extending radially from the FOE. You may go.

In the above-mentioned optical flow estimation device, the optical flow regularization unit is based on the optical flow obtained by removing the rotation component of the rotation parameter from the initially estimated optical flow and the optical flow extending radially from the FOE. The modification may be performed by obtaining the optical flow, and the rotational component may be further returned to the second optical flow.

In the above optical flow estimation device, the camera may be an event camera, and the data may be event data.

In the above-mentioned optical flow estimation device, the rotation parameter estimation unit may estimate the yaw rate as a rotation parameter by setting the pitch rate and the roll rate to 0 based on the initially estimated optical flow.

In the above-mentioned optical flow estimation device, the optical flow regularization unit may correct the direction of the optical flow according to the FOE.

In the initial estimation of the optical flow by the conventional method, an error is likely to occur in the direction. However, according to this configuration, the direction of the optical flow is corrected according to the FOE, so that the accuracy of the optical flow estimation can be improved.

In the above optical flow estimation device, the rotation parameter estimation unit may estimate the rotation parameter using event data in the upper part of the field of view of the camera.

With this configuration, the yaw rate is estimated using the upper portion where the change in yaw rate is large, so that the accuracy of yaw rate estimation can be improved.

In the above optical flow estimation device, the rotation parameter estimation unit may estimate the rotation parameter by applying a vehicle motion model to the yaw rate.

With this configuration, it is possible to eliminate changes in the yaw rate that are impossible for the vehicle, and it is possible to improve the estimation accuracy of the yaw rate.

In the above optical flow estimation device, the rotation parameter estimation unit may apply the motion model by applying a yaw rate to a Kalman filter or a particle filter.

Another aspect of the present invention is an optical flow estimation method that estimates an optical flow based on data from a camera fixed to a moving body, and this optical flow estimation method is an initial stage of optical flow based on the data. An optical flow initial estimation step for estimation, a rotation parameter estimation step for estimating a rotation parameter based on the initially estimated optical flow, and an initially estimated optical flow based on the initially estimated rotation parameter. It has a configuration including an optical flow regularization step that normalizes the initially estimated optical flow by modifying it to comply with the FOE.

Yet another aspect of the present invention is an optical flow estimation system, which is a camera fixed to a moving body and outputs data, an optical flow initial estimation unit that performs initial estimation of optical flow based on the data, and an initial stage. Initial estimation by a rotation parameter estimation unit that estimates rotation parameters based on the estimated optical flow, and by modifying the initially estimated optical flow to follow FOE based on the initially estimated rotation parameters. It has a configuration including an optical flow regularization unit for regularizing the optical flow.

Yet another aspect of the present invention is an optical flow estimation program executed by an apparatus in which data from a camera fixed to a moving body is input, wherein the optical flow estimation program converts the apparatus into the data. An optical flow initial estimation unit that performs initial estimation of optical flow based on the above, a rotation parameter estimation unit that estimates rotation parameters based on the initially estimated optical flow, and an initial estimation based on the initially estimated rotation parameters. By modifying the optical flow so as to comply with the FOE, it has a configuration that functions as an optical flow regularization unit that regularizes the initially estimated optical flow.

Yet another aspect of the present invention is a yaw rate estimation device that estimates the yaw rate of the vehicle based on data from a camera fixed to a moving body, and the yaw rate estimation device is an optical flow based on the data. Initial estimation is performed by an optical flow initial estimation unit that performs initial estimation, and a rotation parameter estimation unit that estimates yaw rate as a rotation parameter by setting the pitch rate and roll rate to 0 based on the initially estimated optical flow. It has a configuration including an optical flow regularization unit for regularizing the optical flow.

Yet another aspect of the present invention is a yaw rate estimation method for estimating the yaw rate of the vehicle based on data from a camera fixed to a moving body, and the yaw rate estimation method is based on the data for optical flow. Initial estimation is performed with an optical flow initial estimation step for initial estimation, and a rotation parameter estimation step for estimating yaw rate as a rotation parameter by setting the pitch rate and roll rate to 0 based on the initially estimated optical flow. It has a configuration including an optical flow regularization step for regularizing the optical flow.

Yet another aspect of the present invention is a yaw rate estimation system, which is a camera fixed to a moving body and outputs data, and an initial optical flow estimation based on the data. By setting the pitch rate and roll rate to 0 based on the estimation unit and the initially estimated optical flow, the rotation parameter estimation unit that estimates the yaw rate as a rotation parameter and the initially estimated optical flow are normalized. It has a configuration including an optical flow regularization unit.

Yet another aspect of the present invention is a yaw rate estimation program executed by a device in which data from a camera fixed to a moving body is input, and the yaw rate estimation program uses the device based on the data. An optical flow initial estimation unit that estimates the initial estimation of the optical flow, a rotation parameter estimation unit that estimates the yaw rate as a rotation parameter by setting the pitch rate and roll rate to 0 based on the initially estimated optical flow, and a rotation parameter estimation unit. It has a configuration that functions as an optical flow regularization unit that regularizes the initially estimated optical flow.

According to the present invention, since the optical flow initially estimated by the conventional method is regularized so as to follow the FOE, the accuracy of the optical flow estimation can be improved.

FIG. 1 is a diagram showing a configuration of an optical flow estimation system according to an embodiment of the present invention. FIG. 2 is a diagram showing a time window. FIG. 3 is a diagram illustrating the cost function for each term. FIG. 4 is a diagram illustrating the definition of FOE. FIG. 5 is a diagram showing a motion model of an in-vehicle camera. FIG. 6 is a diagram illustrating the properties of FOE. FIG. 7 is a diagram showing an example of a location where an event occurs in the field of view. FIG. 8 is a diagram illustrating rotational motion and translational motion. FIG. 9 is a diagram illustrating regularization by FOE. FIG. 10 is a flowchart of the optical flow estimation method according to the embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the embodiments described below show an example of the case where the present invention is carried out, and the present invention is not limited to the specific configuration described below. In carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted.

In the optical flow estimation system of the present embodiment, in estimating the optical flow based on the event data obtained by the event camera, in order to compensate for the incompleteness of the optical flow estimation based on the event data, the motion of the in-vehicle camera A new regularization is performed using the characteristics and the properties of FOE (Focus Of Expansion). In the conventional method, an optical flow pointing in a uniform direction or an optical flow pointing in a clearly wrong direction can be estimated from the event data, but the optical flow estimation system of the present embodiment can estimate the optical flow of the camera from the optical flow. By estimating the rotation parameter and performing regularization using the properties of FOE in consideration of the rotation component of the optical flow, the optical flow having the correct direction is estimated.

FIG. 1 is a diagram showing a configuration of an optical flow estimation system according to an embodiment of the present invention. The optical flow estimation system 100 includes an event camera 10 and an optical flow estimation device 20. Since the optical flow estimation device 20 estimates the yaw rate, which is a rotation parameter, in order to regularize the optical flow, the optical flow estimation device 20 also functions as a yaw rate estimation device, and the optical flow estimation system 100 also functions as a yaw rate estimation system.

The event camera 10 is an in-vehicle camera fixed to the vehicle so that its optical axis faces the traveling direction of the vehicle. The event camera 10 is fixed to the vehicle so that the rectangular field of view includes the road surface in front of the vehicle and the landscape in the distance. The road surface immediately in front of the vehicle (which may include the front portion of the vehicle) is captured in the lower part of the field of view, and distant objects and tall objects are captured in the upper part of the field of view.

The event camera 10 detects a change in brightness for each pixel arranged two-dimensionally. That is, the event camera 10 is a camera inspired by a living body and is completely different from a standard frame-based camera. A frame-based camera outputs the luminance values of all pixels as a frame image at a fixed time interval (for example, at intervals of about 30 ms).

On the other hand, the event camera 10 does not have the concept of a frame, and when the luminance value changes to a predetermined threshold value or more, the time stamp of the changed time, the position of the changed pixel, and the polarity of the change (positive or negative). The information of (which changed) is output. That is, the output of the event camera 10 is an asynchronous data string.

The time stamp of the event camera 10 is on the order of microseconds, and the minimum time interval of detectable events is about 12 μs. As described above, the event camera 10 has a characteristic that the time resolution is extremely high. Further, in the event camera 10, since the event is not generated for the time when there is no change and the pixel where there is no change, there is no redundant data in the event data, and the data amount of the event data can be suppressed to a small size.

Further, the event camera 10 has a characteristic that the dynamic range is very large because it captures the change in brightness on the order of log. The dynamic range of a standard frame-based camera is about 50 dB, while the dynamic range of an event camera is about 120 dB.

As described above, the event camera 10 has the characteristics that the time resolution is high, the amount of data is small, and the dynamic range is large as compared with the standard frame-based camera. These characteristics are extremely effective when used as an in-vehicle camera. However, since the output of the event camera 10 is significantly different from that of the frame-based camera, the output of the event camera 10 cannot be directly applied to the conventional computer vision algorithm. In the present embodiment, the optical flow is estimated from the output of the event camera, but in this case as well, a device different from the case of estimating the optical flow from the output of the frame-based camera is required.

The optical flow estimation device 20 includes an optical flow initial estimation unit 21, a rotation parameter estimation unit 22, and an optical flow regularization unit 23. Each of these components is realized by one or more processors executing the optical flow estimation program of the present embodiment. For this purpose, the optical flow estimation device 20 includes one or a plurality of processors as an information processing device. The optical flow estimation program may be stored in the storage device of the optical flow estimation device 20 and read and executed by the processor, or may be stored in a portable storage medium and read and executed by the processor. good.

The optical flow initial estimation unit 21 estimates the optical flow based on the output of the event camera 10 according to the method of Non-Patent Document 1. The estimation of the optical flow here is an initial estimation, and as will be described later, the optical flow is finally estimated by the regularization by the optical flow regularization unit 23.

The optical flow initial estimation unit 21 defines a time window having a predetermined time width, and optimizes a cost function designed for the relationship between brightness, optical flow, and events in the time window, so that only event data can be obtained. Luminance and optical flow are estimated at the same time from.

By this method, it is possible to estimate the optical flow in real time in a complicated environment in which both the event camera and the object are moving, such as when the event camera is mounted on a vehicle.

FIG. 2 is a diagram showing a time window according to an embodiment of the present invention. Let I be the spatial size of the time window, let T be the total time width, and divide the total time width T into K cells with a width of δ _t . The cost function is defined as in equation (1).

Here, L is the luminance, u is the optical flow, the subscript x represents the spatial derivative, and the subscript t represents the time derivative. Further, t _p is the time when the previous event occurred, φ (x) is the number of events in the time window at the position x, and i represents the index of each event. Further, θ is the threshold value of the luminance change for generating an event in the event camera 10, and ρ _i represents the polarity of the event of the index i.

第５項以外の項はすべてＬ１ノルムが使用されており、エッジを保存して最適化を行うことができる。 The function h _θ of the fifth term is defined by the equation (2).

The L1 norm is used for all terms other than the fifth term, and the edges can be preserved for optimization.

FIG. 3 is a diagram illustrating the cost function of the embodiment of the present invention for each term. As shown in FIG. 3, the first term is the L1 norm of the spatial derivative of the optical flow, and the cost of the place where the derivative is large (that is, the place where the change is drastic) increases. Therefore, the first term is a term for evaluating the spatial smoothness of the estimated optical flow.

The second term is the L1 norm of the time derivative of the optical flow, and the cost at the time when the derivative is large (that is, the time when the change is drastic) becomes large. Therefore, the second term is a term for evaluating the temporal smoothness of the estimated optical flow. The third term is the L1 norm of the spatial derivative of the image, and is a term for evaluating the spatial smoothness of the estimated luminance image.

Before explaining the fourth item, the definition of optical flow will be described. Optical flow is defined as a pixel movement vector between frames. Now, let the luminance value at the position (x, y) at time t be I (x, y, t), and assume that the pixel moves by Δx, Δy after a minute time of Δt has elapsed. At this time, since the luminance value is the same before and after the movement, the definition formula of the following formula (3) can be obtained.

By Taylor-expanding the left side of this equation (3) and truncating the terms after the second order, the following equation (4) is obtained.

In order for equation (4) to hold, the second and subsequent terms on the right side must be zero. Therefore, the constraint equation of the optical flow of the following equation (5) can be obtained. At this time, both sides are divided by Δt.

In addition, since the amount of movement of the pixel per unit time is an optical flow, u = Δx / Δt, v = Δy / Δt, and I _x = ∂I / ∂x, I _y = ∂I / ∂y, If _∂I / ∂t = It, the constraint equation is expressed as the following equation (6).

Further, when expressed in the form of an inner product with u = (u, v) and I _x = (I _x , I _y ), the constraint equation becomes the following equation (7), and the fourth term of the cost function of the equation (1). It becomes the same as the notation of.

The fourth term is the L1 norm on the left side of the equation (7), and approaches 0 if the luminance and the optical flow match. Therefore, this fourth term is a term for evaluating the consistency between the luminance and the optical flow.

The fifth and sixth terms are data terms relating to the event camera. Based on the mechanism by which the event occurs, the cost is designed as described below with respect to the two properties of the consistency of the brightness value at the location where the event occurred and the invariance of the brightness at the location where the event does not occur in the time window. As a result, the luminance image is reconstructed.

The fifth term is a term in which when there is a change of θ or more in the luminance value, the excess is taken as the cost, and when there is no change of θ or more in the luminance value, the cost is set to 0. That is, when an event occurs, it is when there is a change of θ or more in the luminance value, and when an event occurs in the same pixel, it exceeds θ from the luminance value when the previous event occurred. The distance is the cost of this fifth item. Since there is no change of the threshold value θ or more from the change of the previous event to the next occurrence of the event, the fifth term becomes 0 during this period and nothing is added to the cost.

The optical flow initial estimation unit 21 estimates the luminance and the optical flow by optimizing this cost function (Equation (1)) in the time window. After the optimization for one frame is completed, the optical flow initial estimation unit 21 slides the time window by one cell and performs optimization while overlapping the time windows to estimate the brightness and the optical flow. go.

Prior to the explanation of the rotation parameter estimation unit 22 and the optical flow regularization unit 23, the concept of FOE and the motion analysis using the FOE characteristics of the event camera 10 as an in-vehicle camera will be described.

FIG. 4 is a diagram illustrating the definition of FOE. Generally, when the camera performs only translational motion, FOE is defined as the intersection of the translational axis of the direction of the movement vector and the image plane in a minute time of the camera, as shown in FIG.

Here, the uppercase letters X, Y, and Z in FIG. 4 represent the camera coordinate system, and the lowercase letters x, y represent the image coordinate system. Assuming that the position of the FOE in the image coordinate system is (x ₀ , y ₀ ) and the translation vector is T, the following equation (8) is simply derived from the relationship between the camera coordinate system and the image coordinate system using the focal length f. Can be done.

Therefore, from this equation (8), the coordinates of the FOE are expressed by the following equation (9).

The event camera 10 as an in-vehicle camera has a feature that the event camera 10 is fixed to the vehicle, and the translational motion component in the lateral direction is very small as compared with the translational motion component in the traveling direction, that is, T _x << It is characterized by T _z and T _y << T _z . Therefore, the event camera 10 as an in-vehicle camera has a property that the FOE is constant and the position thereof is substantially the center of the field of view. If the optical axis of the event camera 10 and the traveling direction of the vehicle are completely aligned, the FOE will be the center of the field of view, but if it is deviated, translational motion other than the traveling direction will occur, so the FOE will not be the center of the field of view. ..

By using the characteristics of FOE due to the fact that the event camera 10 is an in-vehicle camera, it is possible to easily estimate the optical flow and the rotation parameter of the event camera 10.

FIG. 5 is a diagram showing a motion model of an in-vehicle camera. As shown in FIG. 5, when the event camera 10 makes a translational motion T = (T _x , T _y , T _z ) ^T and a rotational motion Ω = (Ω _x , Ω _y , Ω _z ) ^T. Consider the movement from the point P (X _n , Y _n , Z _n ) in the camera coordinate system of to the point P'(X _{n + 1} , Y _{n + 1} , Z _{n + 1} ).

At this time, since the relative movement vectors of the point P with respect to the event camera 10 are T'=-T and Ω'=-Ω, the relationship between the point P and the point P'is expressed by the following equation (10). ..

At this time, R is a relative rotation matrix of the point P, and when the rotation angle is small, the second-order or higher term of the Taylor expansion of the sine and cosine can be omitted. It is expressed as 11).

Substituting equation (11) into equation (10) leads to equation (12) below.

The amount of change in each direction is expressed by the following equation (13).

Next, consider the movement from the point p (x _n , y _n ) on the image to the point p'(x _{n + 1} , y _{n + 1} ) when moving from the point P to the point P'. The point p (x _n , y _n ) on the image is expressed by the following equation (14) from the projection transformation equation using the point P (X _n , Y _n , _Zn ) in the camera coordinate system.

Assuming that the moving speed of the point p on the screen is v = (v _x , _vy ), the following equation (15) is derived by differentiating the equation (14).

Here, when the sampling interval is sufficiently small, the relationship between the corresponding points on the image between the frames is as shown in the following equation (16).

By substituting the equation (13) considering the motion of the own vehicle and the equation (9) defining the FOE into the equation (16) and arranging them assuming TZ / _{Z n} ≠ 0, the following equation (17) is used. Is obtained.

This equation (17) is composed of three elements: the coordinates of the FOE, the coordinates of the corresponding points, and the rotation parameter. That is, the rotation parameter can be estimated by obtaining the FOE in advance and finding the corresponding point.

The rotation parameter estimation unit 22 of the present embodiment further estimates the final yaw rate by applying the motion model of the vehicle to the obtained yaw rate. Since the event data obtained in this embodiment is obtained by the in-vehicle event camera 10, it can be assumed that the yaw angle does not change extremely. Therefore, the accuracy of the estimated yaw rate can be improved by performing filtering so as to eliminate such an extreme change in yaw angle. Specifically, the rotation parameter estimation unit 22 obtains the final estimated yaw rate by applying the yaw rate to a Kalman filter or a particle filter.

FIG. 6 is a diagram illustrating the properties of FOE. The movement vector of the corresponding point is the optical flow itself, and the term containing Ω is the rotation component of the optical flow. As shown in FIG. 6, by removing the component due to rotation from the optical flow, the optical flow extends radially from the FOE.

In the present embodiment, regularization is performed by utilizing such a property of FOE. Since the optical flow estimated by using the event camera 10 is estimated from sparse data called event data, it is incomplete and a particularly accurate orientation may not be obtained. Therefore, in the present embodiment, the orientation estimation accuracy is improved by adding regularization using FOE.

The rotation parameter estimation unit 22 estimates the yaw rate as a rotation parameter by setting the pitch rate and the roll rate to 0 based on the initially estimated optical flow. That is, the rotation parameter estimation unit 22 considers only the rotation (yaw rate) Ω _Y around the Y axis due to the handle operation generated in the curve, considering that the event camera 10 is mounted on the vehicle, and rotates around the X axis (rotation around the X axis). Pitch rate) Ω _X and rotation around the Z axis (roll rate) Ω _Z shall not exist. Substituting Ω _X = Ω _Z = 0 into equation (17) gives equation (18) below.

The points p and p'on the image assumed by the movement of the own vehicle are replaced with the corresponding points p = (x, y) and the points p'= (x', y') determined from the estimated luminance image. Then, the following equation (19) is obtained.

Therefore, if one corresponding point p (x, y) and p'(x', y') are known, the rotation parameter Ω _Y can be calculated from the coordinates of the corresponding point and the FOE. Here, the corresponding point is a set of points indicating the same pixel when the movement of the pixel between frames is considered, and is synonymous with the above-mentioned optical flow. In the present embodiment, since the dense optical flow for all pixels is estimated, the equation (19) uses the initially estimated optical flow u ₌ (ux, _yy ) to formulate the following equation (20). Can be rewritten as follows.

From this equation (20), Ω _Y for all pixels can be obtained. In order to obtain the estimated value Ω ^ _Y from this, a vote using the event is performed. FIG. 7 is a diagram showing an example of a location where an event occurs in the field of view. After obtaining Ω _Y of all pixels by the equation (20), it is discretized by the width of Δ, and the mode value at the event occurrence point in the upper part of the field of view (all pixels) shown in FIG. 7 is Ω ^ _Y. .. The reason for voting only at the place where the event occurred is the mechanism of brightness restoration and optical flow.

That is, the optical flow is estimated to match the restored luminance by the fourth term of the cost function relating to the optical flow. This luminance value is estimated so that the event when the event occurs and the luminance match, according to the term of the event camera in the sixth term of the cost function. That is, it can be said that the place where the event occurs where the input event data is most reflected has high reliability.

On the other hand, the points other than the event occurrence points are estimated from the surrounding values by the smoothness terms from the first term to the third term of the cost function, so that it can be said that the reliability is low. Therefore, by voting only in the upper part of the field of view, the rotational component can be accurately captured. The upper portion may be defined as, for example, the upper half portion of the visual field divided by the height at the center.

FIG. 8 is a diagram illustrating rotational motion and translational motion. In the example of FIG. 8, the vehicle is curved to the left. The lower part of the field of view is close to the event camera 10, and the optical flow at places such as the white line and the wall on the side of the road is relatively greatly affected by the translational motion of the event camera 10, and the component due to rotation appears remarkably. do not have.

On the other hand, the upper part of the field of view is a part far from the event camera 10, and the optical flow of a place such as a building is relatively greatly affected by the rotational movement of the event camera 10, and the component due to the rotation appears remarkably. As described above, it is effective to use the upper part of the visual field in order to accurately obtain the rotation parameter. Therefore, the rotation parameter estimation unit 22 votes only in the upper part of the visual field.

Similar to the rotation parameter estimation unit 22, the optical flow regularization unit 23 replaces the corresponding point with the optical flow u in the equation (18). Further, the term including the rotation parameter Ω _Y in the equation (18) represents the rotation component of the optical flow. When the rotational component of the optical flow is set as u ^Ω ₌ (ux ^Ω , _yy ^Ω ) as in the following equation (21), the following equation (22) is obtained.

In the formula (22), the above-mentioned property of FOE is simply expressed. The flow u'excluding rotation is expressed by the following equation (23).

Here, the arrow extending radially from the FOE to each coordinate is referred to as _ufoe , and the arrow having the same length as u'as shown in the following equation (24) is referred to as _u'foe .

For these _u'foe and u', u ^'is determined by the ratio represented by α as shown in the following equation (25).

The optical flow regularization unit 23 corrects (regularizes) the optical flow excluding the rotation component so as to follow the FOE by optimizing the _u'foe and u'square error as an objective function. After this optimization, the optical flow regularization unit 23 returns the rotation component u ^Ω to u ^'as in the following equation (26) to obtain u ^.

FIG. 9 is a diagram illustrating regularization by FOE. It is considered that the regularization by the optical flow regularization unit 23 modifies the optical flow to follow the FOE as shown in FIG. 9 and makes it better. In this way, by repeating the initial estimation of the optical flow and the subsequent estimation and regularization of the rotation parameters by sliding the time window, the optical flow can be regularized in a more correct direction, and the estimation is performed accordingly. The accuracy of the rotation parameter Ω ^ _Y is also improved.

FIG. 10 is a flowchart of the optical flow estimation method according to the embodiment of the present invention. The optical flow estimation method is executed in the optical flow estimation device 20. First, the optical flow initial estimation unit 21 performs initial estimation of the optical flow for the designated time window (step S101). Next, the rotation parameter estimation unit 22 estimates the rotation parameter (yaw rate) of the event camera 10 (that is, the vehicle) based on the initially estimated optical flow (step S102).

The optical flow regularization unit 23 regularizes the initially estimated optical flow using FOE (step S103). The optical flow estimation device 20 slides the time window to be processed (step S104), and repeats steps S101 to S103.

As described above, according to the optical flow estimation system 100 of the present embodiment, when the event camera is fixed facing the traveling direction of the vehicle, the FOE corrected by the yaw rate is fixed. , The final optical flow is estimated by regularizing the initially estimated optical flow based on the event data. That is, the optical flow is constructed by first solving a problem without regularity (convex problem) by FOE (initial estimation) and then solving a problem with regularization (non-convex problem) using it as an initial value (final estimation). Improve the estimation accuracy of.

Further, when the initially estimated optical flow is corrected according to the FOE, a configuration using the direction of the optical flow is adopted. This makes it possible to improve the accuracy of estimating the orientation that is susceptible to noise.

Further, the yaw rate is estimated by adopting the mode value of the yaw rate obtained for all the pixels, and in that case, voting is performed only in the upper part of the field of view of the event camera 10. As a result, the yaw rate is estimated using only the portion where the optical flow reflecting the yaw rate is likely to appear, so that the accuracy of the yaw rate estimation can be improved.

In the above embodiment, the optical flow estimation device in which event data is generated by an event camera mounted on the vehicle and the optical flow is estimated from the event data has been described, but the optical flow estimation device of the present invention is described. , Not limited to this. That is, the data used for estimating the optical flow is not limited to the event data, but may be normal image data obtained by a normal camera. In this way, even in the case of image data obtained with a normal camera, the initially estimated optical flow is modified to follow the FOE based on the initially estimated rotation parameters, and the initially estimated optical flow is made regular. By doing so, the accuracy of optical flow estimation can be improved.

Further, the camera is not limited to the one mounted on the vehicle, and may be provided on a moving body other than the vehicle. Further, in the above embodiment, assuming that the camera is mounted on the vehicle, the rotation parameter estimation unit 22 sets the pitch rate and the roll rate to 0, so that the optical flow estimation estimates the yaw rate as the rotation parameter. Although the device has been described, the rotation parameter estimation unit 22 may estimate the pitch rate or the roll rate as the rotation parameter depending on the type of the moving body.

INDUSTRIAL APPLICABILITY The present invention can improve the accuracy of optical flow estimation, and is useful as an optical flow estimation device or the like that estimates optical flow based on data from a camera.

10 Event camera 20 Optical flow estimator (yaw rate estimator)
21 Optical flow initial estimation unit 22 Rotation parameter estimation unit 23 Optical flow regularization unit 100 Optical flow estimation system (yaw rate estimation system)

Claims (16)

An optical flow estimator that estimates optical flow based on data from a camera fixed to a moving object.
An optical flow initial estimation unit that performs initial estimation of optical flow based on the above data, and an optical flow initial estimation unit.
A rotation parameter estimation unit that estimates rotation parameters based on the initially estimated optical flow, and a rotation parameter estimation unit.
An optical flow regularization unit that regularizes the initially estimated optical flow by modifying the initially estimated optical flow to follow the FOE based on the initially estimated rotation parameter.
An optical flow estimator equipped with. The modification according to claim 1, wherein the optical flow regularization unit makes the modification based on the optical flow obtained by removing the rotation component of the rotation parameter from the initially estimated optical flow and the optical flow extending radially from the FOE. Optical flow estimator. The optical flow regularization unit obtains the second optical flow based on the optical flow obtained by removing the rotation component of the rotation parameter from the initially estimated optical flow and the optical flow extending radially from the FOE. The optical flow estimation device according to claim 2, wherein the rotation component is returned to the second optical flow. The optical flow estimation device according to any one of claims 1 to 3, wherein the camera is an event camera and the data is event data. The optical according to any one of claims 1 to 4, wherein the rotation parameter estimation unit estimates the yaw rate as a rotation parameter by setting the pitch rate and the roll rate to 0 based on the initially estimated optical flow. Flow estimator. The optical flow estimation device according to any one of claims 1 to 5, wherein the optical flow regularization unit corrects the direction of the optical flow according to the FOE. The optical flow estimation device according to any one of claims 1 to 6, wherein the rotation parameter estimation unit estimates the rotation parameter using event data in the upper part of the field of view of the camera. The optical flow estimation device according to claim 5, wherein the rotation parameter estimation unit estimates the rotation parameters by applying a vehicle motion model to the yaw rate. The optical flow estimation device according to claim 8, wherein the rotation parameter estimation unit applies the motion model by applying a yaw rate to a Kalman filter or a particle filter. An optical flow estimation method that estimates optical flow based on data from a camera fixed to a moving object.
An optical flow initial estimation step for initial estimation of optical flow based on the above data, and an optical flow initial estimation step.
A rotation parameter estimation step that estimates rotation parameters based on the initially estimated optical flow, and
An optical flow regularization step that regularizes the initially estimated optical flow by modifying the initially estimated optical flow to follow the FOE based on the initially estimated rotation parameter.
An optical flow estimation method. A camera that is fixed to a moving object and outputs data,
An optical flow initial estimation unit that performs initial estimation of optical flow based on the above data, and an optical flow initial estimation unit.
A rotation parameter estimation unit that estimates rotation parameters based on the initially estimated optical flow, and a rotation parameter estimation unit.
An optical flow regularization unit that regularizes the initially estimated optical flow by modifying the initially estimated optical flow to follow the FOE based on the initially estimated rotation parameter.
An optical flow estimation system equipped with. An optical flow estimation program executed by a device for inputting data from a camera fixed to a moving body, wherein the device is used.
Optical flow initial estimation unit, which performs initial estimation of optical flow based on the above data.
The rotation parameter estimation unit that estimates the rotation parameter based on the initially estimated optical flow, and the initial estimated optical flow based on the initially estimated rotation parameter are modified to follow the FOE. An optical flow regularization unit that regularizes the estimated optical flow,
An optical flow estimation program that functions as. A yaw rate estimation device that estimates the yaw rate of the vehicle based on data from a camera fixed to a moving body.
An optical flow initial estimation unit that performs initial estimation of optical flow based on the above data, and an optical flow initial estimation unit.
A rotation parameter estimation unit that estimates the yaw rate as a rotation parameter by setting the pitch rate and roll rate to 0 based on the initially estimated optical flow.
An optical flow regularization unit that regularizes the initially estimated optical flow,
Equipped with a yaw rate estimator. It is a yaw rate estimation method that estimates the yaw rate of the vehicle based on the data from the camera fixed to the moving body.
An optical flow initial estimation step for initial estimation of optical flow based on the above data, and an optical flow initial estimation step.
A rotation parameter estimation step that estimates the yaw rate as a rotation parameter by setting the pitch rate and roll rate to 0 based on the initially estimated optical flow.
An optical flow regularization step that regularizes the initially estimated optical flow, and
Yaw rate estimation method, including. A camera that is fixed to a moving object and outputs data,
An optical flow initial estimation unit that performs initial estimation of optical flow based on the above data, and an optical flow initial estimation unit.
A rotation parameter estimation unit that estimates the yaw rate as a rotation parameter by setting the pitch rate and roll rate to 0 based on the initially estimated optical flow.
An optical flow regularization unit that regularizes the initially estimated optical flow,
Equipped with a yaw rate estimation system. A yaw rate estimation program executed by a device for inputting data from a camera fixed to a moving body, wherein the device is used.
Optical flow initial estimation unit, which performs initial estimation of optical flow based on the above data.
A rotation parameter estimation unit that estimates the yaw rate as a rotation parameter by setting the pitch rate and roll rate to 0 based on the initially estimated optical flow, and an optical flow regularization that regularizes the initially estimated optical flow. Kabu,
A yaw rate estimation program that functions as.

Images