FR3069689B1 - METHOD FOR ESTIMATING THE MOVEMENT OF A POINT IN AN IMAGE SEQUENCE - Google Patents

Abstract

The present invention relates to a method for estimating the movement of at least one point in a sequence of images varying in time, said sequence comprising at least a first image (t), a second image (t + n- 1) and a third image (t + n). The method comprises a step of determining, in the third image (t + n), a fourth zone of pixels corresponding to said point by calculating the optical flow between the first image (t) and the third image (t + n) at from a determined first pixel area, and a step of determining, in the third image (t + n), a fifth pixel area, referred to as a "refined area", from a determined third pixel area and the determined fourth pixel area.

Description

The present invention relates to the field of image processing and relates more specifically to a method of estimating the motion of a point in a sequence of images varying in time. The invention aims in particular to follow a point reliably and accurately in a plurality of consecutive images. The invention applies in particular to a motor vehicle comprising an on-board camera in order, for example, to reconstruct elements of the vehicle environment in three dimensions or to determine the trajectory of the vehicle.

In image processing, the optical flow is an approximation of the movement of pixels corresponding to the same object observed in a sequence of images varying in time. The optical flow makes it possible, in particular, to follow a point represented in the image sequence in order, for example, to detect the movement of the point in the image sequence, to segment a object associated with the point being tracked, to perform collision time calculations between the vehicle and an object associated with the tracked point (eg another vehicle), etc.

In known manner, an optical flow corresponds to the movement of a point between two images and is expressed as a vector for moving the point between said two images.

In a motor vehicle, it is known to use the optical flow to track points in images generated by a video camera mounted in the vehicle. Such a follow-up makes it possible to reconstruct the three-dimensional environment observed by the vehicle or the trajectory of the vehicle, in particular to assist the driver in his driving. For example, the reconstruction of the trajectory and / or the three-dimensional environment of the vehicle respectively allow the tracking of the marking lines on the ground or the detection of an obstacle that can trigger an emergency braking of the vehicle.

Several methods can be used to determine the optical flow. An example of a known method, called the Lukas-Kanade method, is based on a least squares minimization algorithm between two consecutive image sections of the image sequence.

In a known manner, it can be seen that the level of error relative to the value of the flow associated with such an estimate is all the higher, that is to say that the quality of the optical flow is even lower, that the speed of the vehicle is low or that the point which one follows is distant, that is to say not very dynamic in the sequence of images. Also, in this case, tracking a point between two images may lead to determining an erroneous position of the object. Indeed, the optical flow of a point being estimated between two consecutive images, it follows that the errors on the optical flows of several consecutive images, in which one follows the same point, accumulate over the images, resulting in a drift in the follow-up of the point. Thus, the 3D reconstruction of the vehicle environment or the determination of the trajectory of the vehicle can be significantly erroneous so that this presents a significant danger during the driver assistance provided to the driver.

The present invention aims to remedy in part these drawbacks by proposing a reliable and effective solution to reduce the drift generated by the accumulation of errors on the optical flows calculated for a sequence of images and to increase the accuracy in the tracking of a point in a sequence of images. To this end, the invention firstly relates to a method for estimating the movement of at least one point in a sequence of images varying in time, said sequence comprising at least a first image, a second image and a third image, the second image being posterior to the first image and the third image being posterior to the second image, the method comprising: a step of determining, in the first image, at least a first corresponding pixel area at a point to be followed, a step of determining, in the second image, a second zone of pixels corresponding to said point by calculating the optical flow between the first image and the second image from the first determined pixel zone, A step of determining, in the third image, a third zone of pixels corresponding to said point by calculating the optical flow between the second image and the third th image from the second pixel area determined. The method is remarkable in that it further comprises: a step of determining, in the third image, a fourth zone of pixels corresponding to said point by calculating the optical flow between the first image and the third image from the first pixel area determined and • a step of determining, in the third image, a fifth pixel area, called "refined area", from the determined third pixel area and the determined fourth pixel area.

Thus, by using a plurality of optical flows calculated at least from three different images to determine the position of the point in the third image, the method according to the invention makes it possible to avoid the accumulation of errors in estimating successive optical flows. which is obtained during monitoring only based on estimates of optical flows between successive image pairs.

Preferably, the determination of the fifth pixel area is performed by averaging the determined third pixel area and the determined fourth pixel area.

In one embodiment, the three images are consecutive in the image sequence.

As a variant, intermediate images may exist between the first and / or the second image and / or the third image and the method may then comprise an additional step of estimating the optical flow between this additional image and at least two of the three images mentioned. previously, thus giving a new measure of the position of the point in the third image.

Advantageously, the second image and the third image are consecutive in the image sequence.

In one embodiment, the first image, the second image, and the third image are consecutive in the image sequence.

According to one aspect of the invention, the method comprises a step of storing the first pixel area.

According to one aspect of the invention, the method comprises a step of storing the second pixel zone,

According to one aspect of the invention, the method comprises a step of storing the third pixel area.

According to one aspect of the invention, the method comprises a step of storing the fourth pixel area.

According to one aspect of the invention, the method comprises a step of storing the fifth pixel area.

In one embodiment, the first pixel area constitutes a reference pixel area that serves as a basis for determining a plurality of third pixel areas and fourth pixel areas from a plurality of second images and third different images.

Advantageously, the reference pixel area is used as long as the distance between the third pixel area and the fourth pixel area is less than a predetermined distance threshold.

Preferably, the predetermined distance threshold corresponds to the noise level of the third image.

The distance between the third zone of pixels and the fourth zone of pixels can be determined by using a Mahalanobis distance (distance standardized by the variances) compared to an uncertainty value given by a so-called chi-2 distribution ("chi squared"). ") At 2 degrees of freedom.

Advantageously, the distance between the third pixel zone and the fourth pixel zone being less than the predetermined threshold, the reference pixel area is used as long as the appearance between the first pixel area and the fourth pixel area is similar. .

By the term "similar" is meant that the pixels of the first pixel area and the pixels of the fourth pixel area are close in terms of values.

In one embodiment, the similarity between the appearance of the first pixel area and the appearance of the fourth pixel area is estimated by computing signatures based on Haar wavelets on the first pixel area and on the fourth pixel area. pixel area, and respectively comparing said signatures of the pixels of each of the areas between them to determine whether the two pixel areas are sufficiently similar or not. Alternatively, any other means suitable for determining and comparing the appearance of the first pixel area and the appearance of the fourth pixel area could be used.

According to one aspect of the invention, the method further comprises a step of detecting a lack of estimation of the point between two images of the sequence, such a defect notably being able to correspond to an occultation of the real point by an object or a temporary disappearance of that point. The invention also relates to a calculator for a motor vehicle, intended to estimate the movement of at least one point in a sequence of time-varying images generated by a camera mounted in said vehicle, said sequence comprising at least a first image, a second image and a third image, the second image being posterior to the first image and the third image being posterior to the second image. The computer is configured to determine: in the first image, at least a first pixel area corresponding to a point to be followed, in the second image, a second pixel area corresponding to the said point by calculating the optical flow between the first image and the second image from the determined first pixel area; • in the third image, a third pixel area corresponding to said point by calculating the optical flow between the second image and the third image from the determined second pixel area; . The calculator is remarkable in that it is furthermore configured to determine: in the third image, a fourth zone of pixels corresponding to said point by calculating the optical flow between the first image and the third image from the first image area; pixels determined, and • in the third image, a fifth pixel area, called "refined area", from the determined third pixel area and the determined fourth pixel area. The invention finally relates to a motor vehicle comprising a camera and a computer, as presented above, connected to said camera so as to receive at least one sequence of images captured by the camera in order to estimate the movement of at least one point in said sequence and allow assistance in driving the vehicle.

Figure 1 schematically illustrates an example of a vehicle according to the invention.

Figure 2 schematically illustrates an embodiment of the method according to the invention.

FIG. 3 schematically illustrates an exemplary implementation of the method according to the invention with three consecutive images.

FIG. 4 schematically illustrates an exemplary implementation of the method according to the invention with three non-consecutive images.

Figure 5 schematically illustrates successive optical flow estimates between an image t and an image t + n.

Figure 6A illustrates a first example of application.

Figure 6B illustrates a second example of application.

Fig. 6C illustrates a third exemplary application.

Figure 6D illustrates a fourth application example.

Figure 6E illustrates a fifth example of application.

Figure 6F illustrates a sixth exemplary application.

Figure 6G illustrates a seventh example of application.

Figure 7 schematically illustrates the estimates of all successive optical flows between an image t and an image t + n.

Figure 8 illustrates an eighth example of application.

FIG. 1 shows an example of vehicle V according to the invention.

The vehicle V comprises a driver assistance system 10 comprising a camera 100 mounted in the vehicle V and a computer 200 for processing the images captured by said camera 100.

The camera 100 may for example be a monocular-type camera that can be mounted on the upper part of the front windshield 1A of the vehicle V so as to film the road 2A on which the vehicle V and the environment E of the vehicle is traveling. located in the field of view of the camera 100. Note however that the camera 100 could be mounted at any other suitable place of the vehicle 1.

Preferably, the camera 100 and the computer 200 are implemented by the same physical entity. Alternatively, they could constitute two separate physical entities connected via a link or a communication network of the vehicle 1.

The computer 200 is particularly intended to estimate the movement of a point in a sequence of images captured by the camera 100 in order, for example, to reconstruct the three-dimensional environment observed by the vehicle V or the trajectory of the vehicle V to assist the driver in his conduct.

An image sequence comprises a succession of time-varying images, generated for example at a number of at least 16 frames per second, for example 16, 24, 25, 30, 50 or 100.

With reference to FIGS. 3 to 5, the image sequence comprises at least a first image, a second image and a third image. The second image is posterior to the first image and the third image is posterior to the second image.

In the example of FIG. 3, the three images (t, t + 1, t + 2) are consecutive. In the more general example of FIGS. 4 and 5, where n is a natural integer strictly greater than 1, the three images (t, t + n-1, t + n) are consecutive when n = 2 and non-consecutive when n is strictly greater than 2.

The computer 200 is configured to perform a plurality of tasks for estimating the movement of points represented in the image sequence. For this purpose, the computer 200 comprises, for example, one or more processors or one or more microcontrollers as well as electronic components making it possible to implement different functions, for example filtering signals, in particular signals received from the camera 100.

A point corresponds to a real element of the environment of the vehicle filmed by the camera 100. A point is represented in the images in the form of a pixel or a plurality of pixels. For example, when assisting with the driving of the vehicle 1, it may be useful to follow points on a traffic sign, a traffic light, a road 2A, a shoulder, a bridge, etc.

In the following we will present the tracking of the movement of a point in a sequence of images, but it goes without saying that several points could be followed at the same time by duplicating the method described below.

A point is followed in the image sequence using an optical flow algorithm. Several methods can be used to determine the optical flow. An example of a known method, called the Lukas-Kanade method, is based on a least squares minimization algorithm between two consecutive image sections of the image sequence. However, it goes without saying that any other method of tracking the movement of a point in a sequence of images could be used in the context of the present invention.

In order to follow the movement of a point in a sequence of images, the computer 200 is first configured to determine in the first image at least a first pixel area corresponding to a point to follow, to determine in the second image a second pixel area corresponding to said point by calculating the optical flow between the first image and the second image from the first determined pixel area, and for determining in the third image a third pixel area corresponding to said point by calculating the optical flow between the second image and the third image from the second determined pixel area.

Remarkably, the computer 200 is also configured to determine in the third image a fourth pixel area corresponding to said point by calculating the optical flow between the first image and the third image from the first determined pixel area, and to determine in the third image, a fifth pixel area, referred to as the "refined area", from the determined third pixel area and the determined fourth pixel area. The invention will now be described in its implementation with reference to FIG.

In a step E0, the camera 100 first generates a sequence of images representing the environment of the vehicle 1, which it transmits to the computer 200.

The so-called extraction noise, generated during the optical flow calculation, can be modeled according to a Gaussian distribution where "mu" corresponds to the mean of the distribution and "sigma" corresponds to the standard deviation of the distribution. . In practice, "mu" corresponds to the position of the point in the image according to its coordinates (u, v) and "sigma" corresponds to the noise generated during the optical flow calculation (du, dv).

We thus note, with reference to FIGS. 3 and 4: • (mu01, sigma01) the average and standard deviation pair calculated between an image 0 and an image 1, • (mm 2, sigmai 2) the average torque and of standard deviation calculated between an image 1 and an image 2, • (mu (ni) _n, sigma (ni) _n) the mean and standard deviation pair calculated between an image (t + n-1) ) and an image (t + n), etc.

We then note:

the pair of mean and standard deviation calculated between an image (t) and an image (t-n).

In a step E1, the computer 200 determines (or selects), in the first image, a first pixel area corresponding to a point to follow.

In a step E2, the computer 200 determines, in the second image, a second pixel area corresponding to said point by calculating the optical flow between the first image and the second image from the first determined pixel area.

In a step E3, the computer 200 determines, in the third image, a third zone of pixels corresponding to said point by calculating the optical flow between the second image and the third image from the determined second pixel zone.

In a step E4, the computer 200 determines, in the third image, a fourth pixel area corresponding to said point by calculating the optical flow between the first image and the third image from the first determined pixel area.

Finally, in a step E5, the computer 200 determines, in the third image, a fifth zone of pixels, called "refined zone", from the determined third pixel area and the determined fourth pixel area, preferably by means of the third pixel area and the fourth pixel area as will be described hereinafter.

Implementation 1: three consecutive images in the image sequence

In a first exemplary implementation, the first image 0, the second image 1 and the third image 2 are consecutive in the image sequence.

With reference to FIG. 3, we then have:

which represents the third pixel area and the corresponding standard deviation, and

which represents the fourth pixel area and the corresponding standard deviation.

We then deduce:

which represent the fifth pixel area and the corresponding standard deviation.

Implementation 2: three images not necessarily consecutive in the sequence of images

In a second example of implementation, it is considered that the first image t and the second image (t + n-1) are not consecutive in the image sequence (n being strictly greater than 2).

With reference to FIG. 4, we then have:

In this embodiment, the image 0 constitutes a reference image and n can be modified to successively use a plurality of consecutive pairs of consecutive images [(t + n-1), (t + n)] in order to refine the process and reduce the noise level while allowing effective tracking of the point.

In particular, the image 0 can be chosen as being very old in the chronology of the sequence with respect to the pair of consecutive images [(t + n-1), (t + n)] while presenting a high similarity with the image. current image used (t + n), i.e. (with reference to FIG. 5), that the position difference Δ between the third pixel area and the fourth pixel area is small.

When the position difference Δ1 between the third pixel area and the fourth pixel area exceeds a predetermined distance threshold or the appearance difference Δ2 between the first pixel area and the fourth pixel area exceeds a threshold of predetermined appearance, another image of the sequence may be chosen as the reference image. The position Δ1 difference between the third pixel area and the fourth pixel area can be determined using, for example, in a manner known per se, a Mahalanobis distance (distance standardized by the variances) compared to an uncertainty value. given by a so-called chi-2 distribution with 2 degrees of freedom. The apparent difference Δ2 between the first pixel area and the fourth pixel area can be calculated by determining a signature of each pixel of a zone, by

example by using Haar wavelets, and comparing the signatures of the pixels of each of the areas between them to determine if the two pixel areas are sufficiently similar or not.

Application case

In the examples given below, the number of the reference image is annotated with the letter K, the current image is denoted n and the intermediate images are denoted 2,..., N-1.

M (Match) is the case where an area of pixels in a posterior image of the sequence, determined using the optical flow, corresponds to a pixel area of an earlier image from which said optical stream has been calculated. For example, the fourth pixel area determined in the third image corresponds to the first pixel area determined in the first image when the algorithm used for the optical flow gives a valid match, which, for example, in the case of Lukas Kanade , corresponds to a residue of least squares less than a predetermined threshold.

F (Fail) denotes the case where an area of pixels in a posterior image of the sequence, determined using the optical flow, does not correspond to an area of pixels of an earlier image from which said flow has been calculated. optical. For example, the determined first pixel area and the determined fourth pixel area do not match when the algorithm used for the optical stream gives an invalid match, which, for example, in the case of Lukas Kanade, corresponds to a residue of least squares greater than a predetermined threshold.

FP (Fail but Predicted) is the case where the determined first pixel area and the determined fourth pixel area do not match, but an external index, for example three-dimensional reconstruction from an image prior to the fourth image, makes it possible to establish a prediction of the fourth zone of pixels.

Example A

The application case illustrated in FIG. 6A corresponds to the case for which the position difference Δ1 between the third pixel area of the image n obtained from the images 1K to n-1 and the fourth pixel area of the image. image n obtained directly from the reference image 1K is less than or equal to the predetermined threshold.

In this case, the first pixel area (image 1 K) and the fourth pixel area (image n) correspond to each other (M) on the one hand, and the second pixel area (image n-1) and the third pixel area (image n) correspond to each other (M) on the other hand.

Example B

In the case of application illustrated in FIG. 6B: the first pixel area of the image 1 and the fourth pixel area of the image n do not correspond to each other (F); The first pixel area of the 2K image and the fourth pixel area (image n) correspond to each other (M) and in this case the second 2K image constitutes the reference image; The second pixel area of the n-1 image and the third pixel area of the n image correspond to each other (M).

Example C

In the case of application illustrated in FIG. 6C: the first pixel area of the 1K image and the fourth pixel area of the n image correspond to each other (M); The second pixel area of the n-1 image and the third pixel area of the n image do not correspond to each other but a prediction of the third pixel area exists (FP). By the term "prediction" is meant here that, thanks to the preceding acquisitions, it is possible to make a prediction of the third zone of pixels, for example using a model in which it is considered that the speed of the camera is constant and the observed object is static in the scene (a model known as the "prediction base").

Example D

In the case of application illustrated in FIG. 6D: the first pixel area of the image 1 and the fourth pixel area of the image n do not correspond to each other (F); The second pixel area of the n-1 image and the third pixel area of the n image do not correspond to each other (F).

In this case, one can either stop following the point, or do 2D prediction (that is, using an image-based model) or 3D (that is, using a model). considering the object in its real space).

Example E

In the case of application illustrated in FIG. 6E, it is not possible to determine the fourth zone of pixels in the image n from the first pixel area of the image 1, for example because the fourth pixel area appears out of the image n. In this case, the second 2K image can be chosen as the reference image.

In this example: the first pixel area of the 2K image and the fourth pixel area of the n image do not correspond to each other (F); The second pixel area of the image n-1 and the third pixel area of the image n do not correspond to each other (F) because the object has left the image n so it is no longer possible to continue to make matches between the image n and the previous ones.

Example F

In the case of application illustrated in FIG. 6F, it is not possible to determine the fourth zone of pixels in the image n from the first pixel area of the image 1, nor the third zone of pixels. of the image n from the second pixel area of the n-1 image, for example because the third and fourth pixel areas appear out of the image n.

In this case, we can stop following the point from the image n.

Example G

In the case of application illustrated in FIG. 6G, it is not possible to determine the fourth zone of pixels in the image n from the first pixel area of the image 1 (shaded boxes) because the noise generated by the calculation of the optical flow between the image 1K and the image n is greater than the difference in position between the first pixel area and the fourth pixel area.

In this case, the image n can be discarded to follow the point considered and the process is continued using the following image n + 1.

In an exemplary implementation, with reference to FIG. 7, it is possible to use all the optical flows calculated between all the images of the sequence in order to further refine the precision of the tracking of the point.

With reference to FIG. 8, the (M) matches between the images can be used to detect an occlusion. Indeed, when the point is temporarily hidden by another object, the matching algorithm will no longer work. However, it is possible to continue to predict the point followed in order to find it later. An occlusion will display some images for which the process could not match. There was an occlusion of the characteristic during the n-2 and n-1 frames (boxes F in Figure 8). Then, as soon as the point reappears (images n and n + 1), the correspondence can be performed again.

By combining optical flows of different images to follow the same point in different ways, the method according to the invention makes it possible to reduce the impact of the noise on the point tracking, which notably improves the accuracy of the tracking and therefore of the driving assistance.

The method according to the invention is particularly suitable for tracking points used in a driving assistance system for a motor vehicle. It is also advantageously applicable to point tracking used in radar or lidar type systems.

Claims (4)

A method for estimating the motion of at least one point in a time-varying image sequence, said sequence comprising at least a first image (t), a second image (t + n-1) and a third image image (t + n), the second image (t + n-1) being posterior to the first image (t) and the third image (t + n) being posterior to the second image (t + n-1), the method comprising: • a step of determining, in the first image (t), at least a first pixel area corresponding to a point to follow, • a determining step, in the second image (t + n-1) a second pixel area corresponding to said point by calculating the optical flow between the first image (t) and the second image (t + n-1) from the determined first pixel area, • a determination step, in the third image (t + n), of a third zone of pixels corresponding to said point by calculating the optical flow between the second image (t + n-1) and the third image (t + n) from the determined second pixel area, said method being characterized in that it further comprises: a determination step in the third image (t + n), a fourth pixel area corresponding to said point by calculating the optical flow between the first image (t) and the third image (t + n) from the first determined pixel area, • a step determining, in the third image (t + n), a fifth pixel area, called the "refined area", from the determined third pixel area and the determined fourth pixel area. 2. Method according to the preceding claim, wherein the second image (t + n-1) and the third image (t + n) are consecutive in the image sequence. 3. Method according to the preceding claim, wherein the first image (t), the second image (t + n-1) and the third image (t + n) are consecutive in the image sequence. 4. Method according to one of the preceding claims, comprising a step of storing the first pixel area, the second pixel area, the third pixel area and the fourth pixel area. 5. Method according to the preceding claim, comprising a step of storing the fifth pixel area. The method according to one of the preceding claims, wherein the first pixel area constitutes a reference pixel area serving as a basis for determining a plurality of third pixel areas and fourth pixel areas from a plurality of pixels. a plurality of second images (t + n-1) and third images (t + n) different. 7. Method according to the preceding claim, wherein the reference pixel area is used as long as the distance between the third pixel area and the fourth pixel area is less than a predetermined threshold. 8. Method according to the preceding claim, wherein the predetermined threshold corresponds to the noise level of the third image (t + n). Motor vehicle calculator for estimating the movement of at least one point in a time-varying image sequence generated by a camera mounted in said vehicle, said sequence comprising at least a first image (t), a second image (t + n-1) and a third image (t + n), the second image (t + n-1) being posterior to the first image (t) and the third image (t + n) being posterior to the second image (t + n-1), the computer being configured to: • determine, in the first image (t), at least a first pixel area corresponding to a point to be followed, • determine, in the second image ( t + n-1), a second pixel area corresponding to said point by calculating the optical flow between the first image (t) and the second image (t + n-1) from the determined first pixel area, • determine in the third image (t + n), a third zone of pixels corresponding to said point in callus coupling the optical flow between the second image (t + n-1) and the third image (t + n) from the determined second pixel area, said computer being characterized in that it is further configured to: determining, in the third image (t + n), a fourth zone of pixels corresponding to said point by calculating the optical flow between the first image (t) and the third image (t + n) from the first determined pixel area • determining, in the third image (t + n), a fifth pixel area, called "refined area", from the determined third pixel area and the determined fourth pixel area. 10. Motor vehicle comprising a camera and a computer, according to the preceding claim, connected to said camera so as to receive at least one sequence of images captured by the camera to estimate the movement of at least one point in said sequence. and allow assistance in driving the vehicle.