FR3069689A1 - METHOD FOR ESTIMATING THE MOVEMENT OF A POINT IN AN IMAGE SEQUENCE - Google Patents

Abstract

The present invention relates to a method for estimating the movement of at least one point in a sequence of images varying in time, said sequence comprising at least a first image (t), a second image (t + n- 1) and a third image (t + n). The method comprises a step of determining, in the third image (t + n), a fourth zone of pixels corresponding to said point by calculating the optical flow between the first image (t) and the third image (t + n) at from a determined first pixel area, and a step of determining, in the third image (t + n), a fifth pixel area, referred to as a "refined area", from a determined third pixel area and the determined fourth pixel area.

Description

The present invention relates to the field of image processing and more specifically relates to a method of estimating the movement of a point in a sequence of images varying over time.

The invention aims in particular to follow a point in a reliable and precise manner in a plurality of consecutive images. The invention applies in particular to a motor vehicle comprising an on-board camera in order, for example, to reconstruct in three dimensions elements of the environment of the vehicle or else to determine the trajectory of the vehicle.

In image processing, the optical flow is an approximation of the movement of the pixels corresponding to the same object observed in a sequence of images varying over time. The optical flow makes it possible in particular to follow a point represented in the sequence of images in order, for example, to detect the movement of the point in the sequence of images, to carry out a segmentation of an object associated with the point followed, to carry out collision time calculations between the vehicle and an object associated with the tracked point (for example another vehicle), etc.

In known manner, an optical flow corresponds to the movement of a point between two images and is expressed as a vector of displacement of the point between said two images.

In a motor vehicle, it is known to use the optical flow to follow points in images generated by a video camera mounted in the vehicle. Such monitoring makes it possible to reconstruct the three-dimensional environment observed by the vehicle or else the trajectory of the vehicle in order in particular to assist the driver in his driving. For example, the reconstruction of the trajectory and / or of the three-dimensional environment of the vehicle respectively allow the tracking of the marking lines on the ground or the detection of an obstacle that can trigger an emergency braking of the vehicle.

Several methods can be used to determine the optical flow. An example of a known method, called the Lukas-Kanade method, is based on a least squares minimization algorithm between two consecutive image sections of the image sequence.

In a known manner, it can be seen that the level of error relating to the value of the flow associated with such an estimate is all the higher, that is to say that the quality of the optical flow is all the lower, the vehicle speed is low or the point you are following is far away, that is to say not very dynamic in the sequence of images. Also, in this case, following a point between two images can lead to determining an erroneous position of the object. Indeed, the optical flow of a point being estimated between two consecutive images, it follows that the errors on the optical flows of several consecutive images, in which one follows the same point, accumulate over the images, causing a drift in tracking the point. Thus, the 3D reconstruction of the environment of the vehicle or the determination of the trajectory of the vehicle can be significantly wrong so that this presents a significant danger when assisting the driver provided to the driver.

The present invention aims to partially remedy these drawbacks by proposing a reliable and effective solution to reduce the drift generated by the accumulation of errors on the optical flows calculated for a sequence of images and to increase the precision in the tracking of a point in a sequence of images.

To this end, the invention firstly relates to a method of estimating the movement of at least one point in a sequence of images varying over time, said sequence comprising at least a first image, a second image and a third image, the second image being after the first image and the third image being after the second image, the method comprising:

• a step of determining, in the first image, at least a first area of pixels corresponding to a point to be followed, • a step of determining, in the second image, of a second area of pixels corresponding to said point by calculating the optical flow between the first image and the second image from the first determined pixel area, • a step of determining, in the third image, a third pixel area corresponding to said point by calculating the optical flow between the second image and the third image from the second determined pixel area. The process is remarkable in that it further comprises:

A step of determining, in the third image, a fourth area of pixels corresponding to said point by calculating the optical flow between the first image and the third image from the first determined area of pixels and • a step of determining, in the third image, a fifth area of pixels, called "refined area", from the third determined area of pixels and the fourth determined area of pixels.

Thus, by using a plurality of optical streams calculated at least from three different images to determine the position of the point in the third image, the method according to the invention makes it possible to avoid the accumulation of estimation errors of successive optical streams which is obtained during monitoring only based on estimates of optical flows between pairs of successive images.

Preferably, the determination of the fifth pixel zone is carried out by averaging the third determined pixel zone and the fourth determined pixel zone.

In one embodiment, the three images are consecutive in the image sequence.

As a variant, intermediate images can exist between the first and / or the second image and / or the third image and the method can then comprise an additional step of estimating the optical flow between this additional image and at least two of the three images mentioned. previously, thus giving a new measure of the position of the point in the third image.

Advantageously, the second image and the third image are consecutive in the image sequence.

In one embodiment, the first image, the second image and the third image are consecutive in the image sequence.

According to one aspect of the invention, the method comprises of the first area of pixels.

According to one aspect of the invention, the method comprises of the second area of pixels,

According to one aspect of the invention, the method comprises of the third area of pixels.

According to one aspect of the invention, the method includes the fourth pixel area.

According to one aspect of the invention, the method comprises of the fifth pixel area.

In one embodiment, the first pixel area constitutes a reference pixel area serving as a basis for determining a plurality of third pixel areas and fourth different pixel areas from a plurality of second images and of third different images.

Advantageously, the reference pixel area is used as long as the distance between the third pixel area and the fourth pixel area is less than a predetermined distance threshold.

Preferably, the predetermined distance threshold corresponds to the noise level of the third image.

The distance between the third pixel zone and the fourth pixel zone can be determined using a Mahalanobis distance (distance standardized by one one one one storage step storage step storage step storage step storage step storage variances) compared to an uncertainty value given by a so-called chi-2 distribution (“chi square”) with 2 degrees of freedom.

Advantageously also, the distance between the third pixel area and the fourth pixel area being less than the predetermined threshold, the reference pixel area is used as long as the appearance between the first pixel area and the fourth pixel area is similar .

By the term "similar" is meant that the pixels of the first pixel area and the pixels of the fourth pixel area are close in terms of values.

In one embodiment, the similarity between the appearance of the first pixel area and the appearance of the fourth pixel area is estimated by calculating signatures based on Haar wavelets on the first pixel area and on the fourth pixel area, and respectively comparing said pixel signatures of each of the areas with each other to determine whether the two pixel areas are sufficiently similar or not. Alternatively, any other suitable means for determining and comparing the appearance of the first pixel area and the appearance of the fourth pixel area could be used.

According to one aspect of the invention, the method further comprises a step of detecting a point estimation defect between two images of the sequence, such a defect possibly notably corresponding to an occultation of the real point by an object or a temporary disappearance of said point.

The invention also relates to a computer for a motor vehicle, intended to estimate the movement of at least one point in a sequence of images varying in time generated by a camera mounted in said vehicle, said sequence comprising at least a first image, a second image and a third image, the second image being posterior to the first image and the third image being posterior to the second image. The computer is configured to determine:

• in the first image, at least a first area of pixels corresponding to a point to be followed, • in the second image, a second area of pixels corresponding to said point by calculating the optical flow between the first image and the second image from the first determined pixel area, • in the third image, a third pixel area corresponding to said point by calculating the optical flow between the second image and the third image from the second determined pixel area. The calculator is remarkable in that it is further configured to determine:

• in the third image, a fourth pixel area corresponding to said point by calculating the optical flow between the first image and the third image from the first determined pixel area, and • in the third image, a fifth pixel area, called "refined area", from the third determined pixel area and from the fourth determined pixel area.

The invention finally relates to a motor vehicle comprising a camera and a computer, as presented above, connected to said camera so as to receive at least one sequence of images captured by the camera in order to estimate the movement of at least one point in said sequence and allow assistance in driving the vehicle.

FIG. 1 schematically illustrates an example of a vehicle according to the invention.

FIG. 2 schematically illustrates an embodiment of the method according to the invention.

FIG. 3 schematically illustrates an example of implementation of the method according to the invention with three consecutive images.

FIG. 4 schematically illustrates an example of implementation of the method according to the invention with three non-consecutive images.

FIG. 5 schematically illustrates successive optical flow estimates between an image t and an image t + n.

FIG. 6A illustrates a first example of application.

FIG. 6B illustrates a second example of application.

Figure 6C illustrates a third application example.

Figure 6D illustrates a fourth example application.

FIG. 6E illustrates a fifth example of application.

Figure 6F illustrates a sixth example application.

Figure 6G illustrates a seventh example application.

FIG. 7 schematically illustrates the estimates of all the successive optical flows between an image t and an image t + n.

Figure 8 illustrates an eighth example application.

FIG. 1 shows an example of a vehicle V according to the invention.

Vehicle V comprises a driving aid system 10 comprising a camera 100 mounted in vehicle V and a computer 200 allowing the processing of the images captured by said camera 100.

The camera 100 can for example be a monocular type camera which can be mounted on the upper part of the front windshield 1A of the vehicle V so as to film the road 2A on which the vehicle V travels as well as the environment E of the vehicle located in the field of vision of the camera 100. It will however be noted that the camera 100 could be mounted at any other suitable location on the vehicle 1.

Preferably, the camera 100 and the computer 200 are implemented by the same physical entity. As a variant, they could constitute two separate physical entities linked via a link or a communication network of the vehicle 1.

The computer 200 is in particular intended to estimate the movement of a point in a sequence of images captured by the camera 100 in order, for example, to reconstruct the three-dimensional environment observed by the vehicle V or else the trajectory of the vehicle V to assist the driver in his conduct.

A sequence of images comprises a succession of images varying in time, generated for example at a number of at least 16 images per second, for example 16, 24, 25, 30, 50 or 100.

Referring to Figures 3 to 5, the image sequence includes at least a first image, a second image and a third image. The second image is after the first image and the third image is after the second image.

In the example in Figure 3, the three images (t, t + 1, t + 2) are consecutive. In the more general example of Figures 4 and 5, where n is a natural integer strictly greater than 1, the three images (t, t + n-1, t + n) are consecutive when n = 2 and nonconsecutive when n is strictly greater than 2.

The computer 200 is configured to perform a plurality of tasks making it possible to estimate the movement of points represented in the sequence of images. To this end, the computer 200 comprises for example one or more processors or one or more microcontrollers as well as electronic components making it possible to implement various functions, for example of filtering signals, in particular signals received from camera 100.

A point corresponds to a real element of the environment of the vehicle filmed by the camera 100. A point is represented in the images in the form of a pixel or a plurality of pixels. For example, as part of driving assistance for vehicle 1, it may be useful to follow points located on a traffic sign, on a traffic light, on a road 2A, on a shoulder, a bridge, etc.

In the following we will present the tracking of the movement of a point in a sequence of images, but it goes without saying that several points could be followed at the same time by duplicating the method described below.

A point is followed in the image sequence using an optical flow algorithm. Several methods can be used to determine the optical flow. An example of a known method, called the Lukas-Kanade method, is based on a least squares minimization algorithm between two consecutive image sections of the image sequence. However, it goes without saying that any other method of tracking the movement of a point in a sequence of images could be used in the context of the present invention.

In order to follow the movement of a point in a sequence of images, the computer 200 is firstly configured to determine in the first image at least one first area of pixels corresponding to a point to follow, to determine in the second image a second area of pixels corresponding to said point by calculating the optical flow between the first image and the second image from the first determined area of pixels, and to determine in the third image a third area of pixels corresponding to said point by calculating the optical flow between the second image and the third image from the second determined pixel area.

Remarkably, the computer 200 is also configured to determine in the third image a fourth area of pixels corresponding to said point by calculating the optical flow between the first image and the third image from the first determined area of pixels, and to determine in the third image, a fifth pixel area, called the "refined area", from the third determined pixel area and from the fourth determined pixel area.

The invention will now be described in its implementation with reference to Figure 2.

In a step E0, the camera 100 firstly generates a sequence of images representing the environment of the vehicle 1, which it transmits to the computer 200.

The so-called extraction noise generated during the optical flow calculation can be modeled according to a Gaussian distribution where "mu" corresponds to the mean of the distribution and "sigma" corresponds to the standard deviation of the distribution. In practice, "mu" corresponds to the position of the point in the image according to its coordinates (u, v) and "sigma" corresponds to the noise generated during the calculation of optical flow (du, dv).

We thus note, with reference to Figures 3 and 4:

• (mu ₀ 1, sigma ₀ 1) the couple of mean and standard deviation calculated between an image 0 and an image 1, • (mm 2, sigmai ₂ ) the couple of mean and standard deviation calculated between an image 1 and an image 2, • (mu ( _n -i) _n, sigma (ni) _n) the couple of mean and standard deviation calculated between an image (t + n-1) and an image (t + n), etc.

We then note:

(mu, sigma

the couple of mean and standard deviation calculated between an image (t) and an image (t-n).

In a step E1, the computer 200 determines (or selects), in the first image, a first area of pixels corresponding to a point to follow.

In a step E2, the computer 200 determines, in the second image, a second area of pixels corresponding to said point by calculating the optical flow between the first image and the second image from the first determined area of pixels.

In a step E3, the computer 200 determines, in the third image, a third area of pixels corresponding to said point by calculating the optical flow between the second image and the third image from the second determined area of pixels.

In a step E4, the computer 200 determines, in the third image, a fourth area of pixels corresponding to said point by calculating the optical flow between the first image and the third image from the first determined area of pixels.

Finally, in a step E5, the computer 200 determines, in the third image, a fifth area of pixels, called “refined area”, from the third determined area of pixels and from the fourth determined area of pixels, preferably by averaging the third pixel area and the fourth pixel area as will be described below.

Implementation 1: three consecutive images in the image sequence

In a first example of implementation, the first image 0, the second image 1 and the third image 2 are consecutive in the sequence of images.

Referring to Figure 3, we then have:

which represents the third pixel area and the corresponding standard deviation, and:

(mu ₀ ^, stgma ₀ 2) which represents the fourth area of pixels and the corresponding standard deviation.

We then deduce:

mu 11 if rames _s ig ma _{0 2} | 1 sigma ^ f sigma ₀ _ ₂ ² multtframes_mu _{0 2} = multifra.mes_sigm.a _{0 2} ^{1 2} sigma ₀₁₂ ² mu ₀ _2 sigma ₀₂ ² which represent the fifth pixel area and the corresponding standard deviation.

Implementation 2: three images not necessarily consecutive in the image sequence

In a second implementation example, we consider that the first image t and the second image (t + n-1) are not consecutive in the image sequence (n then being strictly greater than 2).

With reference to FIG. 4, we then have:

miiojn-i) n = (multif ramesjnu ₀ + mu ^. ^) sïgma ^ jn-ign ⁼ li multiframes_sigm.a _o _ _ljl _ ₁ f + sîgm.a ₀ _ _n ² multiframe s _sigma _On =

1 1 a + 1 o

-: ------------- 2 4 -: ------- sigma _On multif rames_mu _{0 n} = multi fr souls _sigm.a _{0 n} sigma _On

In this embodiment, image 0 constitutes a reference image and n can be modified to successively use a plurality of pairs of consecutive [[t + n-1), (t + n)] images in order to refine the process and reduce the noise level while allowing efficient point tracking.

In particular, image 0 can be chosen as being very old in the chronology of the sequence with respect to the pair of consecutive images [(t + n-1), (t + n)] while having a high similarity with l current image used (t + n), that is to say (with reference to FIG. 5) that the position difference Δ between the third pixel area and the fourth pixel area is small.

When the position difference Δ1 between the third pixel area and the fourth pixel area exceeds a predetermined distance threshold or when the appearance difference Δ2 between the first pixel area and the fourth pixel area exceeds a threshold d 'predetermined appearance, another image of the sequence can be chosen as a reference image.

The position difference Δ1 between the third pixel area and the fourth pixel area can be determined by using, for example, in a manner known per se, a Mahalanobis distance (distance normalized by the variances) compared to an uncertainty value given by a so-called chi-2 distribution with 2 degrees of freedom.

The difference Δ2 in appearance between the first pixel area and the fourth pixel area can be calculated by determining a signature of each pixel in an area, for example using Haar wavelets, and by comparing the pixel signatures of each of the areas together to determine whether the two pixel areas are sufficiently similar or not.

Application cases

In the examples given below, the number of the reference image is annotated with the letter K, the current image is noted n and the intermediate images are noted 2, ..., n-1.

We denote by M (Match) the case where a pixel area in a posterior image of the sequence, determined using the optical flow, corresponds to a pixel area of an earlier image from which said optical flow was calculated. For example, the fourth pixel area determined in the third image corresponds to the first pixel area determined in the first image when the algorithm used for the optical flow gives a valid match, which for example in the case of Lukas Kanade , corresponds to a residual of the least squares below a predetermined threshold.

We denote by F (Fail) the case where an area of pixels in a posterior image of the sequence, determined using the optical flow, does not correspond to an area of pixels of an earlier image from which said flow was calculated. optical. For example, the first determined pixel zone and the fourth determined pixel zone do not correspond when the algorithm used for the optical flow gives an invalid correspondence, which for example, in the case of Lukas Kanade, corresponds to a residue of least squares above a predetermined threshold.

We note FP (Fail but Predicted) the case where the first determined pixel area and the fourth determined pixel area do not correspond, but an external index, for example the three-dimensional reconstruction from an image prior to the fourth image, allows to establish a prediction of the fourth pixel area.

Example A

The application case illustrated in FIG. 6A corresponds to the case for which the position difference Δ1 between the third pixel area of image n obtained from images 1K to n-1 and the fourth pixel area of l image n obtained directly from the reference image 1K is less than or equal to the predetermined threshold.

In this case, the first pixel area (image 1 K) and the fourth pixel area (image n) correspond to one another (M), and the second pixel area (image n-1) and the third pixel area (image n) correspond to each other (M) on the other hand.

Example B

In the case of application illustrated in Figure 6B:

• the first pixel zone of image 1 and the fourth pixel zone of image do not correspond to each other (F);

• the first pixel area of the 2K image and the fourth pixel area (image n) correspond to each other (M) and in this case, the second 2K image constitutes the reference image;

• the second pixel area of image n-1 and the third pixel area of image n correspond to each other (M).

Example C

In the case of application illustrated in Figure 6C:

• the first pixel area of the 1K image and the fourth pixel area of the image n correspond to each other (M);

• the second pixel zone of image n-1 and the third pixel zone of image n do not correspond to each other but a prediction of the third pixel zone exists (FP). By the term “prediction”, it is meant here that, thanks to the preceding acquisitions, it is possible to make a prediction of the third pixel area, for example using a model in which it is considered that the speed of the camera is constant and the object observed is static in the scene (model known as “base prediction”).

Example D

In the case of application illustrated in Figure 6D:

• the first pixel zone of image 1 and the fourth pixel zone of image do not correspond to each other (F);

• the second pixel area of image n-1 and the third pixel area of image n do not correspond to each other (F).

In this case, we can either stop following the point, or make 2D prediction (i.e. using an image-based model) or 3D (i.e. using a model considering l object in its real space).

Example E

In the case of application illustrated in FIG. 6E, it is not possible to determine the fourth pixel area in image n from the first pixel area in image 1, for example because the fourth pixel area appears outside of image n. In this case, the second 2K image can be chosen as the reference image.

In this example:

• the first pixel area of the 2K image and the fourth pixel area of the image do not correspond to each other (F);

• the second pixel zone of image n-1 and the third pixel zone of image n do not correspond to each other (F) because the object has left image n therefore it is no longer possible to continue to make correspondences between image n and the previous ones.

Example F

In the case of application illustrated in FIG. 6F, it is not possible to determine the fourth pixel area in image n from the first pixel area of image 1, nor the third pixel area of image n from the second pixel area of image n-1, for example because the third and fourth pixel areas appear outside of image n.

In this case, we can stop following the point from image n.

Example G

In the case of application illustrated in FIG. 6G, it is not possible to determine the fourth area of pixels in image n from the first area of pixels in image 1 (shaded boxes) because the noise generated by the calculation of the optical flow between the image 1K and the image n is greater than the difference in position between the first area of pixels and the fourth area of pixels.

In this case, the image n can be discarded to follow the point considered and the process is continued using the following image n + 1.

In an exemplary implementation, with reference to FIG. 7, one can use the set of optical flows calculated between all the images of the sequence in order to further refine the precision of the tracking of the point.

With reference to FIG. 8, the correspondences (M) between the images can be used to detect an occlusion. When the point is temporarily hidden by another object, the matching algorithm will no longer work. However, it is possible to continue to predict the point followed in order to find it later. An occlusion will display some images for which the process could not match. There was an occlusion of the characteristic during frames n-2 and n-1 (boxes F in Figure 8). Then, as soon as the point reappears (images n and n + 1), the correspondence can again be carried out.

By combining optical streams of different images to follow the same point in different ways, the method according to the invention makes it possible to reduce the impact of noise on point tracking, which in particular improves the accuracy of tracking and therefore of the driving assistance.

The method according to the invention is particularly suitable for tracking points used in a driving assistance system for a motor vehicle. It is also advantageously applicable to point tracking used in radar or lidar type systems.

Claims (10)

1. Method for estimating the movement of at least one point in a sequence of images varying over time, said sequence comprising at least a first image (t), a second image (t + n-1) and a third image (t + n), the second image (t + n-1) being posterior to the first image (t) and the third image (t + n) being posterior to the second image (t + n-1), the process comprising: • a step of determining, in the first image (t), at least a first area of pixels corresponding to a point to follow, • a step of determining, in the second image (t + n-1), of a second pixel area corresponding to said point by calculating the optical flow between the first image (t) and the second image (t + n-1) from the first determined pixel area, • a determination step, in the third image (t + n), of a third pixel area corresponding to said point by calculating the optical flow between the second image (t + n-1) and the third image (t + n) from the second pixel area determined, said method being characterized in that it further comprises: A step of determining, in the third image (t + n), of a fourth area of pixels corresponding to said point by calculating the optical flow between the first image (t) and the third image (t + n) from the first determined pixel area, • a step of determining, in the third image (t + n), a fifth pixel area, called the "refined area", from the third determined pixel area and from the fourth determined pixel area. 2. Method according to the preceding claim, in which the second image (t + n-1) and the third image (t + n) are consecutive in the image sequence. 3. Method according to the preceding claim, wherein the first image (t), the second image (t + n-1) and the third image (t + n) are consecutive in the image sequence. 4. Method according to one of the preceding claims, comprising a step of storing the first pixel area, the second pixel area, the third pixel area and the fourth pixel area. 5. Method according to the preceding claim, comprising a step of storing the fifth area of pixels. 6. Method according to one of the preceding claims, in which the first pixel area constitutes a reference pixel area serving as a basis for determining a plurality of third pixel areas and fourth different pixel areas from 'a plurality of second images (t + n-1) and third images (t + n) different. 7. Method according to the preceding claim, in which the reference pixel area is used as long as the distance between the third pixel area and the fourth pixel area is less than a predetermined threshold. 8. Method according to the preceding claim, wherein the predetermined threshold corresponds to the noise level of the third image (t + n). 9. Motor vehicle computer, intended to estimate the movement of at least one point in a time-varying sequence of images generated by a camera mounted in said vehicle, said sequence comprising at least a first image (t), a second image (t + n-1) and a third image (t + n), the second image (t + n-1) being after the first image (t) and the third image (t + n) being after the second image (t + n-1), the computer being configured for: • determine, in the first image (t), at least a first area of pixels corresponding to a point to be followed, • determine, in the second image (t + n-1), a second area of pixels corresponding to said point by calculating the optical flow between the first image (t) and the second image (t + n-1) from the first determined pixel area, • determining, in the third image (t + n), a third corresponding pixel area at said point by calculating the optical flow between the second image (t + n-1) and the third image (t + n) from the second determined pixel area, said computer being characterized in that it is further configured for : • determine, in the third image (t + n), a fourth area of pixels corresponding to said point by calculating the optical flow between the first image (t) and the third image (t + n) from the first area of pixels determined, • determine, in the third image (t + n), a fifth pixel area, called the "refined area", from the third determined pixel area and from the fourth determined pixel area. 10. Motor vehicle comprising a camera and a computer, according to the preceding claim, connected to said camera so as to receive at least one sequence of images captured by the camera in order to estimate the movement of at least one point in said sequence and allow assistance in driving the vehicle.