EP1920261A1

EP1920261A1 - Method and apparatus for determining motion of a vehicle

Info

Publication number: EP1920261A1
Application number: EP06779110A
Authority: EP
Inventors: Mark Richard Tucker
Original assignee: TRW Ltd
Current assignee: TRW Ltd
Priority date: 2005-08-10
Filing date: 2006-08-09
Publication date: 2008-05-14
Also published as: WO2007017693A1; GB0516403D0

Abstract

A method of determining the motion of a host vehicle (10) relative to a road, comprises obtaining a first image of a scene at a first instant from an image capture device (20) fitted to the host vehicle, obtaining a second image of substantially the same scene at a second instant which is different from the first instant, also from an image capture device fitted to the vehicle identifying at least two features which are present in both the first image and the second image, the features lying in a known plane within the image that is assumed to correspond to the plane containing the road surface; determining the relative movement of the features between obtaining the first image and second image, and combining the determined relative movement of the feature with the time between capture of the first image and the second image to determine the motion of the vehicle relative to the road surface.

Description

METHOD AND APPARATUS FOR DETERMINING MOTION OF A

VEHICLE

This invention relates to improvements in methods and apparatus for determining the motion of a vehicle. It also relates to vehicle control systems that incorporate such methods and apparatus.

It is now well established to provide automotive vehicles with control systems that assist the driver of the vehicle in maintaining control. Most cars are now equipped with antilock brake systems that rapidly release and re-apply the brakes to assist the driver in the event that a wheel is about to lock. An increasing number of manufacturers are also now fitting vehicles with some form of traction control or stability control system. These systems help a driver by reducing the power applied to a wheel through the engine management, or even applying the brakes to one or more wheels to help prevent wheelspin or to control a slide.

In each case, a measure of the motion of the vehicle along the road is needed. Several techniques for achieving this measure are available. In one technique, a measurement of the speed of an individual wheel can be made. Wheel speed information can be obtained through a wheel speed sensor. The vehicle longitudinal motion relative to the ground can be obtained from the vehicle speedometer or from the wheel sensors. However, this information may be inaccurate if the wheels are spinning or the vehicle is sliding such as occurs during extreme cases of oversteer or understeer.

In an alternative technique, a measure of the vehicle ego-motion can be made using additional sensors of yaw or lateral acceleration. Unfortunately, this increases the number of sensors required and increases costs, and even then inaccurate measurements may be obtained if the vehicle is sliding or wheels are spinning.

According to a first aspect the invention provides a method of determining the motion of a host vehicle relative- to a road, the method comprising: obtaining a first image of a scene at a first instant from an image capture device fitted to the host vehicle, obtaining a second image of substantially the same scene at a second instant which is different from the first instant, also from an image capture device fitted to the vehicle; identifying at least two features which are present in both the first image and the second image, the features lying in a known plane within the image that is assumed to correspond to the plane containing the road surface; determining the relative movement of the features between obtaining the first image and second image; and combining the determined relative movement of the feature with the time between capture of the first image and the second image to determine the motion of the vehicle relative to the road surface.

By processing information obtained relating to features in the plane of the road surface taken from images captured from a host vehicle, a robust measure of vehicle motion can be obtained for use by a host of vehicle control systems.

The method may comprise obtaining the first image and the second image using a single device, or it could obtain them using different devices. A series of images may be obtained using a camera fitted to the host vehicle, the method therefore obtaining images as output from the camera. These images may be raw image data or may be pre-processed, perhaps to improve their quality.

The images may each comprise a plurality of image pixels representing the scene covered by the image, the. pixels arranged in an array. For example, an array of 600 by 600 pixels may preferably be captured to make up each image.

The method may comprise obtaining images at first and second instants which are sufficiently close in time, taking into account the maximum expected speed of travel of the host vehicle or maximum expected rotational speed, that the first image and the second image capture scenes always include at least one common portion that corresponds to the same area of road surface in each image. This means the images will always overlap. The slower the vehicle is travelling, the more the images will be of substantially the same portion of road surface.

The method may identify multiple features within each image. All or some of these features may be used to determine the relative movement of features between frames. This may provide increased accuracy in the event that one or more of the features is itself moving relative to the road, since the method relies on identifying stationary features. So, where more than two multiple features are identified, those that are moving can be filtered out and ignored. If only two are identified this is not possible unless certain that it is a stationary feature, such as a road marking.

The relative movement of a feature (or features) between the first and second images may be determined by translation and/or rotation of one feature of set of features until it overlays the other feature or set of features as best as possible with the assumed planes coincident. Alternatively, the images, or the identified features may be passed through a Kalman filter. Such filters are well known for other applications, most famously the tracking of the path and velocity of space rockets . . . .

The method may comprise storing the identified features as a feature set or a map showing the relative position of each feature identified in an image.

The first and second images may be transformed prior to locating features from a perspective view as captured by an image capture device to a plan view of the features. By this we mean the view of the image that would be obtained from a capture device if it were directly overhead of the scene. This can be achieved by transforming the first and second images to a plan view before identifying features, or by identifying features and then transforming.

Assuming the aspect from which the images are obtained is known for each image, a simple predefined transformation can be used to convert the image or features onto a plan view.

The method may determine the plane that is assumed to be the plane in which features lie from information known about the height of device used to capture images, and the direction in which the device faces relative to the vehicle.

The method may be adapted to determine the vehicle velocity relative to the road, its so called road speed. It may additionally or alternatively the adapted to measure motion of the vehicle relative to the road surface, such as its yaw angle. This is the angle between the direction of travel of the host vehicle and the direction in which it is facing. The angle will be greater as the vehicle slides during oversteer or understeer for example.

The method may comprise identifying features which lie within the known plane, which may be a single two-dimensional plane which is assumed to correspond with the plane of the road surface. This is a reasonable assumption for most features, such as lane markings, cats eyes, manhole covers and the like. This assumption has the key benefit of restricting the area of search for features, greatly reducing the amount of processing power needed compared with a system in which no such assumption is made. The method may therefore employ a search strategy to identify features which lie on this plane whilst ignoring any other features.

The method may employ information obtained from one or more sensors associated with the host vehicle to determine the known plane within which features are to be identified. The sensor or sensors may include one or more inclinometers which indicate the inclination of the vehicle relative to the horizontal.

The method may take historical information from the sensor or sensors to determine where the known plane will lie. For example, under harsh acceleration the vehicle may lift at the front, and under hard braking may dive down at the front. In each case, a different plane may be assumed to correspond to the road surface within the image as the camera will view the road surface from a different angle. Output data from an accelerometer or inclinometer can provide this information.

The method may be further enhanced in that the horizon present within an image may be identified, and only the area below this horizon processed to identify features. This will reduce the amount of processing needed, and ensure that features above the road such as trees and lampposts are filtered out.

The assumed known road surface plane may also be employed when the method determines how far a feature has moved between images. Since each image will contain identified features lying on that plane, the movement of the each image needed to provide a best match overlay can be restricted to movements in which the planes co-incide. Again this makes processing easier.

The method may be used to determine the motion of a two-wheeled vehicle such as a motorbike, or a vehicle with three or four or more wheels such as a trike, automobile or truck.

The image may be captured using a camera which is fitted to the vehicle, such as a CCD camera. These devices are well known, and it is possible to provide a robust CCD camera which is suitable for fitting unobtrusively to a vehicle, perhaps behind the rear view mirror for a front mounted camera which would advantageously benefit from a clear view when in the swept area of the windscreen wipers. Other positions could be used, such as behind a radiator grill or other air intake.

According to a second aspect the invention provides a vehicle motion determining system comprising: an image capture device for capturing a first image of a scene at a first instant and a second image of substantially the same scene at a second instant which is different from the first image; and an image processing device which provides: identifying means adapted to identify at least two features which are present in both the first image and the second image, the feature lying in a known plane within the image that is assumed to correspond to the plane containing the road surface; determining means adapted to determine the relative movement of the feature between obtaining the first image and second image; and 5. combining means adapted to combine the determined relative movement of the feature with the time between capture of the first and second images to determine the motion of the vehicle relative to the road surface.

The image capture device may comprise a camera which maybe secured to 0 a portion of a host vehicle and which may have within its field of view a portion of a road surface ahead of the vehicle. It is preferred that it looks ahead to capture a portion of the road, but it could also point straight down, perhaps looking at the road below or to the side of the vehicle.

5 The camera may be a passive device in that it captures images without the need for an additional source of illumination of the scene.

The signal processing device may comprise a part of a vehicle control system. It may comprise an image processing unit which is connected to a 0 camera that captures images. It may be connected through cables or wirelessly. It may be connected to a memory in which images can be stored for processing and to which features identified in images can be stored.

5 According to a third aspect the invention provides a computer program which is stored in a memory and which, when executed by a processor, causes the processor to perform the method of the first aspect of the invention.

0 There will now be described, by way of example only, one embodiment of the present invention with reference to the accompanying drawings of which:

Figure 1 shows an embodiment of a vehicle motion determining system according to a second aspect of the invention fitted to a- host vehicle; - -

Figure 2 is an image of the vehicle from the side showing the area of the road captured by the camera and its height above the road surface;

Figure 3 is a first sample image captured at a first instant from the camera of Figure 1;

Figure 4 shows the location of features identified in the image of Figure 3;

Figure 5 shows the identified features after transformation of the "image" to provide a plan view map of features;

Figure 6 is a second sample image captured at a second instant from the camera of Figure 1;

Figure 7 shows the location of features identified in the second image of Figure 6;

Figure 8 shows the identified features after transformation of the "image" to provide a plan view map of features;

Figure 9 shows an overlay of plan views of features for the images of Figures 3 and 6; and Figure 10 is a flow diagram showing the steps performed by a method according to a first aspect of the invention to determine vehicle motion from the system of Figure 1.

As shown in Figure 1, a camera 20 is fitted to the front of a host vehicle 10. The camera 20 looks forward from behind a rear view mirror. Figure 2 is an illustration of the vehicle from the side which shows the amount of road surface ahead of the vehicle which is imaged by the camera, the camera being approx 30cm above the road and tilted downwards at approx 10 degrees to the horizontal (can you give me better figures for this as I have guessed them) .

Many types of camera can be used, but for sake of example it may be assumed that a CCD camera which comprises an array of detecting elements, which are read out by a standard readout circuit 30. The circuit outputs a series of images, each defined digitally by a frame of data. The digital values making up each image comprising the values of a digital value for each pixel in the array. These frames of image data are held in a temporary memory 35 or buffer for subsequent processing.

The output of the temporary memory 35 is passed to an image processing unit 40. The image processing unit comprises a signal processing device 42 connected to a memory 44. The memory 44 stores program code to be executed by the processing device 42 and space for the storage of image data. Power for the camera and processing unit is supplied from the vehicles 12 volt battery, and may be stepped down to a lower voltage, perhaps 5volts, if needed.

The program executed by the processing unit 40 causes it to perform the steps set out in the form of a block diagram in Figure 10. Firstly, a first image is obtained 70 by capturing the output of the camera at a first instant. This may be at a time t= 0 seconds. Figure 3 shows a typical captured image in which a number or objects including road markings and trees can be seen. These include lane markings and catseyes, as well as a vehicle on the road.

In the next step a number of features 300 are extracted 74 from a first image stored in the memory. This can be seen in Figure 4 with each feature marked by a cross. The features are extracted from specific areas of the image, that are selected to correspond to areas in which it is likely that features will be present. For example, it may be an area in which a feature was identified in a previous frame. To assist in locating features, the search is limited by the assumption that they are constrained within the predefined two-dimensional plane that is assumed to correspond to the surface of the road on which the vehicle is travelling. Clearly this will require assumptions to be made about the gradient of the road and the location and angle of the camera, but nonetheless it is reasonable to make such assumptions.

A number of techniques can be applied in locating features within the assumed plane. These include correlation, corner/edge detection, line detection etc, and are well known per se in the field of image processing.

As mentioned, an assumption has to be made about the relative orientation of the camera and the plane, i.e. the road surface. The extracted features are then transformed to provide a plan view and this view is stored in memory.

In a next step the captured first image is transformed 72 from a perspective view into a virtual plan view of the scene. This can be seen in Figure 5. By this we mean that an image is produced which corresponds to that which would have been captured from a camera fixed above the scene. To do this, a known image plane corresponding to an assumed position of the road surface is used and a transform is applied which gives a plan view with respect to this assumed plane. The assumption of where to place the known plane can be made using knowledge of the height of the camera, its orientation and the orientation and behaviour of the car.

Having identified features and their location within the first frame, the method comprises storing 76 these features and their relative positions in the form of a map in the memory.

Once the features of the first image have been located, the program causes the processing unit to repeat the process for a second image captured 78 at a second time, say time t= 0.05 seconds. This will typically be the next frame produced by the camera, but could be several frames later. Figure 6 shows a sample second image captured at a second instant, clearly showing that the host vehicle has moved along a road. Figure 7 shows the features identified in the image, this time indicated by an "x" at the centre of each image. A perspective transformation is applied 80, and the features are extracted 82 and again transformed to provide a map of features in a plan view and stored 84 in the memory. This is shown in Figure 8 of the accompanying drawings. Note that a different plane for the road surface may have been assumed if the behaviour of the vehicle has changed, i.e. if it is accelerating.

After processing both the first and second images, the memory will contain two plan view "maps" of features extracted from the first and second images respectively, both in a plan view, e.g. looking down onto the assumed plane containing the features.

In the next step, the processing unit overlays 86 the first and second plan views as shown in Figure 9, and transforms 88 the second plan view so as to make it match as best as possible the first view. This is achieved through the application of any combination of rotation and translation of the second image. The match is considered complete when the two sets of features are overlapped as well as can be, or within reason. The overlay of the sample image of Figure 3 with the sample image captured at a later time is shown in Figure 6 of the accompanying drawings.

The rotation and/or translation of the second map of features to match the first (or the other way round if preferred) needed to give the best overlay gives the distance and angle of movement of the camera, and hence its host vehicle, between the capture of the two frames.

In a final step, the processing unit combines the frame rate (i.e. the time between the capture of the first and second frames) with rotation and translation needed for the best match to determine vehicle speed 90. The amount of rotation can also be combined with frame rate to provide a measure of yaw of the vehicle.

In more detail, a number of options for determining vehicle motion from the overlay of the frames can be used. Whilst the invention has been described conceptually in terms of comparing images, it is preferred in practice to use a filter such as a Kalman filter which covers the whole estimation from the image pixel position inputs to velocity and yaw rates. These filters are well known to the man skilled in the art.

Having determined the speed and yaw, these signals may be output to any vehicle control system that requires them, such as a stability control system. They may be averaged before being output if desired.

Claims

1. A method of determining the motion of a host vehicle relative to a road, the method comprising: obtaining a first image of a scene at a first instant.fr.om an image, capture . device fitted to the host vehicle, obtaining a second image of substantially the same scene at a second instant which is different from the first instant, also from an image capture device fitted to the vehicle; identifying at least two features which are present in both the first image and the second image, the features lying in a known plane within the image that is assumed to correspond to the plane containing the road surface; determining the relative movement of the features between obtaining the first image and second image; and combining the determined relative movement of the feature with the time between capture of the first image and the second image to determine the motion of the vehicle relative to the road surface.

2. The method of claim 1 which comprises identifying a plurality of features within each image and using all of the plurality of features to determine the motion of the vehicle relative to the road.

3. The method of claim 1 or 2 which further comprises obtaining the first image and the second image using a single device fitted to the host vehicle.

4. The method of claim 1, 2 or claim 3 which further comprises obtaining images at first and second instants which are sufficiently close in time, taking into account the maximum expected speed of travel of the host vehicle or maximum expected rotational speed, that the first image and the second image capture scenes always include at least one common portion that corresponds to the same area of road surface in each image.

5. The method of any preceding claim which comprises identifying more than two features within each image, all or. some of these features being used to determine the relative movement of features between frames .

6. The method of any preceding claim in which the relative movement of a feature (or features) between the first and second images is determined by translation and/or rotation of one feature of set of features until it overlays the other feature or set of features as best as possible with the assumed planes coincident.

7. The method of any preceding claim in which the relative movement of a feature (or features) between the first and second images is determined by a Kalman filter.

8. The method of any preceding claim which includes a step of storing the identified features as a feature set or a map showing the relative position of each feature identified in an image.

9. The method of any preceding claim in which the first and second images are transformed prior to locating features from a perspective view as captured by an image capture device to a plan view of the features.

10. The method of any preceding claim which further includes determining the vehicle velocity relative to the road.

11. The method of any preceding claim which employs information obtained from one or more sensors associated with the host vehicle to determine the known plane within which features are to be identified.

12. The method of any preceding claim which further includes identifying an the horizon present within an image and only processing the area below this horizon to identify features.

13. A vehicle motion determining system comprising: an image capture device for capturing a first image of a scene at a first instant and a second image of substantially the same scene at a second instant which is different from the first image; and an image processing device which provides: identifying means adapted to identify at least two features which are present in both the first image and the second image, the feature lying in a known plane within the image that is assumed to correspond to the plane containing the road surface; determining means adapted to determine the relative movement of the feature between obtaining the first image and second image; and combining means adapted to combine the determined relative movement of the feature with the time between capture of the first and second images to determine the motion of the vehicle relative to the road surface.

14. A computer program which is stored in a memory and which, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 13.