Classifications

• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01P—MEASURING LINEAR OR ANGULAR SPEED, ACCELERATION, DECELERATION, OR SHOCK; INDICATING PRESENCE, ABSENCE, OR DIRECTION, OF MOVEMENT
  • G01P3/00—Measuring linear or angular speed; Measuring differences of linear or angular speeds
  • G01P3/64—Devices characterised by the determination of the time taken to traverse a fixed distance
  • G01P3/68—Devices characterised by the determination of the time taken to traverse a fixed distance using optical means, i.e. using infrared, visible, or ultraviolet light
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
  • B60W40/10—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to vehicle motion
  • B60W40/105—Speed
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01P—MEASURING LINEAR OR ANGULAR SPEED, ACCELERATION, DECELERATION, OR SHOCK; INDICATING PRESENCE, ABSENCE, OR DIRECTION, OF MOVEMENT
  • G01P3/00—Measuring linear or angular speed; Measuring differences of linear or angular speeds
  • G01P3/64—Devices characterised by the determination of the time taken to traverse a fixed distance
  • G01P3/80—Devices characterised by the determination of the time taken to traverse a fixed distance using auto-correlation or cross-correlation detection means
  • G01P3/806—Devices characterised by the determination of the time taken to traverse a fixed distance using auto-correlation or cross-correlation detection means in devices of the type to be classified in G01P3/68
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/20—Analysis of motion
  • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
  • B60W2420/40—Photo, light or radio wave sensitive means, e.g. infrared sensors
  • B60W2420/403—Image sensing, e.g. optical camera
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
  • B60W30/10—Path keeping
  • B60W30/12—Lane keeping
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30248—Vehicle exterior or interior
  • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
  • G06T2207/30256—Lane; Road marking

Definitions

• This invention relates to improvements in methods and apparatus for determining the motion of a vehicle. It also relates to vehicle control systems that incorporate such methods and apparatus.
• a measure of the motion of the vehicle along the road is needed.
• Several techniques for achieving this measure are available.
• a measurement of the speed of an individual wheel can be made. Wheel speed information can be obtained through a wheel speed sensor. The vehicle longitudinal motion relative to the ground can be obtained from the vehicle speedometer or from the wheel sensors. However, this information may be inaccurate if the wheels are spinning or the vehicle is sliding such as occurs during extreme cases of oversteer or understeer.
• a measure of the vehicle ego-motion can be made using additional sensors of yaw or lateral acceleration. Unfortunately, this increases the number of sensors required and increases costs, and even then inaccurate measurements may be obtained if the vehicle is sliding or wheels are spinning.
• the invention provides a method of determining the motion of a host vehicle relative- to a road, the method comprising: obtaining a first image of a scene at a first instant from an image capture device fitted to the host vehicle, obtaining a second image of substantially the same scene at a second instant which is different from the first instant, also from an image capture device fitted to the vehicle; identifying at least two features which are present in both the first image and the second image, the features lying in a known plane within the image that is assumed to correspond to the plane containing the road surface; determining the relative movement of the features between obtaining the first image and second image; and combining the determined relative movement of the feature with the time between capture of the first image and the second image to determine the motion of the vehicle relative to the road surface.
• the method may comprise obtaining the first image and the second image using a single device, or it could obtain them using different devices.
• a series of images may be obtained using a camera fitted to the host vehicle, the method therefore obtaining images as output from the camera. These images may be raw image data or may be pre-processed, perhaps to improve their quality.
• the images may each comprise a plurality of image pixels representing the scene covered by the image, the. pixels arranged in an array.
• an array of 600 by 600 pixels may preferably be captured to make up each image.
• the method may comprise obtaining images at first and second instants which are sufficiently close in time, taking into account the maximum expected speed of travel of the host vehicle or maximum expected rotational speed, that the first image and the second image capture scenes always include at least one common portion that corresponds to the same area of road surface in each image. This means the images will always overlap. The slower the vehicle is travelling, the more the images will be of substantially the same portion of road surface.
• the method may identify multiple features within each image. All or some of these features may be used to determine the relative movement of features between frames. This may provide increased accuracy in the event that one or more of the features is itself moving relative to the road, since the method relies on identifying stationary features. So, where more than two multiple features are identified, those that are moving can be filtered out and ignored. If only two are identified this is not possible unless certain that it is a stationary feature, such as a road marking.
• the relative movement of a feature (or features) between the first and second images may be determined by translation and/or rotation of one feature of set of features until it overlays the other feature or set of features as best as possible with the assumed planes coincident.
• the images, or the identified features may be passed through a Kalman filter.
• filters are well known for other applications, most famously the tracking of the path and velocity of space rockets . . . .
• the method may comprise storing the identified features as a feature set or a map showing the relative position of each feature identified in an image.
• the first and second images may be transformed prior to locating features from a perspective view as captured by an image capture device to a plan view of the features.
• a perspective view as captured by an image capture device By this we mean the view of the image that would be obtained from a capture device if it were directly overhead of the scene. This can be achieved by transforming the first and second images to a plan view before identifying features, or by identifying features and then transforming.
• the method may determine the plane that is assumed to be the plane in which features lie from information known about the height of device used to capture images, and the direction in which the device faces relative to the vehicle.
• the method may be adapted to determine the vehicle velocity relative to the road, its so called road speed. It may additionally or alternatively the adapted to measure motion of the vehicle relative to the road surface, such as its yaw angle. This is the angle between the direction of travel of the host vehicle and the direction in which it is facing. The angle will be greater as the vehicle slides during oversteer or understeer for example.
• the method may comprise identifying features which lie within the known plane, which may be a single two-dimensional plane which is assumed to correspond with the plane of the road surface. This is a reasonable assumption for most features, such as lane markings, cats eyes, manhole covers and the like. This assumption has the key benefit of restricting the area of search for features, greatly reducing the amount of processing power needed compared with a system in which no such assumption is made.
• the method may therefore employ a search strategy to identify features which lie on this plane whilst ignoring any other features.
• the method may employ information obtained from one or more sensors associated with the host vehicle to determine the known plane within which features are to be identified.
• the sensor or sensors may include one or more inclinometers which indicate the inclination of the vehicle relative to the horizontal.
• the method may take historical information from the sensor or sensors to determine where the known plane will lie. For example, under harsh acceleration the vehicle may lift at the front, and under hard braking may dive down at the front. In each case, a different plane may be assumed to correspond to the road surface within the image as the camera will view the road surface from a different angle. Output data from an accelerometer or inclinometer can provide this information.
• the method may be further enhanced in that the horizon present within an image may be identified, and only the area below this horizon processed to identify features. This will reduce the amount of processing needed, and ensure that features above the road such as trees and lampposts are filtered out.
• the assumed known road surface plane may also be employed when the method determines how far a feature has moved between images. Since each image will contain identified features lying on that plane, the movement of the each image needed to provide a best match overlay can be restricted to movements in which the planes co-incide. Again this makes processing easier.
• the method may be used to determine the motion of a two-wheeled vehicle such as a motorbike, or a vehicle with three or four or more wheels such as a trike, automobile or truck.
• the image may be captured using a camera which is fitted to the vehicle, such as a CCD camera.
• a camera which is fitted to the vehicle
• CCD camera a camera which is fitted to the vehicle
• These devices are well known, and it is possible to provide a robust CCD camera which is suitable for fitting unobtrusively to a vehicle, perhaps behind the rear view mirror for a front mounted camera which would advantageously benefit from a clear view when in the swept area of the windscreen wipers.
• Other positions could be used, such as behind a radiator grill or other air intake.
• the invention provides a vehicle motion determining system comprising: an image capture device for capturing a first image of a scene at a first instant and a second image of substantially the same scene at a second instant which is different from the first image; and an image processing device which provides: identifying means adapted to identify at least two features which are present in both the first image and the second image, the feature lying in a known plane within the image that is assumed to correspond to the plane containing the road surface; determining means adapted to determine the relative movement of the feature between obtaining the first image and second image; and 5. combining means adapted to combine the determined relative movement of the feature with the time between capture of the first and second images to determine the motion of the vehicle relative to the road surface.
• the image capture device may comprise a camera which maybe secured to 0 a portion of a host vehicle and which may have within its field of view a portion of a road surface ahead of the vehicle. It is preferred that it looks ahead to capture a portion of the road, but it could also point straight down, perhaps looking at the road below or to the side of the vehicle.
• the camera may be a passive device in that it captures images without the need for an additional source of illumination of the scene.
• the signal processing device may comprise a part of a vehicle control system. It may comprise an image processing unit which is connected to a 0 camera that captures images. It may be connected through cables or wirelessly. It may be connected to a memory in which images can be stored for processing and to which features identified in images can be stored.
• the invention provides a computer program which is stored in a memory and which, when executed by a processor, causes the processor to perform the method of the first aspect of the invention.
• Figure 1 shows an embodiment of a vehicle motion determining system according to a second aspect of the invention fitted to a- host vehicle; - -
• Figure 2 is an image of the vehicle from the side showing the area of the road captured by the camera and its height above the road surface;
• Figure 3 is a first sample image captured at a first instant from the camera of Figure 1;
• Figure 4 shows the location of features identified in the image of Figure 3
• Figure 5 shows the identified features after transformation of the "image" to provide a plan view map of features
• Figure 6 is a second sample image captured at a second instant from the camera of Figure 1;
• Figure 7 shows the location of features identified in the second image of Figure 6
• Figure 8 shows the identified features after transformation of the "image" to provide a plan view map of features
• Figure 9 shows an overlay of plan views of features for the images of Figures 3 and 6; and Figure 10 is a flow diagram showing the steps performed by a method according to a first aspect of the invention to determine vehicle motion from the system of Figure 1.
• a camera 20 is fitted to the front of a host vehicle 10.
• the camera 20 looks forward from behind a rear view mirror.
• Figure 2 is an illustration of the vehicle from the side which shows the amount of road surface ahead of the vehicle which is imaged by the camera, the camera being approx 30cm above the road and tilted downwards at approx 10 degrees to the horizontal (can you give me better figures for this as I have guessed them) .
• CCD camera which comprises an array of detecting elements, which are read out by a standard readout circuit 30.
• the circuit outputs a series of images, each defined digitally by a frame of data.
• the digital values making up each image comprising the values of a digital value for each pixel in the array.
• These frames of image data are held in a temporary memory 35 or buffer for subsequent processing.
• the output of the temporary memory 35 is passed to an image processing unit 40.
• the image processing unit comprises a signal processing device 42 connected to a memory 44.
• the memory 44 stores program code to be executed by the processing device 42 and space for the storage of image data.
• Power for the camera and processing unit is supplied from the vehicles 12 volt battery, and may be stepped down to a lower voltage, perhaps 5volts, if needed.
• Figure 3 shows a typical captured image in which a number or objects including road markings and trees can be seen. These include lane markings and catseyes, as well as a vehicle on the road.
• a number of features 300 are extracted 74 from a first image stored in the memory.
• the features are extracted from specific areas of the image, that are selected to correspond to areas in which it is likely that features will be present. For example, it may be an area in which a feature was identified in a previous frame.
• the search is limited by the assumption that they are constrained within the predefined two-dimensional plane that is assumed to correspond to the surface of the road on which the vehicle is travelling. Clearly this will require assumptions to be made about the gradient of the road and the location and angle of the camera, but nonetheless it is reasonable to make such assumptions.
• a number of techniques can be applied in locating features within the assumed plane. These include correlation, corner/edge detection, line detection etc, and are well known per se in the field of image processing.
• the captured first image is transformed 72 from a perspective view into a virtual plan view of the scene.
• This can be seen in Figure 5.
• an image is produced which corresponds to that which would have been captured from a camera fixed above the scene.
• a known image plane corresponding to an assumed position of the road surface is used and a transform is applied which gives a plan view with respect to this assumed plane.
• the assumption of where to place the known plane can be made using knowledge of the height of the camera, its orientation and the orientation and behaviour of the car.
• the method comprises storing 76 these features and their relative positions in the form of a map in the memory.
• Figure 6 shows a sample second image captured at a second instant, clearly showing that the host vehicle has moved along a road.
• Figure 7 shows the features identified in the image, this time indicated by an "x" at the centre of each image.
• a perspective transformation is applied 80, and the features are extracted 82 and again transformed to provide a map of features in a plan view and stored 84 in the memory. This is shown in Figure 8 of the accompanying drawings. Note that a different plane for the road surface may have been assumed if the behaviour of the vehicle has changed, i.e. if it is accelerating.
• the memory After processing both the first and second images, the memory will contain two plan view "maps" of features extracted from the first and second images respectively, both in a plan view, e.g. looking down onto the assumed plane containing the features.
• the processing unit overlays 86 the first and second plan views as shown in Figure 9, and transforms 88 the second plan view so as to make it match as best as possible the first view. This is achieved through the application of any combination of rotation and translation of the second image. The match is considered complete when the two sets of features are overlapped as well as can be, or within reason.
• the overlay of the sample image of Figure 3 with the sample image captured at a later time is shown in Figure 6 of the accompanying drawings.
• the rotation and/or translation of the second map of features to match the first (or the other way round if preferred) needed to give the best overlay gives the distance and angle of movement of the camera, and hence its host vehicle, between the capture of the two frames.
• the processing unit combines the frame rate (i.e. the time between the capture of the first and second frames) with rotation and translation needed for the best match to determine vehicle speed 90.
• the amount of rotation can also be combined with frame rate to provide a measure of yaw of the vehicle.
• a number of options for determining vehicle motion from the overlay of the frames can be used. Whilst the invention has been described conceptually in terms of comparing images, it is preferred in practice to use a filter such as a Kalman filter which covers the whole estimation from the image pixel position inputs to velocity and yaw rates. These filters are well known to the man skilled in the art.
• these signals may be output to any vehicle control system that requires them, such as a stability control system. They may be averaged before being output if desired.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Automation & Control Theory (AREA)
• Mathematical Physics (AREA)
• Transportation (AREA)
• Mechanical Engineering (AREA)
• Multimedia (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Theoretical Computer Science (AREA)
• Image Analysis (AREA)
• Image Processing (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=34984368

Family Applications (1)

Country Status (3)

Families Citing this family (11)

Citations (4)

Family Cites Families (3)

• 2005
  • 2005-08-10 GB GBGB0516403.3A patent/GB0516403D0/en not_active Ceased
• 2006
  • 2006-08-09 WO PCT/GB2006/002986 patent/WO2007017693A1/en not_active Ceased
  • 2006-08-09 EP EP06779110A patent/EP1920261A1/de not_active Ceased

Patent Citations (4)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events