WO2011027907A1 - Vehicle vicinity monitoring apparatus - Google Patents

Vehicle vicinity monitoring apparatus Download PDF

Info

Publication number
WO2011027907A1
WO2011027907A1 PCT/JP2010/065452 JP2010065452W WO2011027907A1 WO 2011027907 A1 WO2011027907 A1 WO 2011027907A1 JP 2010065452 W JP2010065452 W JP 2010065452W WO 2011027907 A1 WO2011027907 A1 WO 2011027907A1
Authority
WO
WIPO (PCT)
Prior art keywords
shape
lower body
bicycle rider
vehicle
reference template
Prior art date
Application number
PCT/JP2010/065452
Other languages
French (fr)
Inventor
Yuji Yokochi
Atsuhiro Eguchi
Masaki Negoro
Fuminori Taniguchi
Tomokazu Takagi
Naoto Akutsu
Original Assignee
Honda Motor Co., Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Honda Motor Co., Ltd. filed Critical Honda Motor Co., Ltd.
Priority to EP10813852.0A priority Critical patent/EP2465092B1/en
Priority to JP2012505916A priority patent/JP5270794B2/en
Priority to CN201080036182.8A priority patent/CN102473281B/en
Priority to US13/391,395 priority patent/US20120147188A1/en
Publication of WO2011027907A1 publication Critical patent/WO2011027907A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/50Context or environment of the image
    • G06V20/56Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/20Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/22Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
    • B60R1/23Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
    • B60R1/24Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view in front of the vehicle
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/20Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/30Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles providing vision in the non-visible spectrum, e.g. night or infrared vision
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20Movements or behaviour, e.g. gesture recognition
    • G06V40/23Recognition of whole body movements, e.g. for sport training
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/10Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
    • B60R2300/105Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using multiple cameras
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/10Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
    • B60R2300/107Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using stereoscopic cameras
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/20Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of display used
    • B60R2300/205Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of display used using a head-up display
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/80Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
    • B60R2300/8093Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for obstacle warning

Definitions

  • the present invention relates to a vehicle vicinity monitoring apparatus for extracting an object from an infrared image (grayscale image) captured by infrared cameras and a binary image converted from the grayscale image .
  • the known vehicle vicinity monitoring apparatus extract the object from images (a grayscale image and a binary image thereof) in the vicinity of the vehicle captured by infrared cameras, and provide the driver of the vehicle with information about the object.
  • a vehicle vicinity monitoring apparatus regards high-temperature areas of images in the vicinity of the vehicle captured by a pair of left and right infrared cameras ( stereographic cameras) as objects, and
  • the vehicle vicinity monitoring apparatus calculates the distances up to the objects by determining the parallax of the objects between the captured images.
  • the vehicle vicinity monitoring apparatus detects objects that may affect the travel of the vehicle based on the directions in which the objects move and the positions of the objects, and outputs a warning about the detected objects (see U.S. Patent Application
  • the vehicle vicinity monitoring apparatus disclosed in U.S. Patent Application Publication No. 2005/0063565 Al determines whether the brightness dispersion of an image of an object is acceptable or not based on the result of comparison between a feature quantity of the object in a binarized image and a feature quantity of the object in a grayscale image, and changes a process of recognizing pedestrians, for thereby increasing the reliability of pedestrian recognition (see paragraphs [0244] through [0246] of U.S. Patent Application
  • the vehicle vicinity monitoring apparatus disclosed in U.S. Patent Application Publication No. 2005/0276447 Al extracts objects to be binarized from a grayscale image and compares a feature value of the extracted objects with a feature value of the legs of a pedestrian stored in a pedestrian leg feature value storage means, for thereby determining whether the objects to be
  • the vehicle vicinity monitoring apparatus judges that the objects to be binarized are pedestrian legs, then the vehicle vicinity monitoring apparatus recognizes an object including the objects to be binarized as a pedestrian and hence recognizes a pedestrian in the vicinity of the vehicle (see paragraphs [0012] and [0117] of U.S. Patent Application Publication No. 2005/0276447 Al).
  • the vehicle vicinity monitoring apparatus of the related art described above are an excellent system for displaying an image of a less-visible pedestrian ahead of the vehicle which has been detected as an object while the vehicle is traveling at night , and notifying the driver of the presence of a pedestrian with sound and a displayed highlighted frame, for example.
  • the vehicle vicinity monitoring apparatus of the related art still have much to be improved about the detection of a bicycle rider who is pedaling a bicycle at night .
  • a vehicle vicinity monitoring apparatus for detecting a bicycle rider who is pedaling a bicycle, as an object, from images captured by infrared cameras mounted on a vehicle, comprising an upper body and lower body area identifying device for identifying an area including upper and lower bodies estimated as the object from the images, a shape change detecting device for detecting time-dependent changes of upper and lower body shapes in the identified area including the upper and lower bodies, a difference acquiring device for acquiring a difference between the upper and lower body shapes at each of the detected time-dependent changes, and a bicycle rider determining device for judging the object as the bicycle rider if the amplitude of the difference is of a value greater than a threshold value.
  • the vehicle vicinity monitoring apparatus detects the bicycle rider as the object.
  • grayscale images acquired by the infrared cameras or binarized images produced by binarizing the grayscale images may be used as the above images acquired by the infrared cameras .
  • the bicycle rider can appropriately be estimated if the object estimated by the upper body and lower body area identifying device has a feature that the upper body has a smaller time-dependent change and the lower body has a greater time-dependent change.
  • the shape change detecting device may have a
  • reference template including a reference bicycle rider shape made up of an upper body shape and a lower body shape, and detect the time-dependent changes of the upper body shape and the lower body shape in the identified area including the upper and lower bodies by subtracting the upper body shape and the lower body shape in the identified area including the upper and lower bodies from the upper body shape and the lower body shape in the reference bicycle rider shape.
  • the reference template provided in the vehicle vicinity monitoring apparatus may be a reference template of grayscale image if the area including the upper body and the lower body is
  • the reference template may be a reference template of
  • binarized image if the area including the upper body and the lower body is represented by a binarized image.
  • the reference template may comprise a first
  • reference template including a first reference bicycle rider shape in which a right foot is positioned upwardly of a left foot in the lower body shape as viewed in front elevation
  • second reference template including a second reference bicycle rider shape in which a right foot is positioned downwardly of a left foot in the lower body shape as viewed in front elevation
  • the second reference template being the left-right reversal of the first reference template
  • apparatus may further comprise a direction-of-travel detecting device for detecting a change in the direction of travel of the bicycle rider which is the object, if each of the time-dependent changes of the upper and lower body shapes in the identified area including the upper and lower bodies, detected by the shape change detecting device , abruptly changes .
  • a direction-of-travel detecting device for detecting a change in the direction of travel of the bicycle rider which is the object, if each of the time-dependent changes of the upper and lower body shapes in the identified area including the upper and lower bodies, detected by the shape change detecting device , abruptly changes .
  • the upper body and lower body area identifying device may include an upper body area identifying device for identifying an upper body area of the bicycle rider, i.e., the object, if three high- brightness regions estimated as a head and a right hand and a left hand which grip the handle of the bicycle are detected.
  • the vehicle vicinity monitoring apparatus can detect such a bicycle rider as an object highly
  • FIG. 1 is a block diagram of a vehicle vicinity monitoring apparatus according to an embodiment of the present invention
  • FIG. 2 is a schematic perspective view of a vehicle incorporating the vehicle vicinity monitoring apparatus shown in FIG. 1;
  • FIG. 3 is a diagram showing reference templates
  • FIG. 4 is a diagram showing another reference template having a different direction
  • FIG. 5 is a flowchart of an operation sequence of an image processing unit for detecting and determining an object such as a bicycle rider
  • FIG. 6 is a diagram showing a succession of
  • grayscale images that are obtained as respective frames ordered in time from above;
  • FIG. 7A is a diagram showing a succession of image areas including upper and lower bodies identified from respective grayscale images
  • FIG. 7B is a diagram showing a succession of
  • FIG. 8A is a diagram which is illustrative of the principles of an upper and lower body area identifying device for identifying areas including upper and lower bodies estimated as objects from grayscale images;
  • FIG. 8B is a diagram which is illustrative of the principles of an upper and lower body area identifying device for identifying areas including upper and lower bodies estimated as objects from binary images;
  • FIG. 9 is a diagram which is illustrative of the calculation of correlated errors corresponding to shape changes at times ;
  • FIG. 10 is a diagram showing upper body correlated errors, lower body correlated errors, and differential correlated errors thereof;
  • FIG. 11A is a diagram showing how the head of a bicycle rider moves with respect to the road
  • FIG. 11B is a diagram showing how the head of a person moves with respect to the road while the person is jogging.
  • FIG. 12 is a diagram showing correlated errors at the time a bicycle rider changes its direction of travel. Description of Embodiments
  • FIG. 1 is a block diagram of a vehicle vicinity monitoring apparatus 10 according to an embodiment of the present invention
  • FIG. 2 is a schematic perspective view of a vehicle 12 incorporating the vehicle vicinity monitoring apparatus 10 shown in FIG. 1.
  • the vehicle vicinity monitoring apparatus 10 comprises an image processing unit 14 for controlling the vehicle vicinity monitoring apparatus 10, a pair of right and left infrared cameras 16R, 16L connected to the image processing unit 14, a vehicle speed sensor 18 for detecting a vehicle speed Vs of the vehicle 12, a brake sensor 20 for detecting an operated amount (brake operated amount) Br of the brake pedal depressed by the driver of the vehicle 12, a yaw rate sensor 22 for detecting a yaw rate Yr of the vehicle 12, a speaker 24 for issuing an audible warning, and an image display device 26 including a HUD (Head Up Display) 26a for displaying an image captured by the infrared cameras 16R, 16L to enable the driver to visually
  • a HUD Head Up Display
  • the image display device 26 may have a navigation system display unit rather than the HUD 26a.
  • the image processing unit 14 detects a moving object such as a pedestrian, a bicycle rider, or the like ahead of the vehicle 12 from infrared images in the vicinity of the vehicle 12 and signals (vehicle speed Vs, brake operated amount Br, and yaw rate Yr in the illustrated embodiment) representative of the traveling state of the vehicle 12, and issues a warning when the detected moving object is highly likely to stay in a collision course with the vehicle 12.
  • a moving object such as a pedestrian, a bicycle rider, or the like ahead of the vehicle 12 from infrared images in the vicinity of the vehicle 12 and signals (vehicle speed Vs, brake operated amount Br, and yaw rate Yr in the illustrated embodiment) representative of the traveling state of the vehicle 12, and issues a warning when the detected moving object is highly likely to stay in a collision course with the vehicle 12.
  • the image processing unit 14 comprises an A/D converter for converting input analog signals into digital signals, an image memory (storage unit 14m) for storing digital image signals, a CPU (Central Processing Unit) 14c for performing various processing operations, a storage unit 14m including a RAM (Random Access Memory) for storing data as they are being processed by the CPU 14c and a ROM (Read Only Memory) for storing programs executed by the CPU 14c, tables, ma s, etc., and an output circuit for outputting drive signals for the speaker 24, display signals for the image display device 26, and other signals.
  • the infrared cameras 16R, 16L, the yaw rate sensor 22, the vehicle speed sensor 18, and the brake sensor 20 produce output signals which are converted by the A/D converter into digital signals to be supplied to the CPU 14c.
  • the infrared cameras 16R, 16L which function as so-called stereographic cameras, are disposed on a front bumper of the vehicle 12 in
  • the infrared cameras 16R, 16L have respective optical axes extending parallel to each other and are positioned at the same height from the road on which the vehicle 12 travels.
  • the infrared cameras 16R, 16L have such characteristics that their output signals are higher in level as the temperature of objects captured thereby is higher .
  • the HUD 26a has its display screen displayed on the front windshield of the vehicle 12 at a position out of the front field of vision of the driver of the vehicle 12.
  • the ROM (storage unit 14m) of the image processing unit 14 stores therein a reference template 104 which comprises a first reference template 104a including a reference bicycle rider shape (first reference bicycle rider shape) 103a made up of an upper body shape 100 and a lower body shape 102a, and a second reference template 104b including a reference bicycle rider shape (second reference bicycle rider shape) 103b made up of an upper body shape 100 and a lower body shape 102b.
  • Each of the first reference template 104a and the second reference template 104b is a binary image
  • first reference template 104a and the second reference template 104b may be stored in the storage unit 14m.
  • the upper body shapes 100 which represent a head and a torso including hands , as viewed in front elevation in the direction from the bicycle rider toward the vehicle 12, are identical with each other, including a right hand Rh and a left hand Lh for gripping the bicycle handle.
  • the template 104a includes, in the lower body shape 102a as viewed in front elevation, the first reference bicycle rider shape 103a in which a right foot (right leg) Rf is positioned above a left foot (left leg) Lf .
  • the second reference template 104b is the left-right reversal of the first reference template 104a (i.e., one obtained by left-right inverting the first reference template 104a) , and includes, in the lower body shape 102b as viewed in front elevation, the second reference bicycle rider shape 103b in which a right foot Rf is positioned below a left foot Lf .
  • the storage unit 14m also stores therein reference templates, not shown, which correspond to the first and second reference templates , as viewed in front elevation in the direction away from the vehicle 12, a reference template 112 (see FIG. 4) including a reference bicycle rider shape 110 made up of an upper body shape 106 and a lower body shape 108 as viewed in front elevation in the direction across in front of the vehicle 12, and a reference template which is the left-right reversal of the reference template 112 (i.e., obtained by left-right inverting the reference template 112), as viewed in front elevation in the traveling direction opposite to the direction of the reference bicycle rider shape 110 shown in FIG. 4.
  • reference templates not shown, which correspond to the first and second reference templates , as viewed in front elevation in the direction away from the vehicle 12, a reference template 112 (see FIG. 4) including a reference bicycle rider shape 110 made up of an upper body shape 106 and a lower body shape 108 as viewed in front elevation in the direction across in front of the vehicle 12, and a reference template which
  • FIG. 5 is a flowchart of an operation sequence of the image processing unit 14 for detecting
  • step SI shown in FIG. 5 the image processing unit 14 acquires infrared images in a certain range of field angle in front of the vehicle, which are captured in frames by the infrared cameras 16R, 16L and
  • the grayscale image is produced by the infrared camera 16R, and the left grayscale image is produced by the infrared camera 16L.
  • the right grayscale image and the left grayscale image include an object at different horizontal positions therein. The distance from the vehicle 12 up to the object can be calculated based on the difference (parallax) between the different horizontal positions.
  • the right grayscale image which is produced by the infrared camera 16R, is used as a reference image to binarize its image signal, i.e., any area of the
  • each of the captured frames is converted into a binarized image.
  • FIG. 6 shows a succession of grayscale images 30a through 30f that are obtained as respective frames ordered in time from above.
  • step S2 candidates for an object, i.e., a bicycle rider who is pedaling a bicycle , are extracted for detecting the object.
  • the object candidates are represented by high-brightness areas (white areas in binarized images) that are extracted from the grayscale images 30a through 30f or binarized images (not shown) produced therefrom.
  • step S3 upper body and lower body area
  • an area including an upper body and a lower body that is estimated as an object is
  • FIG. 7A shows a succession of image areas PI through P6 including upper and lower bodies identified from the respective grayscale images 30a through 30f .
  • FIG. 7B shows a succession of binarized images 31abn through 31fbn in image areas Plb through P6b corresponding respectively to grayscale images 31a through 3If (FIG. 7A) in the image areas PI through P6.
  • the binarized images 31abn through 31fbn may be interpreted as
  • An upper body and lower body area identifying device for identifying an area including upper and lower bodies estimated as the object from the grayscale images 30a through 30f includes an upper body area
  • the upper body area identifying device identifies an upper body area Pu of the bicycle rider, i.e., the object, if three high-brightness regions estimated as a head 50 and a right hand 51R and a left hand 51L which grip the handle are detected, as indicated in grayscale images 70, 72 of areas Pt , Pt + At at respective times t, t + At in FIG. 8A.
  • the upper body area identifying device is also able to identify three high-brightness regions corresponding to the head 50 and the right hand 51R and the left hand 51L which grip the handle, from binarized images 70bn, 72bn converted from the grayscale images 70, 72.
  • the upper body and lower body area identifying device also includes a lower body area identifying device for identifying a lower body area Pd of the bicycle rider below the upper body area Pu.
  • the lower body area Pd includes two high-brightness regions estimated as both feet (both legs), i.e., a right foot (right leg) 52R and a left foot (left leg) 52L.
  • the two high-brightness regions make shape changes (vertical pedaling movement) and periodic movement (vertical pedal movement) when the object is moving.
  • the upper body area Pu includes the three high-brightness regions, which make no or little shape changes when the object is moving.
  • the lower body area identifying device is also able to identify an area including two high-brightness regions corresponding to the both feet (both legs), i.e., the right foot (right leg) 52R and the left foot (left leg) 52L, which make shape changes (vertical pedaling
  • step S4 shape change detecting device
  • the shape changes are detected as correlated errors between the first reference template 104a and the binarized images 31abn through 31fbn in the areas Plb through P6b, i.e., interframe correlated errors.
  • an upper body correlated error Eu and a lower body correlated error Ed are calculated as the correlated errors .
  • the upper body correlated error Eu is calculated as correlated errors between the upper body shape 100 of the first reference template 104a and the upper body areas Pu of the binarized images 31abn through 31fbn in the image areas Plb through P6b, e.g., the sum of squares of the differences between the
  • the lower body correlated error Ed is calculated as correlated errors between the lower body shape 102a of the first reference template 104a and the lower body areas Pd of the binarized images 31abn through 31fbn in the image areas Plb through P6b.
  • FIG. 9 is illustrative of a calculation of
  • the shapes of the lower body areas Pd of the binarized images in the areas Ptl, Pt3 are largely different from (i.e., in opposite phase to) the lower body shape (lower body reference shape) 102a of the first reference template 104a. At times tl and t3, therefore, the lower body correlated error Ed is maximum. At times t2 and t4, the shapes of the lower body areas Pd of the binarized images in the areas Pt2, Pt4 are
  • the lower body correlated error Ed is minimum.
  • correlated error Eu remains substantially nil at all times as the upper body of the bicycle rider does not move and is kept in the same shape.
  • step S5 difference between the correlated errors of the upper body shape and the lower body shape is calculated
  • FIG. 10 shows a simulation result for easier
  • both the upper body correlated error Eu and the lower body correlated error Ed tend to increase in frames ranging from frame number 0 to frame number 180.
  • the increasing tendency is caused for the following reasons: First, as the first reference template is enlarged, aliasing (blurring) becomes large, thereby increasing the differential error. Secondly, there is an increasing error related to a change in the brightness of the object depending on the distance between the object and the vehicle. Thirdly, there is an increasing error due to a background change (noise) .
  • the lower body correlated error Ed is larger in level than the upper body correlated error Eu because the lower body moves greater than the upper body in each of the frames. Additionally, the lower body correlated error Ed has another feature that the amplitude thereof periodically increases and decreases in synchronism with the bicycle rider's pedaling action.
  • step S5 the difference (correlated error
  • is calculated by subtracting the upper body correlated error Eu from the lower body correlated error Ed.
  • the difference ⁇ is appropriately detected as having periodic amplitude increases and decreases .
  • the object is basically judged as a bicycle rider.
  • step S7 it is determined whether the amplitude period of the difference ⁇ falls within a period
  • step S8 If the amplitude period of the difference ⁇ falls within the period corresponding to the speed of the bicycle, then it is determined whether the upper body shape moves vertically or not in step S8.
  • H Hconst
  • the head 64 of a person who is jogging moves vertically, and the road height H of the head 64 is represented by a periodically varying height Hvar .
  • step S8 If it is judged that the upper body shape does not vertically move in step S8 , then the object is finally judged as a bicycle rider in step S9.
  • the object When the object is finally judged as a bicycle rider, if the object is likely to be in a collision course with the vehicle 12 based on the operated amount Br output from the brake sensor 20, the vehicle speed Vs output from the vehicle speed sensor 18, the yaw rate Yr output from the yaw rate sensor 22, and the distance from the vehicle 12 up to the object (i.e., the bicycle rider) , then the grayscale image of the bicycle rider is displayed on the HUD 26a, and an audible warning is issued through the speaker 24, prompting the driver of the vehicle 12 to take an action to avoid the possible collision with the bicycle rider. If the driver of the vehicle 12 appropriately brakes the vehicle 12 and there is no possibility of collision, then no audible warning is issued through the speaker 24, so that the driver will not be unnecessarily troubled.
  • step S10 If the answer to either one of steps S6, S7, S8 is negative, then the object is finally judged as something other than a bicycle rider.
  • the process of determining a pedestrian according to the related art may be carried out after step S10.
  • the vehicle vicinity monitoring apparatus 10 converts grayscale images acquired by the infrared cameras 16R, 16L mounted on the vehicle 12 into binarized images.
  • the upper body and lower body area identifying device identifies an area including upper and lower bodies estimated as the object from the binarized images.
  • the shape change detecting device detects time- dependent changes of the upper and lower body shapes in the identified area including the upper and lower bodies.
  • the difference acquiring device acquires the difference between the upper and lower body shapes at each of the detected time-dependent changes.
  • step S6 judges the object as the bicycle rider.
  • the vehicle vicinity monitoring apparatus 10 is thus able to detect a bicycle rider near the vehicle 12 at night.
  • the upper body and lower body area identifying device (step S3) can appropriately estimate the object as a bicycle rider if the object has a feature that the upper body has a smaller time-dependent change and the lower body has a greater time-dependent change.
  • the shape change detecting device (step S4) has the reference template 104 including the reference bicycle rider shape made up of a certain upper body shape and a certain lower body shape .
  • detecting device can detect time-dependent changes of the upper body shape and the lower body shape in the
  • identified area including the upper and lower bodies by subtracting the upper body shape and the lower body shape in the identified area including the upper and lower bodies from the upper body shape and the lower body shape in the reference bicycle rider shape.
  • the reference template 104 comprises the first reference template 104a including the first reference bicycle rider shape wherein the right foot is positioned upwardly of the left foot in the lower body shape as viewed in front elevation, and the second reference template 104b including the second reference bicycle rider shape wherein the right foot is positioned
  • the shape change detecting device can detect time-dependent changes of the upper body shape and the lower body shape, using the first and second reference templates 104a, 104b.
  • the upper body and lower body area identifying device (step S3) identifies an area including upper and lower bodies estimated as the object from the binarized images .
  • the upper body and lower body should preferably include the upper body area identifying device for identifying the upper body area of the bicycle rider as the object if three high-brightness regions estimated as the head and the right hand and the left hand which grip the handle are detected.
  • the vehicle vicinity monitoring apparatus 10 can detect such a bicycle rider as an object highly accurately from images output from infrared cameras .
  • the vehicle vicinity monitoring apparatus 10 converts grayscale images acquired by the infrared cameras 16R, 16L mounted on the vehicle 12 into binarized images.
  • the upper body and lower body area identifying device For detecting an object, i.e., a bicycle rider who is pedaling a bicycle, from the binarized images, the upper body and lower body area identifying device (step S3) identifies an area including upper and lower bodies estimated as the object from the binarized images.
  • the upper body and lower body area identifying device may identify an area including upper and lower bodies estimated as the object from the grayscale images acquired by the infrared cameras 16R, 16L.
  • the shape change detecting device (step S4) detects time-dependent changes of the grayscale images of the upper and lower body shapes in the
  • the difference acquiring device acquires the difference between the upper and lower body shapes in the grayscale images at each of the detected time-dependent changes. If the amplitude of the difference is of a value greater than the threshold value, then the bicycle rider determining device (step S6) judges the object as the bicycle rider.
  • time-dependent shape changes of the binarized images converted from the grayscale images are detected based on the reference templates (the first and second reference templates 104a, 104b) of binarized image schematically shown in FIG. 3.
  • a reference template of grayscale image may be used instead of the reference template of binarized image.
  • time-dependent shape changes in the grayscale images 31b through 31f of the areas P2 through P6 may be detected using, as a reference template, the grayscale image 31a of the area PI among the areas PI through P6 (see FIG. 7A) including the upper and lower bodies identified from the grayscale images 30a through 30f (see FIG. 6) acquired by the infrared cameras 16R, 16L.
  • the binarized image 31abn in the area Plb (see FIG. 7B) corresponding to the grayscale images 31a through 31f in the area PI may be used as a reference template.
  • the vehicle vicinity monitoring apparatus 10 should preferably further include a direction-of-travel
  • the detecting device for detecting a change in the direction of travel of the object, i.e., the bicycle rider, if the correlated error Ed derived from time-dependent changes of the upper and lower body shapes in the identified area including the upper and lower bodies, detected by the shape change detecting device, abruptly changes, as indicated by a correlated error Edx between time tl2 and time tl3 in FIG. 12.
  • a highly reliable correlated error Ed in areas Ptll, Ptl2 is detected based on the first reference template 104a until a time shortly after time tl2.
  • the direction of travel of the bicycle rider is estimated as having changed from a front-back direction to a left-right direction after time tl2, and the reference template is updated from the first reference template 104a to the reference template 112 shown in FIG. 4, after which the processing in steps SI through S9 is performed. In this manner, the period of movement is extracted again as indicated by the

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • Mechanical Engineering (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Psychiatry (AREA)
  • Human Computer Interaction (AREA)
  • Social Psychology (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)
  • Traffic Control Systems (AREA)
  • Closed-Circuit Television Systems (AREA)

Abstract

A vehicle vicinity monitoring apparatus detects a bicycle rider from infrared images. The vehicle vicinity monitoring apparatus converts the infrared images into binarized images, and calculates correlated errors (Ed) of areas (Pt1 through Pt4) from a reference template (104a). The vehicle vicinity monitoring apparatus judges an object of interest as the bicycle rider when the amplitude of the correlated errors (Ed) changes periodically.

Description

DESCRIPTION
Title of Invention
VEHICLE VICINITY MONITORING APPARATUS
Technical Field
The present invention relates to a vehicle vicinity monitoring apparatus for extracting an object from an infrared image (grayscale image) captured by infrared cameras and a binary image converted from the grayscale image .
Background Art
Heretofore, there have been known vehicle vicinity monitoring apparatus for monitoring an object such as a pedestrian or the like that may possibly be on a
collision course with the vehicle. The known vehicle vicinity monitoring apparatus extract the object from images (a grayscale image and a binary image thereof) in the vicinity of the vehicle captured by infrared cameras, and provide the driver of the vehicle with information about the object.
A vehicle vicinity monitoring apparatus regards high-temperature areas of images in the vicinity of the vehicle captured by a pair of left and right infrared cameras ( stereographic cameras) as objects, and
calculates the distances up to the objects by determining the parallax of the objects between the captured images. The vehicle vicinity monitoring apparatus then detects objects that may affect the travel of the vehicle based on the directions in which the objects move and the positions of the objects, and outputs a warning about the detected objects (see U.S. Patent Application
Publications No. 2005/0063565 Al and No. 2005/0276447 Al) .
The vehicle vicinity monitoring apparatus disclosed in U.S. Patent Application Publication No. 2005/0063565 Al determines whether the brightness dispersion of an image of an object is acceptable or not based on the result of comparison between a feature quantity of the object in a binarized image and a feature quantity of the object in a grayscale image, and changes a process of recognizing pedestrians, for thereby increasing the reliability of pedestrian recognition (see paragraphs [0244] through [0246] of U.S. Patent Application
Publication No. 2005/0063565 Al).
The vehicle vicinity monitoring apparatus disclosed in U.S. Patent Application Publication No. 2005/0276447 Al extracts objects to be binarized from a grayscale image and compares a feature value of the extracted objects with a feature value of the legs of a pedestrian stored in a pedestrian leg feature value storage means, for thereby determining whether the objects to be
binarized are pedestrian legs or not. If the vehicle vicinity monitoring apparatus judges that the objects to be binarized are pedestrian legs, then the vehicle vicinity monitoring apparatus recognizes an object including the objects to be binarized as a pedestrian and hence recognizes a pedestrian in the vicinity of the vehicle (see paragraphs [0012] and [0117] of U.S. Patent Application Publication No. 2005/0276447 Al).
The vehicle vicinity monitoring apparatus of the related art described above are an excellent system for displaying an image of a less-visible pedestrian ahead of the vehicle which has been detected as an object while the vehicle is traveling at night , and notifying the driver of the presence of a pedestrian with sound and a displayed highlighted frame, for example.
However, the vehicle vicinity monitoring apparatus of the related art still have much to be improved about the detection of a bicycle rider who is pedaling a bicycle at night .
Summary of Invention
It is an object of the present invention to provide a vehicle vicinity monitoring apparatus which is capable of detecting a bicycle rider.
According to the present invention, there is
provided a vehicle vicinity monitoring apparatus for detecting a bicycle rider who is pedaling a bicycle, as an object, from images captured by infrared cameras mounted on a vehicle, comprising an upper body and lower body area identifying device for identifying an area including upper and lower bodies estimated as the object from the images, a shape change detecting device for detecting time-dependent changes of upper and lower body shapes in the identified area including the upper and lower bodies, a difference acquiring device for acquiring a difference between the upper and lower body shapes at each of the detected time-dependent changes, and a bicycle rider determining device for judging the object as the bicycle rider if the amplitude of the difference is of a value greater than a threshold value.
If the amplitude of a time-dependent change of the difference between the upper body shape and the lower body shape in an area including upper and lower bodies estimated as a bicycle rider is of a value greater than a threshold value, then the area including the upper and lower bodies is identified as a bicycle rider. In this manner, the vehicle vicinity monitoring apparatus detects the bicycle rider as the object.
When the upper body and lower body area identifying device is to identify an area including upper and lower bodies from images acquired by the infrared cameras , grayscale images acquired by the infrared cameras or binarized images produced by binarizing the grayscale images may be used as the above images acquired by the infrared cameras .
The bicycle rider can appropriately be estimated if the object estimated by the upper body and lower body area identifying device has a feature that the upper body has a smaller time-dependent change and the lower body has a greater time-dependent change.
The shape change detecting device may have a
reference template including a reference bicycle rider shape made up of an upper body shape and a lower body shape, and detect the time-dependent changes of the upper body shape and the lower body shape in the identified area including the upper and lower bodies by subtracting the upper body shape and the lower body shape in the identified area including the upper and lower bodies from the upper body shape and the lower body shape in the reference bicycle rider shape. The reference template provided in the vehicle vicinity monitoring apparatus may be a reference template of grayscale image if the area including the upper body and the lower body is
represented by a grayscale image. Alternatively, the reference template may be a reference template of
binarized image if the area including the upper body and the lower body is represented by a binarized image.
The reference template may comprise a first
reference template including a first reference bicycle rider shape in which a right foot is positioned upwardly of a left foot in the lower body shape as viewed in front elevation, and a second reference template including a second reference bicycle rider shape in which a right foot is positioned downwardly of a left foot in the lower body shape as viewed in front elevation, the second reference template being the left-right reversal of the first reference template, wherein the shape change detecting device may detect the time-dependent changes of the upper and lower body shapes , using the first
reference template and the second reference template.
Preferably, the vehicle vicinity monitoring
apparatus may further comprise a direction-of-travel detecting device for detecting a change in the direction of travel of the bicycle rider which is the object, if each of the time-dependent changes of the upper and lower body shapes in the identified area including the upper and lower bodies, detected by the shape change detecting device , abruptly changes .
Preferably, the upper body and lower body area identifying device may include an upper body area identifying device for identifying an upper body area of the bicycle rider, i.e., the object, if three high- brightness regions estimated as a head and a right hand and a left hand which grip the handle of the bicycle are detected.
Heretofore, it has been difficult to detect a bicycle rider. However, according to the present invention, the vehicle vicinity monitoring apparatus can detect such a bicycle rider as an object highly
accurately from images output f om the infrared cameras .
The above and other objects, features, and
advantages of the present invention will become more apparent from the following description when taken in conjunction with the accompanying drawings in which preferred embodiments of the present invention are shown by way of illustrative example.
Brief Description of Drawings
FIG. 1 is a block diagram of a vehicle vicinity monitoring apparatus according to an embodiment of the present invention;
FIG. 2 is a schematic perspective view of a vehicle incorporating the vehicle vicinity monitoring apparatus shown in FIG. 1;
FIG. 3 is a diagram showing reference templates;
FIG. 4 is a diagram showing another reference template having a different direction; FIG. 5 is a flowchart of an operation sequence of an image processing unit for detecting and determining an object such as a bicycle rider;
FIG. 6 is a diagram showing a succession of
grayscale images that are obtained as respective frames ordered in time from above;
FIG. 7A is a diagram showing a succession of image areas including upper and lower bodies identified from respective grayscale images;
FIG. 7B is a diagram showing a succession of
binarized images corresponding respectively to the image areas shown in FIG. 7A;
FIG. 8A is a diagram which is illustrative of the principles of an upper and lower body area identifying device for identifying areas including upper and lower bodies estimated as objects from grayscale images;
FIG. 8B is a diagram which is illustrative of the principles of an upper and lower body area identifying device for identifying areas including upper and lower bodies estimated as objects from binary images;
FIG. 9 is a diagram which is illustrative of the calculation of correlated errors corresponding to shape changes at times ;
FIG. 10 is a diagram showing upper body correlated errors, lower body correlated errors, and differential correlated errors thereof;
FIG. 11A is a diagram showing how the head of a bicycle rider moves with respect to the road;
FIG. 11B is a diagram showing how the head of a person moves with respect to the road while the person is jogging; and
FIG. 12 is a diagram showing correlated errors at the time a bicycle rider changes its direction of travel. Description of Embodiments
A vehicle vicinity monitoring apparatus according to an embodiment of the present invention will be described in detail below with reference to FIGS. 1 through 12. (Total Structure)
FIG. 1 is a block diagram of a vehicle vicinity monitoring apparatus 10 according to an embodiment of the present invention, and FIG. 2 is a schematic perspective view of a vehicle 12 incorporating the vehicle vicinity monitoring apparatus 10 shown in FIG. 1.
As shown in FIGS. 1 and 2, the vehicle vicinity monitoring apparatus 10 comprises an image processing unit 14 for controlling the vehicle vicinity monitoring apparatus 10, a pair of right and left infrared cameras 16R, 16L connected to the image processing unit 14, a vehicle speed sensor 18 for detecting a vehicle speed Vs of the vehicle 12, a brake sensor 20 for detecting an operated amount (brake operated amount) Br of the brake pedal depressed by the driver of the vehicle 12, a yaw rate sensor 22 for detecting a yaw rate Yr of the vehicle 12, a speaker 24 for issuing an audible warning, and an image display device 26 including a HUD (Head Up Display) 26a for displaying an image captured by the infrared cameras 16R, 16L to enable the driver to visually
recognize an object such as a pedestrian or the like (a moving object) which is highly likely to stay in a collision course with the vehicle 12.
The image display device 26 may have a navigation system display unit rather than the HUD 26a.
The image processing unit 14 detects a moving object such as a pedestrian, a bicycle rider, or the like ahead of the vehicle 12 from infrared images in the vicinity of the vehicle 12 and signals (vehicle speed Vs, brake operated amount Br, and yaw rate Yr in the illustrated embodiment) representative of the traveling state of the vehicle 12, and issues a warning when the detected moving object is highly likely to stay in a collision course with the vehicle 12.
The image processing unit 14 comprises an A/D converter for converting input analog signals into digital signals, an image memory (storage unit 14m) for storing digital image signals, a CPU (Central Processing Unit) 14c for performing various processing operations, a storage unit 14m including a RAM (Random Access Memory) for storing data as they are being processed by the CPU 14c and a ROM (Read Only Memory) for storing programs executed by the CPU 14c, tables, ma s, etc., and an output circuit for outputting drive signals for the speaker 24, display signals for the image display device 26, and other signals. The infrared cameras 16R, 16L, the yaw rate sensor 22, the vehicle speed sensor 18, and the brake sensor 20 produce output signals which are converted by the A/D converter into digital signals to be supplied to the CPU 14c.
The CPU 14c of the image processing unit 14
functions as various functional means (also called functional sections) by reading the supplied digital signals and executing programs, and sends drive signals (audio signals and display signals) to the speaker 24 and the image display device 26. Since these functions can also be implemented by hardware, the functional means will hereinafter be referred to as functional devices.
As shown in FIG. 2, the infrared cameras 16R, 16L, which function as so-called stereographic cameras, are disposed on a front bumper of the vehicle 12 in
respective positions that are substantially symmetrical with respect to the transverse center of the vehicle 12. The infrared cameras 16R, 16L have respective optical axes extending parallel to each other and are positioned at the same height from the road on which the vehicle 12 travels. The infrared cameras 16R, 16L have such characteristics that their output signals are higher in level as the temperature of objects captured thereby is higher .
The HUD 26a has its display screen displayed on the front windshield of the vehicle 12 at a position out of the front field of vision of the driver of the vehicle 12.
As shown in FIG. 3, the ROM (storage unit 14m) of the image processing unit 14 stores therein a reference template 104 which comprises a first reference template 104a including a reference bicycle rider shape (first reference bicycle rider shape) 103a made up of an upper body shape 100 and a lower body shape 102a, and a second reference template 104b including a reference bicycle rider shape (second reference bicycle rider shape) 103b made up of an upper body shape 100 and a lower body shape 102b. Each of the first reference template 104a and the second reference template 104b is a binary image
including white areas which represent high-brightness portions such as human body parts and black areas , shown hatched, which represent low-brightness portions such as a background.
As described later, only either one of the first reference template 104a and the second reference template 104b may be stored in the storage unit 14m.
In the first reference template 104a and the second reference template 104b, the upper body shapes 100, which represent a head and a torso including hands , as viewed in front elevation in the direction from the bicycle rider toward the vehicle 12, are identical with each other, including a right hand Rh and a left hand Lh for gripping the bicycle handle. The first reference
template 104a includes, in the lower body shape 102a as viewed in front elevation, the first reference bicycle rider shape 103a in which a right foot (right leg) Rf is positioned above a left foot (left leg) Lf . The second reference template 104b is the left-right reversal of the first reference template 104a (i.e., one obtained by left-right inverting the first reference template 104a) , and includes, in the lower body shape 102b as viewed in front elevation, the second reference bicycle rider shape 103b in which a right foot Rf is positioned below a left foot Lf .
The storage unit 14m also stores therein reference templates, not shown, which correspond to the first and second reference templates , as viewed in front elevation in the direction away from the vehicle 12, a reference template 112 (see FIG. 4) including a reference bicycle rider shape 110 made up of an upper body shape 106 and a lower body shape 108 as viewed in front elevation in the direction across in front of the vehicle 12, and a reference template which is the left-right reversal of the reference template 112 (i.e., obtained by left-right inverting the reference template 112), as viewed in front elevation in the traveling direction opposite to the direction of the reference bicycle rider shape 110 shown in FIG. 4.
Operation of the vehicle vicinity monitoring
apparatus 10 according to the present embodiment will be described below.
(Operation for Detecting and Determining an Object)
FIG. 5 is a flowchart of an operation sequence of the image processing unit 14 for detecting and
determining an object such as a bicycle rider.
In step SI shown in FIG. 5, the image processing unit 14 acquires infrared images in a certain range of field angle in front of the vehicle, which are captured in frames by the infrared cameras 16R, 16L and
represented by output signals in the frames, converts the acquired infrared images into digital image signals, and stores the digital image signals as right and left grayscale images in the image memory. The right
grayscale image is produced by the infrared camera 16R, and the left grayscale image is produced by the infrared camera 16L. The right grayscale image and the left grayscale image include an object at different horizontal positions therein. The distance from the vehicle 12 up to the object can be calculated based on the difference (parallax) between the different horizontal positions.
After the grayscale images are obtained in step SI, the right grayscale image, which is produced by the infrared camera 16R, is used as a reference image to binarize its image signal, i.e., any area of the
grayscale image which is higher than a threshold
brightness level is converted into "1" (white) and any area of the grayscale image which is lower than the threshold brightness level is converted into "0" (black) . Therefore, each of the captured frames is converted into a binarized image.
FIG. 6 shows a succession of grayscale images 30a through 30f that are obtained as respective frames ordered in time from above.
In step S2, candidates for an object, i.e., a bicycle rider who is pedaling a bicycle , are extracted for detecting the object. The object candidates are represented by high-brightness areas (white areas in binarized images) that are extracted from the grayscale images 30a through 30f or binarized images (not shown) produced therefrom.
In step S3 (upper body and lower body area
identifying device), an area including an upper body and a lower body that is estimated as an object, is
identified among the object candidates, which have been extracted from the grayscale images 30a through 30f or binarized images produced therefrom. FIG. 7A shows a succession of image areas PI through P6 including upper and lower bodies identified from the respective grayscale images 30a through 30f . FIG. 7B shows a succession of binarized images 31abn through 31fbn in image areas Plb through P6b corresponding respectively to grayscale images 31a through 3If (FIG. 7A) in the image areas PI through P6. The binarized images 31abn through 31fbn may be interpreted as
intensified images of the grayscale images 31a through 31f.
An upper body and lower body area identifying device for identifying an area including upper and lower bodies estimated as the object from the grayscale images 30a through 30f (FIG. 6) includes an upper body area
identifying device. The upper body area identifying device identifies an upper body area Pu of the bicycle rider, i.e., the object, if three high-brightness regions estimated as a head 50 and a right hand 51R and a left hand 51L which grip the handle are detected, as indicated in grayscale images 70, 72 of areas Pt , Pt + At at respective times t, t + At in FIG. 8A.
As shown in FIG. 8B, the upper body area identifying device is also able to identify three high-brightness regions corresponding to the head 50 and the right hand 51R and the left hand 51L which grip the handle, from binarized images 70bn, 72bn converted from the grayscale images 70, 72.
The upper body and lower body area identifying device also includes a lower body area identifying device for identifying a lower body area Pd of the bicycle rider below the upper body area Pu. The lower body area Pd includes two high-brightness regions estimated as both feet (both legs), i.e., a right foot (right leg) 52R and a left foot (left leg) 52L. The two high-brightness regions make shape changes (vertical pedaling movement) and periodic movement (vertical pedal movement) when the object is moving. On the other hand, the upper body area Pu includes the three high-brightness regions, which make no or little shape changes when the object is moving.
In this case as well, from the binarized images
70bn, 72bn converted from the grayscale images 70, 72, the lower body area identifying device is also able to identify an area including two high-brightness regions corresponding to the both feet (both legs), i.e., the right foot (right leg) 52R and the left foot (left leg) 52L, which make shape changes (vertical pedaling
movement) and periodic movement (vertical pedal movement) when the object is moving, as the lower body area Pd (binarized images 70bn, 72bn) corresponding to the lower body area Pd (grayscale images 70, 72), below the upper body area Pu of the binarized images of the bicycle rider corresponding to the upper body area Pu which includes the three high-brightness regions that make no or little shape changes when the object is moving.
In step S4 (shape change detecting device), shape changes in the upper body area Pu and the lower body area are detected.
In the present embodiment , the shape changes are detected as correlated errors between the first reference template 104a and the binarized images 31abn through 31fbn in the areas Plb through P6b, i.e., interframe correlated errors. As described below, an upper body correlated error Eu and a lower body correlated error Ed are calculated as the correlated errors . An entire body correlated error Eall can be calculated from the upper body correlated error Eu and the lower body correlated error Ed, as Eall = Eu + Ed.
More specifically, the upper body correlated error Eu is calculated as correlated errors between the upper body shape 100 of the first reference template 104a and the upper body areas Pu of the binarized images 31abn through 31fbn in the image areas Plb through P6b, e.g., the sum of squares of the differences between the
corresponding pixel values . The lower body correlated error Ed is calculated as correlated errors between the lower body shape 102a of the first reference template 104a and the lower body areas Pd of the binarized images 31abn through 31fbn in the image areas Plb through P6b.
FIG. 9 is illustrative of a calculation of
correlated errors corresponding to shape changes at times. At times tl and t3, the shapes of the lower body areas Pd of the binarized images in the areas Ptl, Pt3 are largely different from (i.e., in opposite phase to) the lower body shape (lower body reference shape) 102a of the first reference template 104a. At times tl and t3, therefore, the lower body correlated error Ed is maximum. At times t2 and t4, the shapes of the lower body areas Pd of the binarized images in the areas Pt2, Pt4 are
substantially the same as (i.e., in phase with) the lower body shape (the lower body reference shape) 102a of the first reference template 104a. At times t2 and t4, therefore, the lower body correlated error Ed is minimum.
In FIG. 9, a lower body correlated error Edi
represents correlated errors between the binarized images in the areas Ptl through Pt4 and the second reference template 104b which is obtained by left-right inverting the first reference template 104a. The detecting
accuracy, i.e., the detecting reliability, is thus increased based on the fact that the correlated errors with respect to the first and second reference templates 104a, 104b are in opposite phase to each other.
Though not shown in FIG. 9, the upper body
correlated error Eu remains substantially nil at all times as the upper body of the bicycle rider does not move and is kept in the same shape.
In step S5 (difference acquiring device), the difference between the correlated errors of the upper body shape and the lower body shape is calculated
(acquired) at each detected time-dependent change. More specifically, in each frame, i.e., at each detection time, the difference ΔΕ is calculated as ΔΕ = the lower body correlated error (Ed) - the upper body correlated error (Eu) « the lower body correlated error (Ed) .
FIG. 10 shows a simulation result for easier
understanding of the present invention. As shown in FIG. 10, when a bicycle rider who is pedaling a bicycle, as an object, is relatively approaching the vehicle 12 while facing toward or away from the vehicle 12, both the upper body correlated error Eu and the lower body correlated error Ed tend to increase in frames ranging from frame number 0 to frame number 180.
The increasing tendency is caused for the following reasons: First, as the first reference template is enlarged, aliasing (blurring) becomes large, thereby increasing the differential error. Secondly, there is an increasing error related to a change in the brightness of the object depending on the distance between the object and the vehicle. Thirdly, there is an increasing error due to a background change (noise) .
The lower body correlated error Ed is larger in level than the upper body correlated error Eu because the lower body moves greater than the upper body in each of the frames. Additionally, the lower body correlated error Ed has another feature that the amplitude thereof periodically increases and decreases in synchronism with the bicycle rider's pedaling action.
In step S5, the difference (correlated error
difference) ΔΕ is calculated by subtracting the upper body correlated error Eu from the lower body correlated error Ed.
Since an error due to enlargement of the reference template 104a, an error related to the brightness change of the object depending on the distance, and an error due to the background change, which are commonly included in the upper and lower body correlated errors Eu, Ed, are removed, the difference ΔΕ is appropriately detected as having periodic amplitude increases and decreases .
In step S6 (bicycle rider determining device), it is determined whether the amplitude (AEmax - AEmin) of the difference ΔΕ as a detected time-dependent change is of a value greater than a threshold value TH (TH = 0.05 in the present embodiment) {the difference (ΔΕ) > the threshold value (TH)} or not. If the amplitude (AEmax - AEmin) of the difference ΔΕ is of a value greater than the
threshold value TH, then the object is basically judged as a bicycle rider.
In step S7 , it is determined whether the amplitude period of the difference ΔΕ falls within a period
(generally, 0.88 [Hz]) corresponding to the speed of the bicycle or not.
If the amplitude period of the difference ΔΕ falls within the period corresponding to the speed of the bicycle, then it is determined whether the upper body shape moves vertically or not in step S8.
As shown in FIG. 11A, the height (road height) H of the head 60 of the bicycle rider from the road 62 is a constant height Hconst (H = Hconst). On the other hand, as shown in FIG. 11B, the head 64 of a person who is jogging moves vertically, and the road height H of the head 64 is represented by a periodically varying height Hvar .
If it is judged that the upper body shape does not vertically move in step S8 , then the object is finally judged as a bicycle rider in step S9.
When the object is finally judged as a bicycle rider, if the object is likely to be in a collision course with the vehicle 12 based on the operated amount Br output from the brake sensor 20, the vehicle speed Vs output from the vehicle speed sensor 18, the yaw rate Yr output from the yaw rate sensor 22, and the distance from the vehicle 12 up to the object (i.e., the bicycle rider) , then the grayscale image of the bicycle rider is displayed on the HUD 26a, and an audible warning is issued through the speaker 24, prompting the driver of the vehicle 12 to take an action to avoid the possible collision with the bicycle rider. If the driver of the vehicle 12 appropriately brakes the vehicle 12 and there is no possibility of collision, then no audible warning is issued through the speaker 24, so that the driver will not be unnecessarily troubled.
If the answer to either one of steps S6, S7, S8 is negative, then the object is finally judged as something other than a bicycle rider. The process of determining a pedestrian according to the related art may be carried out after step S10.
According to the present embodiment, as described above, the vehicle vicinity monitoring apparatus 10 converts grayscale images acquired by the infrared cameras 16R, 16L mounted on the vehicle 12 into binarized images. For detecting an object, i.e., a bicycle rider who is pedaling a bicycle, from the binarized images, the upper body and lower body area identifying device (step S3) identifies an area including upper and lower bodies estimated as the object from the binarized images. Then, the shape change detecting device (step S4) detects time- dependent changes of the upper and lower body shapes in the identified area including the upper and lower bodies. The difference acquiring device (step S5) acquires the difference between the upper and lower body shapes at each of the detected time-dependent changes. If the amplitude of the difference is of a value greater than the threshold value, then the bicycle rider determining device (step S6) judges the object as the bicycle rider. The vehicle vicinity monitoring apparatus 10 is thus able to detect a bicycle rider near the vehicle 12 at night.
The upper body and lower body area identifying device (step S3) can appropriately estimate the object as a bicycle rider if the object has a feature that the upper body has a smaller time-dependent change and the lower body has a greater time-dependent change.
The shape change detecting device (step S4) has the reference template 104 including the reference bicycle rider shape made up of a certain upper body shape and a certain lower body shape . Thus , the shape change
detecting device can detect time-dependent changes of the upper body shape and the lower body shape in the
identified area including the upper and lower bodies by subtracting the upper body shape and the lower body shape in the identified area including the upper and lower bodies from the upper body shape and the lower body shape in the reference bicycle rider shape.
The reference template 104 comprises the first reference template 104a including the first reference bicycle rider shape wherein the right foot is positioned upwardly of the left foot in the lower body shape as viewed in front elevation, and the second reference template 104b including the second reference bicycle rider shape wherein the right foot is positioned
downwardly of the left foot in the lower body shape as viewed in front elevation, the second reference template 104b being the left-right reversal of the first reference template 104a. The shape change detecting device (step S4) can detect time-dependent changes of the upper body shape and the lower body shape, using the first and second reference templates 104a, 104b.
The upper body and lower body area identifying device (step S3) identifies an area including upper and lower bodies estimated as the object from the binarized images . The upper body and lower body should preferably include the upper body area identifying device for identifying the upper body area of the bicycle rider as the object if three high-brightness regions estimated as the head and the right hand and the left hand which grip the handle are detected.
Heretofore, it has been difficult to detect a bicycle rider. However, according to the present embodiment, the vehicle vicinity monitoring apparatus 10 can detect such a bicycle rider as an object highly accurately from images output from infrared cameras .
In the above embodiment, the vehicle vicinity monitoring apparatus 10 converts grayscale images acquired by the infrared cameras 16R, 16L mounted on the vehicle 12 into binarized images. For detecting an object, i.e., a bicycle rider who is pedaling a bicycle, from the binarized images, the upper body and lower body area identifying device (step S3) identifies an area including upper and lower bodies estimated as the object from the binarized images. However, the upper body and lower body area identifying device may identify an area including upper and lower bodies estimated as the object from the grayscale images acquired by the infrared cameras 16R, 16L.
In such a case, the shape change detecting device (step S4) detects time-dependent changes of the grayscale images of the upper and lower body shapes in the
identified area including the upper and lower bodies.
The difference acquiring device (step S5) acquires the difference between the upper and lower body shapes in the grayscale images at each of the detected time-dependent changes. If the amplitude of the difference is of a value greater than the threshold value, then the bicycle rider determining device (step S6) judges the object as the bicycle rider.
In the above embodiment, time-dependent shape changes of the binarized images converted from the grayscale images are detected based on the reference templates (the first and second reference templates 104a, 104b) of binarized image schematically shown in FIG. 3. However, a reference template of grayscale image may be used instead of the reference template of binarized image. For example, time-dependent shape changes in the grayscale images 31b through 31f of the areas P2 through P6 may be detected using, as a reference template, the grayscale image 31a of the area PI among the areas PI through P6 (see FIG. 7A) including the upper and lower bodies identified from the grayscale images 30a through 30f (see FIG. 6) acquired by the infrared cameras 16R, 16L.
Alternatively, the binarized image 31abn in the area Plb (see FIG. 7B) corresponding to the grayscale images 31a through 31f in the area PI may be used as a reference template.
(Other Embodiments)
The vehicle vicinity monitoring apparatus 10 should preferably further include a direction-of-travel
detecting device for detecting a change in the direction of travel of the object, i.e., the bicycle rider, if the correlated error Ed derived from time-dependent changes of the upper and lower body shapes in the identified area including the upper and lower bodies, detected by the shape change detecting device, abruptly changes, as indicated by a correlated error Edx between time tl2 and time tl3 in FIG. 12.
In such a case, a highly reliable correlated error Ed in areas Ptll, Ptl2 is detected based on the first reference template 104a until a time shortly after time tl2. Based on an increase in the correlated error subsequent to time tl2, the direction of travel of the bicycle rider is estimated as having changed from a front-back direction to a left-right direction after time tl2, and the reference template is updated from the first reference template 104a to the reference template 112 shown in FIG. 4, after which the processing in steps SI through S9 is performed. In this manner, the period of movement is extracted again as indicated by the
correlated error Ed subsequent to time tl3.
Although certain preferred embodiments of the present invention have been shown and described in detail, it should be understood that various changes and
modifications may be made thereto without departing from the scope of the invention as set forth in the appended claims .

Claims

Claim 1. A vehicle vicinity monitoring apparatus for detecting a bicycle rider who is pedaling a bicycle, as an object, from images captured by infrared cameras mounted on a vehicle, comprising:
an upper body and lower body area identifying device for identifying an area including upper and lower bodies estimated as the object from the images;
a shape change detecting device for detecting time- dependent changes of upper and lower body shapes in the identified area including the upper and lower bodies; a difference acquiring device for acquiring a difference between the upper and lower body shapes at each of the detected time-dependent changes; and
a bicycle rider determining device for judging the object as the bicycle rider if the amplitude of the difference is of a value greater than a threshold value.
Claim 2. A vehicle vicinity monitoring apparatus according to claim 1, wherein the object estimated by the upper body and lower body area identifying device has a feature that the upper body has a smaller time-dependent change and the lower body has a greater time-dependent change.
Claim 3. A vehicle vicinity monitoring apparatus according to claim 1 or 2, wherein the shape change detecting device has a reference template including a reference bicycle rider shape made up of an upper body shape and a lower body shape, and detects the time- dependent changes of the upper body shape and the lower body shape in the identified area including the upper and lower bodies by subtracting the upper body shape and the lower body shape in the identified area including the upper and lower bodies from the upper body shape and the lower body shape in the reference bicycle rider shape.
Claim 4. A vehicle vicinity monitoring apparatus according to claim 3, wherein the reference template comprises :
a first reference template including a first
reference bicycle rider shape in which a right foot is positioned upwardly of a left foot in the lower body shape as viewed in front elevation; and
a second reference template including a second reference bicycle rider shape in which a right foot is positioned downwardly of a left foot in the lower body shape as viewed in front elevation, the second reference template being the left-right reversal of the first reference template;
wherein the shape change detecting device detects the time-dependent changes of the upper and lower body shapes, using the first reference template and the second reference template.
Claim 5. A vehicle vicinity monitoring apparatus according to claim 1 or 2 , further comprising:
a direction-of-travel detecting device for detecting a change in a direction of travel of the bicycle rider which is the object, if each of the time-dependent changes of the upper and lower body shapes in the
identified area including the upper and lower bodies, detected by the shape change detecting device, abruptly changes .
Claim 6. A vehicle vicinity monitoring apparatus according to claim 1 or 2 , wherein the upper body and lower body area identifying device includes an upper body area identifying device for identifying an upper body area of the bicycle rider which is the object if three high-brightness regions estimated as a head and a right hand and a left hand which grip the handle of the bicycle are detected.
PCT/JP2010/065452 2009-09-03 2010-09-02 Vehicle vicinity monitoring apparatus WO2011027907A1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP10813852.0A EP2465092B1 (en) 2009-09-03 2010-09-02 Vehicle vicinity monitoring apparatus
JP2012505916A JP5270794B2 (en) 2009-09-03 2010-09-02 Vehicle periphery monitoring device
CN201080036182.8A CN102473281B (en) 2009-09-03 2010-09-02 Vehicle vicinity monitoring apparatus
US13/391,395 US20120147188A1 (en) 2009-09-03 2010-09-02 Vehicle vicinity monitoring apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2009-203411 2009-09-03
JP2009203411 2009-09-03

Publications (1)

Publication Number Publication Date
WO2011027907A1 true WO2011027907A1 (en) 2011-03-10

Family

ID=43649440

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2010/065452 WO2011027907A1 (en) 2009-09-03 2010-09-02 Vehicle vicinity monitoring apparatus

Country Status (5)

Country Link
US (1) US20120147188A1 (en)
EP (1) EP2465092B1 (en)
JP (1) JP5270794B2 (en)
CN (1) CN102473281B (en)
WO (1) WO2011027907A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012127294A1 (en) * 2011-03-18 2012-09-27 Toyota Jidosha Kabushiki Kaisha Bicycle detection system and irradiation control system
JP2013016071A (en) * 2011-07-05 2013-01-24 Seiko Epson Corp Detection device and detection method
JP2013073311A (en) * 2011-09-27 2013-04-22 Honda Motor Co Ltd Object type determination device
JP2013232080A (en) * 2012-04-27 2013-11-14 Denso Corp Object identification device

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5486031B2 (en) * 2012-03-06 2014-05-07 本田技研工業株式会社 Light distribution control device
US9784577B2 (en) * 2012-03-16 2017-10-10 Lg Innotek Co., Ltd. Measuring distance from object by using size of pattern projected onto object
IN2015KN00339A (en) * 2012-07-27 2015-07-10 Nissan Motor
JP6178580B2 (en) * 2013-01-28 2017-08-09 富士通テン株式会社 Object detection apparatus, object detection system, and object detection method
US20160292516A1 (en) * 2013-11-20 2016-10-06 Nec Corporation Two-wheel vehicle riding person number determination method, two-wheel vehicle riding person number determination system, two-wheel vehicle riding person number determination apparatus, and program
CN103745196B (en) * 2013-12-27 2016-10-05 东软集团股份有限公司 Broad sense pedestrian detection method and device
US10065562B2 (en) 2013-12-31 2018-09-04 International Business Mahcines Corporation Vehicle collision avoidance
CN104834887B (en) * 2014-02-11 2018-09-11 株式会社理光 Move pedestrian's representation method, recognition methods and its device
JP6255309B2 (en) * 2014-06-12 2017-12-27 東芝テック株式会社 Information processing apparatus and notification system
JP6340957B2 (en) * 2014-07-02 2018-06-13 株式会社デンソー Object detection apparatus and object detection program
JP6626623B2 (en) * 2015-03-27 2019-12-25 中日本高速道路株式会社 Moving object detection device
WO2017019264A1 (en) * 2015-07-30 2017-02-02 Elevationtv Llc Safe backup system for vehicles
JP6620715B2 (en) * 2016-10-20 2019-12-18 株式会社デンソー Image recognition device
CN108062868B (en) * 2016-11-09 2021-06-29 奥迪股份公司 Bicycle detection system and method for vehicle and vehicle
WO2020110915A1 (en) * 2018-11-30 2020-06-04 ソニー株式会社 Information processing device, information processing system, and information processing method
US10991130B2 (en) * 2019-07-29 2021-04-27 Verizon Patent And Licensing Inc. Systems and methods for implementing a sensor based real time tracking system
FR3123490A1 (en) 2021-06-01 2022-12-02 Psa Automobiles Sa Method for managing the display of the lateral safety distance to be observed by a motor vehicle in the event of overtaking a vulnerable road user

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000097963A (en) * 1998-09-21 2000-04-07 Mitsubishi Electric Corp Discrimination apparatus for moving body
JP2008090748A (en) * 2006-10-04 2008-04-17 Toyota Motor Corp Warning apparatus for vehicle
JP2009199138A (en) * 2008-02-19 2009-09-03 Honda Motor Co Ltd Device for monitoring vehicle periphery, vehicle, and method for monitoring vehicle periphery
WO2010047054A1 (en) * 2008-10-20 2010-04-29 本田技研工業株式会社 Vehicle periphery monitoring apparatus

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4007012B2 (en) * 2002-02-04 2007-11-14 日産自動車株式会社 Vehicle protection device
JP3987048B2 (en) * 2003-03-20 2007-10-03 本田技研工業株式会社 Vehicle periphery monitoring device
US7330566B2 (en) * 2003-05-15 2008-02-12 Microsoft Corporation Video-based gait recognition
JP3922245B2 (en) * 2003-11-20 2007-05-30 日産自動車株式会社 Vehicle periphery monitoring apparatus and method
JP2007187618A (en) * 2006-01-16 2007-07-26 Omron Corp Object identifying device
JP2007255977A (en) * 2006-03-22 2007-10-04 Nissan Motor Co Ltd Object detection method and object detector
JP4772622B2 (en) * 2006-08-18 2011-09-14 アルパイン株式会社 Perimeter monitoring system
WO2008065717A1 (en) * 2006-11-29 2008-06-05 Fujitsu Limited Pedestrian detecting system, and pedestrian detecting method
US20090016571A1 (en) * 2007-03-30 2009-01-15 Louis Tijerina Blur display for automotive night vision systems with enhanced form perception from low-resolution camera images
JP4470067B2 (en) * 2007-08-07 2010-06-02 本田技研工業株式会社 Object type determination device, vehicle
JP4861952B2 (en) * 2007-09-28 2012-01-25 富士フイルム株式会社 Image processing apparatus and imaging apparatus

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000097963A (en) * 1998-09-21 2000-04-07 Mitsubishi Electric Corp Discrimination apparatus for moving body
JP2008090748A (en) * 2006-10-04 2008-04-17 Toyota Motor Corp Warning apparatus for vehicle
JP2009199138A (en) * 2008-02-19 2009-09-03 Honda Motor Co Ltd Device for monitoring vehicle periphery, vehicle, and method for monitoring vehicle periphery
WO2010047054A1 (en) * 2008-10-20 2010-04-29 本田技研工業株式会社 Vehicle periphery monitoring apparatus

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012127294A1 (en) * 2011-03-18 2012-09-27 Toyota Jidosha Kabushiki Kaisha Bicycle detection system and irradiation control system
JP2013016071A (en) * 2011-07-05 2013-01-24 Seiko Epson Corp Detection device and detection method
JP2013073311A (en) * 2011-09-27 2013-04-22 Honda Motor Co Ltd Object type determination device
JP2013232080A (en) * 2012-04-27 2013-11-14 Denso Corp Object identification device

Also Published As

Publication number Publication date
US20120147188A1 (en) 2012-06-14
EP2465092A4 (en) 2013-05-22
JP5270794B2 (en) 2013-08-21
CN102473281A (en) 2012-05-23
CN102473281B (en) 2015-06-10
JP2013504098A (en) 2013-02-04
EP2465092B1 (en) 2015-02-11
EP2465092A1 (en) 2012-06-20

Similar Documents

Publication Publication Date Title
EP2465092B1 (en) Vehicle vicinity monitoring apparatus
JP4410292B1 (en) Vehicle periphery monitoring device
US9235990B2 (en) Vehicle periphery monitoring device
JP4173901B2 (en) Vehicle periphery monitoring device
EP2759999B1 (en) Apparatus for monitoring surroundings of vehicle
US7233233B2 (en) Vehicle surroundings monitoring apparatus
JP5529910B2 (en) Vehicle periphery monitoring device
JP4470067B2 (en) Object type determination device, vehicle
JP5774770B2 (en) Vehicle periphery monitoring device
JP4528283B2 (en) Vehicle periphery monitoring device
JP4644273B2 (en) Vehicle periphery monitoring device
US9160986B2 (en) Device for monitoring surroundings of a vehicle
JP4887537B2 (en) Vehicle periphery monitoring device
JP4813304B2 (en) Vehicle periphery monitoring device
JP3844750B2 (en) Infrared image recognition device and alarm device using infrared image recognition device
US9030560B2 (en) Apparatus for monitoring surroundings of a vehicle
JP5430633B2 (en) Vehicle periphery monitoring device
JP4922368B2 (en) Vehicle periphery monitoring device

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 201080036182.8

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10813852

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2012505916

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 13391395

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 2010813852

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE