WO2011048454A1 - Vehicle night vision support system and control method for the same - Google Patents

Vehicle night vision support system and control method for the same Download PDF

Info

Publication number
WO2011048454A1
WO2011048454A1 PCT/IB2010/002532 IB2010002532W WO2011048454A1 WO 2011048454 A1 WO2011048454 A1 WO 2011048454A1 IB 2010002532 W IB2010002532 W IB 2010002532W WO 2011048454 A1 WO2011048454 A1 WO 2011048454A1
Authority
WO
WIPO (PCT)
Prior art keywords
display
vehicle
image
driver
control unit
Prior art date
Application number
PCT/IB2010/002532
Other languages
French (fr)
Other versions
WO2011048454A8 (en
Inventor
Kunihiko Toyofuku
Original Assignee
Toyota Jidosha Kabushiki Kaisha
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Toyota Jidosha Kabushiki Kaisha filed Critical Toyota Jidosha Kabushiki Kaisha
Priority to US13/391,157 priority Critical patent/US9061632B2/en
Priority to DE112010004133.4T priority patent/DE112010004133B4/en
Priority to CN201080047902.0A priority patent/CN102596643B/en
Publication of WO2011048454A1 publication Critical patent/WO2011048454A1/en
Publication of WO2011048454A8 publication Critical patent/WO2011048454A8/en

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/30—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles providing vision in the non-visible spectrum, e.g. night or infrared vision
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
    • B60R1/23—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
    • B60R1/24—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view in front of the vehicle
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G08G1/166—Anti-collision systems for active traffic, e.g. moving vehicles, pedestrians, bikes
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/10—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
    • B60R2300/106—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using night vision cameras
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/20—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of display used
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
    • B60R2300/302—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing combining image information with GPS information or vehicle data, e.g. vehicle speed, gyro, steering angle data
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
    • B60R2300/307—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing virtually distinguishing relevant parts of a scene from the background of the scene
    • B60R2300/308—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing virtually distinguishing relevant parts of a scene from the background of the scene by overlaying the real scene, e.g. through a head-up display on the windscreen
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
    • B60R2300/8093—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for obstacle warning
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0101—Head-up displays characterised by optical features
    • G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems

Definitions

  • the invention relates to a vehicle night vision support system that displays an image, captured in the nighttime by an infrared camera mounted on a vehicle, on an in-vehicle display, or the like, so that the captured image can be visually recognized in a vehicle cabin, and a control method for the same.
  • JP-A-2005-081860 describes a vehicle display system.
  • the vehicle display system includes a driving environment detecting means that detects a driving environment ahead of a vehicle, and changes an area captured by an image capturing means, such as an infrared camera, into a wide angle depending on the detected driving environment.
  • the vehicle display system detects, for example, the presence of a preceding vehicle as the driving environment ahead of the vehicle, and changes the captured area into a wide angle when there is the preceding vehicle.
  • JP-A-2005-088630 describes a vehicle night vision system.
  • the vehicle night vision system swivels an image capturing means, such as an infrared camera, horizontally in a vehicle transverse direction on the basis of a steering angle signal detected by a steering angle sensor. Note that the image capturing means is swiveled horizontally in the vehicle transverse direction in synchronization with head lights.
  • an image captured by the infrared camera is displayed on a display device (which is also used for, for example, a navigation display) installed at the center portion of an instrument panel, or the like, or is displayed on a head up display (HUD).
  • the HUD is a display device that, for example, reflects light, emitted from a light-emitting device installed at the upper portion of the instrument panel, from a front glass to allow a driver to visually recognize the emitted light.
  • an image captured by an infrared camera is, for example, displayed on a main display installed at the center portion of an instrument panel, or the like, and warning information, such as an icon that indicates the shape of an object, is displayed on a sub-display, such as an HUD, when the displayed image contains a warning target object, such as a pedestrian.
  • the driver may experience a feeling of strangeness because of a difference between the angle of view of the infrared camera and the field of view of the driver.
  • the angle of view of the infrared camera is normally narrower than the field of view of a driver in consideration of a resolution, a requirement for capturing an image at a location farther from the illumination limit of the head lamps, and the like. This is because the accuracy of remote image capturing decreases when the image capturing range is set to a wide angle.
  • the invention provides a vehicle night vision support system that is able to contribute to further safe driving without making a driver experiencing a feeling , of strangeness, and a control method for the same.
  • a first aspect of the invention provides a vehicle night vision support system.
  • the vehicle night vision support system includes: an infrared camera that captures an image ahead of a vehicle; a main display that displays the image captured by the infrared camera; a sub-display; and a control unit that determines whether there is an object in the displayed image on the main display and, when the control unit determines that there is the object in the displayed image, controls the sub-display so as to display warning information on the object, and the control unit estimates or detects whether an object that has disappeared from the displayed image is present in the field of view of the driver and, when the control unit estimates or detects that the object that has disappeared from the displayed image is present in the field of view of the driver, controls the sub-display so as to display the warning display.
  • a second aspect of the invention provides a control method for a vehicle night vision support system that includes an infrared camera that captures an image ahead of a vehicle, a main display that displays the image captured by the infrared camera and a sub-display.
  • the control method includes: determining whether there is an object in the displayed image on the main display; when it is determined that there is the object in the displayed image, controlling the sub-display so as to display warning information on the object; estimating or detecting whether an object that has disappeared from the displayed image is present in the field of view of the driver; and when it is estimated or detected that the object that has disappeared from the displayed image is present in the field of view of the driver, controlling the sub-display so as to display the warning information.
  • the sub-display is controlled so as to display the warning information.
  • the sub-display is controlled so as to display the warning information.
  • the sub-display may be a head up display.
  • the vehicle night vision support system may further include: a vehicle speed detector that detects a vehicle speed; and a steering angle detector that detects a steering angle, wherein the control unit may calculate a relative position of the object with respect to the vehicle on the basis of a position of the object in the displayed image, and the control unit may estimate whether the object that has disappeared from the displayed image is present in the field of view of the driver on the basis of the relative position and a driving state of the vehicle, calculated on the basis of the vehicle speed detected by the vehicle speed detector and the steering angle detected by the steering angle detector.
  • a vehicle speed detector that detects a vehicle speed
  • a steering angle detector that detects a steering angle
  • the main display may display a center portion of the image captured by the infrared camera with respect to a horizontal direction of the captured image.
  • FIG. 1 is an example of the system configuration of each of vehicle night vision support systems 1 and 2 according to first and second embodiments of the invention
  • FIG. 2 is an example of the configuration of an infrared camera 12
  • FIG. 3 is a view that shows a state where an infrared image ahead of a vehicle and an emphasized image are displayed on a main display 20 and warning information is displayed on a sub-display 30;
  • FIG. 4 is a view that shows a state where the emphasized image disappears from the main display 20 but the warning information is still displayed on the sub-display 30;
  • FIG. 5 is a view that shows the relationship between the image capturing range of the infrared camera 12 and the field of view of a driver;
  • FIG. 6 is a view that shows a state where it is determined that a pedestrian has reached the outside of the field of view of the driver and then the warning information is turned off;
  • FIG. 7 is a flowchart that shows the flow of characteristic processes executed by a night vision support ECU 40 according to the first embodiment.
  • FIG. 8 is a flowchart that shows the flow of characteristic processes executed by a night vision support ECU 40 according to the second embodiment.
  • FIG. 1 is an example of the system configuration of the vehicle night vision support system 1 according to the first embodiment of the invention.
  • the vehicle night vision support system 1 includes an infrared camera device 10, a main display 20, a sub-display 30 and a night vision support electronic control unit (ECU) 40 as major components.
  • ECU night vision support electronic control unit
  • the infrared camera device 10 includes an infrared irradiation device and an infrared camera 12.
  • the infrared irradiation device is, for example, attached to a front grill, and irradiates near infrared forward of the vehicle.
  • the infrared camera 12 is, for example, attached to the inside of the upper center portion of a front glass, and captures an image diagonally forward of the vehicle.
  • the infrared camera device 10 is not limited to a device that irradiates infrared to thereby capture reflected light but it may be a device that captures infrared radiated from a pedestrian because of body heat.
  • the infrared camera device 10 may be configured to mainly capture the light reflected from a pedestrian.
  • FIG. 2 is an example of the configuration of the infrared camera 12.
  • the infrared camera 12 includes, for example, a visible light cut filter 13 and a charge coupled device (CCD) camera 14.
  • the CCD camera 14 includes, for example, an image capturing lens 15, an image sensor 16 and an electronic control unit 17.
  • the infrared camera 12 may include a complementary metal-oxide-semiconductor (CMOS) camera, or the like, instead of the CCD camera.
  • CMOS complementary metal-oxide-semiconductor
  • the visible light cut filter 13 is, for example, stuck on the front glass.
  • the image sensor 16 is, for example, an interline image sensor.
  • the image sensor 16 has photodiodes, CCDs and transfer gates.
  • the photodiodes are photoreceptors that carry out photoelectric conversion and arranged in a two-dimensional planar shape.
  • the CCDs are arranged in a two-dimensional planar shape in correspondence with the photodiodes.
  • the transfer gates are provided between the photodiodes and the CCDs, and function as analog switches.
  • a light condensing microlens is attached in front (image capturing lens 15 side) of each photodiode. Note that the above structure is just schematically illustrated to simply describe the functions of the image sensor 16, and any design change for improvement in function, or the like, may be allowed.
  • the image sensor 16 is not limited to the interline image sensor; it may be a full-frame transfer or frame transfer image sensor of which the CCD itself functions as a photoreceptor.
  • the electronic control unit 17 is, for example, a microcomputer, an electronic circuit, or the like.
  • the electronic control unit 17 controls the open-close timing of each transfer gate of the image sensor 16 to regulate the shutter speed or image capturing period (for example, about several tens of times per one second) of the infrared camera 12. Then, the electronic control unit 17, for example, amplifies data, output from the output circuit of the image sensor 16, with a predetermined gain, and then outputs the amplified data to the night vision support ECU 40 as image data.
  • the irradiation range of the infrared irradiation device substantially coincides with the irradiation range of the infrared camera 12.
  • the image capturing range of the infrared camera 12 has an angle narrower than that of the irradiation range of the head lamps so as to be able to capture a more remote image.
  • the main display 20 is a touch panel display device.
  • the main display 20 is installed at, for example, the center portion of an instrument panel, or the like, and uses a video graphics array (VGA) as a graphic system to display an image including a moving image.
  • VGA video graphics array
  • the main display 20 detects a variation in voltage caused by user's touching the display surface to thereby recognize a touched position ⁇ With that, various GUI switches are set on the display surface to allow the user to conduct various input operations. In addition, a plurality of switches that allow various operations are provided around the display surface of the main display 20.
  • the main display 20 may be shared as a display/input device of another in-vehicle device, such as a navigation system.
  • the main display 20 functions as the main display 20 only when the vehicle night vision support system 1 is in operation (in the nighttime), and functions as the display/input device of another in-vehicle device during times other than the nighttime.
  • the sub-display 30 is, for example, an HUD.
  • the sub-display 30 projects an image at a display position on the front glass to display the image as a virtual image that is seen as if the image were present at a position on an extension of line of sight when the display position is viewed from the eyepoint of a driver.
  • the entire sub-display 30 is, for example, accommodated in a recess formed on the upper surface of the instrument panel.
  • the sub-display 30 includes a liquid crystal display, a reflector, and the like.
  • the liquid crystal display is arranged on the back side of a combination meter and emits display light toward the traveling direction of the vehicle.
  • the reflector reflects light emitted from the liquid crystal chorus, ... sentence
  • the night vision support ECU 40 is, for example, a microcomputer in which a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), and the like, are connected to one another via a bus.
  • the night vision support ECU 40 further includes a storage device, such as a hard disc drive (HDD), a digital versatile disk (DVD) drive, a compact disc-recordable (CD-R) drive and an electronically erasable and programmable read only memory (EEPROM), an I/O port, a timer, a counter, and the like.
  • the storage device stores programs and data executed by the CPU.
  • In-vehicle sensors such as a vehicle speed sensor 50 and a steering angle sensor 52, are connected to the night vision support ECU 40 via an in-vehicle communication line, such as a multiplex communication line.
  • the vehicle speed sensor 50 includes, for example, a skid control computer and a wheel speed sensor attached to each wheel of the vehicle.
  • the skid control computer converts a wheel speed pulse signal, output from each wheel speed sensor, into a vehicle speed rectangular wave pulse signal (a vehicle speed signal), and outputs the vehicle speed signal.
  • Each wheel speed sensor for example, includes a magnetic rotor and an active sensor. In the magnetic rotor, rubber is filled with magnetic powder and then positive electrodes and negative electrodes are alternately arranged in the circumferential direction.
  • the active sensor detects a variation in magnetic field due to rotation of the magnetic rotor.
  • the steering angle sensor 52 is, for example, arranged inside a steering column, and outputs a steering angle signal. Communication among the devices via the multiplex communication line is carried out using an appropriate communication protocol.
  • the communication protocol for example, includes a low-speed body communication protocol, a multimedia communication protocol and a FlexRay.
  • the low-speed body communication protocol is typically a controller area network (CAN) or a local interconnect network (LIN).
  • the multimedia communication protocol is typically a media oriented systems transport (MOST).
  • the night vision support ECU 40 includes a pedestrian recognition unit 42, a. pedestrian estimation unit 44 and a display control unit 46 as major functional blocks.
  • the major functional blocks function in such a manner that the CPU executes the programs stored in the storage device. Note that these functional blocks need not be based on distinctly different programs; one program may contain portions that implement a plurality of functional blocks.
  • the vehicle night vision support system 1 is activated by a command from a user through, for example, a GUI switch prepared on the display surface of the main display 20 or a switch attached around the main display 20.
  • the vehicle night vision support system 1 may be automatically activated when the outside illuminance detected by an illuminance sensor is lower than a predetermined value.
  • the pedestrian recognition unit 42 determines whether there is a pedestrian, for example, in the image captured by the infrared camera 12 (in the present embodiment, the captured image is the same as the image displayed on the main display 20).
  • the recognition of a pedestrian in the pedestrian recognition unit 42 is carried out by, for example, a step of extracting feature points from the image captured by the infrared camera 12 and a step of applying pattern matching to a series of feature points indicating the outline of an object to determine whether the object is a pedestrian.
  • the feature points may be simply defined as points of which the luminance difference between the adjacent pixels is larger than or equal to a predetermined value; instead, characteristic points may be extracted using a specific filter for this kind of image processing.
  • the display control unit 46 controls the main display 20 so as to superimpose a rectangular outline (just an example) for emphasizing the recognized pedestrian, on the image of the recognized pedestrian.
  • Superimposing the rectangular outline is a so-called overlay display, and is carried out in such a manner that part of pixel values that indicate the image captured by the infrared camera 12 are replaced with a desired pixel value.
  • a desired pixel value hereinafter, such a image is termed emphasized image.
  • the display control unit 46 controls the sub-display 30 so as to display information for notifying a driver of the presence of the pedestrian, such as an icon that indicates a pedestrian, at a desired position.
  • the desired position may be always constant; however, it may be changed depending on, for example, whether a pedestrian is recognized on the right side or left side of the vehicle.
  • the above information such as an icon that indicates a pedestrian, is termed warning information.
  • FIG. 3 is a view that shows a state where the infrared image ahead of the vehicle and the emphasized image are displayed on the main display 20 and the warning information is displayed on the sub-display 30.
  • the image capturing range of the infrared camera 12 has an angle narrower than that of the irradiation range of the head lamps, so even a pedestrian that has disappeared from the image displayed on the main display 20 may possibly remain in the field of view of the driver (which is assumed to be substantially equal to the irradiation range of the head lamps in the nighttime).
  • FIG. 5 is a view that shows the relationship between the image capturing range of the infrared camera 12 and the field of view of a driver.
  • the display control unit 46 displays warning information on the sub-display 30 only when the pedestrian recognition unit 42 determines that there is a pedestrian in the image captured by the infrared camera 12 (in the case of (A) in FIG. 5), the warning information disappears from the sub-display 30 notwithstanding the pedestrian that has disappeared from the image displayed on the main display 20 remains T/IB2010/002532
  • the driver may experience a feeling of strangeness.
  • the vehicle night vision support system 1 includes the pedestrian estimation unit 44 that estimates (determines) whether a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver. Even when the pedestrian estimation unit 44 estimates that a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver (in the case of (B) in FIG. 5), the warning information is still displayed on the sub-display 30.
  • the pedestrian estimation unit 44 performs estimation in the following steps. First, a relative position of a pedestrian, recognized by the pedestrian recognition unit 42, with respect to the vehicle is calculated on the basis of the position of the pedestrian on the image (actually, the pedestrian has a predetermined size, so the feet, or the like, are used as representative points). The relative position is, for example, calculated in the form of a combination of a distance and a bearing angle with reference to the central axis of the vehicle. Subsequently, the velocity vector of the vehicle is calculated on the basis of the vehicle speed signal input from the vehicle speed sensor 50 and the steering angle signal input from the steering angle sensor 52.
  • a plurality of future positions of the vehicle are calculated to thereby calculate a period of time that is taken until the vehicle reaches the position at which the pedestrian falls outside the field of view of the driver for the first time. It may be estimated that the pedestrian is present in the field of view of the driver during a period of time from the timing the pedestrian recognition unit 42 recognizes the pedestrian to the timing the vehicle reaches the position at which the pedestrian falls outside the field of view of the driver.
  • the sub-display 30 continuously provides the warning information.
  • the warning information display of the sub-display 30 it is possible to contribute to further safe driving.
  • FIG. 4 is a view that shows a state where the emphasized image disappears from the main display 20 but the warning information is still displayed on the sub-display 30.
  • FIG. 6 is a view that shows a state where it is determined that the pedestrian has reached the outside of the field of view of the driver and then the warning information display is turned off.
  • FIG. 7 is a flowchart that shows the flow of characteristic processes executed by the night vision support ECU 40 according to the first embodiment. This flow is repeatedly executed while the vehicle night vision support system 1 is in operation.
  • the pedestrian recognition unit 42 determines whether there is a pedestrian in an image captured by the infrared camera 12 (S100).
  • the display control unit 46 controls the main display 20 so as to emphasize the pedestrian on the image (to display an emphasized image) and controls the sub-display 30 so as to display warning information (S102).
  • the pedestrian estimation unit 44 estimates (determines) whether a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver (S104).
  • the display control unit 46 controls the sub-display 30 so as to display the warning information (S106).
  • the sub-display 30 continuously provides warning information, so it is possible to prevent the driver from experiencing a feeling of strangeness.
  • the warning information display of the sub-display 30 in such a situation that requires attention when driving, it is possible to contribute to further safe driving.
  • vehicle night vision support system 2 according to a second embodiment of the invention will be described.
  • the hardware configuration of the vehicle night vision support system 2 is the same as that of the vehicle night vision support system 1 according to the first embodiment, so the description of the basic configuration, or the like, is omitted and the difference from the first embodiment will be specifically described.
  • the vehicle night vision support system 2 As the vehicle night vision support system 2 according to the present embodiment is activated, only the center portion, excluding the right and left ends, of an image captured by the infrared camera 12 is displayed on the main display 20.
  • Such a display is intended to, for example, avoid hunting of emphasized image.
  • an emphasized image is displayed on the main display 20 and warning information is provided on the sub-display 30 when there is a pedestrian in the center portion, excluding the right and left ends, of the image captured by the infrared camera 12.
  • warning information is provided on the sub-display 30 when there is a pedestrian in the right or left end of the image captured by the infrared camera 12 or when the pedestrian recognition unit 42 estimates that a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver.
  • FIG. 8 is a flowchart that shows the flow of characteristic processes executed by the night vision support ECU 40 according to the second embodiment. This flow is repeatedly executed while the vehicle night vision support system 2 is in operation.
  • the pedestrian recognition unit 42 determines whether there is a pedestrian in an image captured by the infrared camera 12 (S200).
  • the pedestrian recognition unit 42 determines that there is a pedestrian in the image captured by the infrared camera 12, the pedestrian recognition unit 42 further determines whether there is a pedestrian in the center portion of the captured image (S202).
  • the display control unit 46 controls the main display 20 so as to display an emphasized image of the pedestrian and controls the sub-display 30 so as to display warning information (S204).
  • the display control unit 46 controls the sub-display 30 so as to display the warning information (S206).
  • the pedestrian estimation unit 44 estimates (determines) whether a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver (S208).
  • the display control unit 46 controls the sub-display 30 so as to display the warning information (S206).
  • the sub-display 30 continuously provide warning information, so it is possible to prevent the driver from experiencing a feeling of strangeness.
  • the warning information display of the sub-display 30 in such a situation that requires attention when driving, it is possible to contribute to further safe driving.
  • a warning target object intended for emphasized image or warning information may be an object other than a pedestrian.
  • specific embodiments of the main display 20 and sub-display 30 are not limited to the ones described in the above embodiments; instead, for example, a display unit provided in a combination meter may be used as the sub-display 30.
  • the period of time may be reflected on control.
  • the aspect of the invention is useable in automobile manufacturing, automobile parts manufacturing, and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Mechanical Engineering (AREA)
  • Optics & Photonics (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Traffic Control Systems (AREA)

Abstract

A vehicle night vision support system includes: an infrared camera (10) that captures an image ahead of a vehicle; a main display (20) that displays the image captured by the infrared camera; a sub-display (30); and a control unit (40) that controls the sub-display (30) so as to display warning information when it is determined that there is an object in the image displayed on the main display (20), and controls the sub-display (30) so as to display the warning information when it is estimated or detected that an object that has disappeared from the displayed image is present in the field of view of a driver.

Description

VEHICLE NIGHT VISION SUPPORT SYSTEM AND CONTROL METHOD FOR
THE SAME
BACKGROUND OF THE INVENTION
1. Field of the Invention
[0001] The invention relates to a vehicle night vision support system that displays an image, captured in the nighttime by an infrared camera mounted on a vehicle, on an in-vehicle display, or the like, so that the captured image can be visually recognized in a vehicle cabin, and a control method for the same.
2. Description of the Related Art
[0002] There is known an existing system in which an infrared camera that is able to, for example, recognize a pedestrian even in the nighttime captures an image and then the captured image is displayed in a vehicle cabin.
[0003] In relation to this, Japanese Patent Application Publication No. 2005-081860 (JP-A-2005-081860) describes a vehicle display system. The vehicle display system includes a driving environment detecting means that detects a driving environment ahead of a vehicle, and changes an area captured by an image capturing means, such as an infrared camera, into a wide angle depending on the detected driving environment. The vehicle display system detects, for example, the presence of a preceding vehicle as the driving environment ahead of the vehicle, and changes the captured area into a wide angle when there is the preceding vehicle.
[0004] In addition, Japanese Patent Application Publication No, 2005-088630 (JP-A-2005-088630) describes a vehicle night vision system. The vehicle night vision system swivels an image capturing means, such as an infrared camera, horizontally in a vehicle transverse direction on the basis of a steering angle signal detected by a steering angle sensor. Note that the image capturing means is swiveled horizontally in the vehicle transverse direction in synchronization with head lights. T/IB2010/002532
2
[0005] Incidentally, an image captured by the infrared camera is displayed on a display device (which is also used for, for example, a navigation display) installed at the center portion of an instrument panel, or the like, or is displayed on a head up display (HUD). The HUD is a display device that, for example, reflects light, emitted from a light-emitting device installed at the upper portion of the instrument panel, from a front glass to allow a driver to visually recognize the emitted light.
[0006] Then, there has been proceeding toward commercialization of a system that supports vision of a driver on the basis of an image captured by an infrared camera using such a plurality of display devices. In this system, an image captured by an infrared camera is, for example, displayed on a main display installed at the center portion of an instrument panel, or the like, and warning information, such as an icon that indicates the shape of an object, is displayed on a sub-display, such as an HUD, when the displayed image contains a warning target object, such as a pedestrian.
[0007] However, according to the existing systems, the driver may experience a feeling of strangeness because of a difference between the angle of view of the infrared camera and the field of view of the driver.
[0008] The angle of view of the infrared camera is normally narrower than the field of view of a driver in consideration of a resolution, a requirement for capturing an image at a location farther from the illumination limit of the head lamps, and the like. This is because the accuracy of remote image capturing decreases when the image capturing range is set to a wide angle.
[0009] Thus, in the process in which an object ahead of the vehicle approaches the vehicle and then passes by the vehicle to behind the vehicle, there is a period of time during which the driver visually recognizes the object although the object is not captured by the infrared camera (or is captured by the infrared camera but the object is located outside the display range). During the above period of time, the driver may experience a feeling of strangeness against a situation that the warning information of the sub-display disappears although the driver visually recognizes the pedestrian.
[0010] In addition, the situation that the object can be visually recognized by the driver still requires attention to the object when driving. Thus, when the warning information of the sub-display disappears, it may result in an insufficient support.
SUMMARY OF THE INVENTION
[0011] The invention provides a vehicle night vision support system that is able to contribute to further safe driving without making a driver experiencing a feeling , of strangeness, and a control method for the same.
[0012] A first aspect of the invention provides a vehicle night vision support system. The vehicle night vision support system includes: an infrared camera that captures an image ahead of a vehicle; a main display that displays the image captured by the infrared camera; a sub-display; and a control unit that determines whether there is an object in the displayed image on the main display and, when the control unit determines that there is the object in the displayed image, controls the sub-display so as to display warning information on the object, and the control unit estimates or detects whether an object that has disappeared from the displayed image is present in the field of view of the driver and, when the control unit estimates or detects that the object that has disappeared from the displayed image is present in the field of view of the driver, controls the sub-display so as to display the warning display.
[0013] A second aspect of the invention provides a control method for a vehicle night vision support system that includes an infrared camera that captures an image ahead of a vehicle, a main display that displays the image captured by the infrared camera and a sub-display. The control method includes: determining whether there is an object in the displayed image on the main display; when it is determined that there is the object in the displayed image, controlling the sub-display so as to display warning information on the object; estimating or detecting whether an object that has disappeared from the displayed image is present in the field of view of the driver; and when it is estimated or detected that the object that has disappeared from the displayed image is present in the field of view of the driver, controlling the sub-display so as to display the warning information.
[0014] With the first aspect and the second aspect of the invention, not only when there is an object, such as a pedestrian, in the image displayed on the main display but also when it is estimated or detected that an object that has disappeared from the image displayed on the main display is present in the field of view of the driver, the sub-display is controlled so as to display the warning information. Thus, it is possible to contribute to further safe driving without making the driver experience a feeling of strangeness.
[0015] In the aspects of the invention, the sub-display may be a head up display.
[0016] In the aspects of the invention, the vehicle night vision support system may further include: a vehicle speed detector that detects a vehicle speed; and a steering angle detector that detects a steering angle, wherein the control unit may calculate a relative position of the object with respect to the vehicle on the basis of a position of the object in the displayed image, and the control unit may estimate whether the object that has disappeared from the displayed image is present in the field of view of the driver on the basis of the relative position and a driving state of the vehicle, calculated on the basis of the vehicle speed detected by the vehicle speed detector and the steering angle detected by the steering angle detector.
[0017] In the aspects of the invention, the main display may display a center portion of the image captured by the infrared camera with respect to a horizontal direction of the captured image.
BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The foregoing and further objects, features and advantages of the invention will become apparent from the following description of example embodiments with reference to the accompanying drawings, wherein like, numerals are used to represent like elements and wherein:
FIG. 1 is an example of the system configuration of each of vehicle night vision support systems 1 and 2 according to first and second embodiments of the invention;
FIG. 2 is an example of the configuration of an infrared camera 12;
FIG. 3 is a view that shows a state where an infrared image ahead of a vehicle and an emphasized image are displayed on a main display 20 and warning information is displayed on a sub-display 30;
FIG. 4 is a view that shows a state where the emphasized image disappears from the main display 20 but the warning information is still displayed on the sub-display 30;
FIG. 5 is a view that shows the relationship between the image capturing range of the infrared camera 12 and the field of view of a driver;
FIG. 6 is a view that shows a state where it is determined that a pedestrian has reached the outside of the field of view of the driver and then the warning information is turned off;
FIG. 7 is a flowchart that shows the flow of characteristic processes executed by a night vision support ECU 40 according to the first embodiment; and
FIG. 8 is a flowchart that shows the flow of characteristic processes executed by a night vision support ECU 40 according to the second embodiment. DETAILED DESCRIPTION OF EMBODIMENTS
[0019] Hereinafter, embodiments of the invention will be described with reference to the accompanying drawings.
[0020] Hereinafter, a vehicle night vision support system 1 according to a first embodiment of the invention will be described with reference to the accompanying drawings. FIG. 1 is an example of the system configuration of the vehicle night vision support system 1 according to the first embodiment of the invention. The vehicle night vision support system 1 includes an infrared camera device 10, a main display 20, a sub-display 30 and a night vision support electronic control unit (ECU) 40 as major components.
[0021] The infrared camera device 10 includes an infrared irradiation device and an infrared camera 12. The infrared irradiation device is, for example, attached to a front grill, and irradiates near infrared forward of the vehicle. The infrared camera 12 is, for example, attached to the inside of the upper center portion of a front glass, and captures an image diagonally forward of the vehicle. [0022] Note that the infrared camera device 10 is not limited to a device that irradiates infrared to thereby capture reflected light but it may be a device that captures infrared radiated from a pedestrian because of body heat. In addition, because irradiated light of a head lamp and irradiated light of a fog lamp contain infrared, the infrared camera device 10 may be configured to mainly capture the light reflected from a pedestrian.
[0023] FIG. 2 is an example of the configuration of the infrared camera 12. The infrared camera 12 includes, for example, a visible light cut filter 13 and a charge coupled device (CCD) camera 14. The CCD camera 14 includes, for example, an image capturing lens 15, an image sensor 16 and an electronic control unit 17. Note that the infrared camera 12 may include a complementary metal-oxide-semiconductor (CMOS) camera, or the like, instead of the CCD camera. The visible light cut filter 13 is, for example, stuck on the front glass.
[0024] The image sensor 16 is, for example, an interline image sensor. The image sensor 16 has photodiodes, CCDs and transfer gates. The photodiodes are photoreceptors that carry out photoelectric conversion and arranged in a two-dimensional planar shape. The CCDs are arranged in a two-dimensional planar shape in correspondence with the photodiodes. The transfer gates are provided between the photodiodes and the CCDs, and function as analog switches. A light condensing microlens is attached in front (image capturing lens 15 side) of each photodiode. Note that the above structure is just schematically illustrated to simply describe the functions of the image sensor 16, and any design change for improvement in function, or the like, may be allowed. In addition, the image sensor 16 is not limited to the interline image sensor; it may be a full-frame transfer or frame transfer image sensor of which the CCD itself functions as a photoreceptor.
[0025] The electronic control unit 17 is, for example, a microcomputer, an electronic circuit, or the like. The electronic control unit 17 controls the open-close timing of each transfer gate of the image sensor 16 to regulate the shutter speed or image capturing period (for example, about several tens of times per one second) of the infrared camera 12. Then, the electronic control unit 17, for example, amplifies data, output from the output circuit of the image sensor 16, with a predetermined gain, and then outputs the amplified data to the night vision support ECU 40 as image data.
[0026] Here, the irradiation range of the infrared irradiation device substantially coincides with the irradiation range of the infrared camera 12. The image capturing range of the infrared camera 12 has an angle narrower than that of the irradiation range of the head lamps so as to be able to capture a more remote image.
[0027] The main display 20 is a touch panel display device. The main display 20 is installed at, for example, the center portion of an instrument panel, or the like, and uses a video graphics array (VGA) as a graphic system to display an image including a moving image. The main display 20 detects a variation in voltage caused by user's touching the display surface to thereby recognize a touched position^ With that, various GUI switches are set on the display surface to allow the user to conduct various input operations. In addition, a plurality of switches that allow various operations are provided around the display surface of the main display 20.
[0028] The main display 20 may be shared as a display/input device of another in-vehicle device, such as a navigation system. In this case, the main display 20 functions as the main display 20 only when the vehicle night vision support system 1 is in operation (in the nighttime), and functions as the display/input device of another in-vehicle device during times other than the nighttime.
[0029] The sub-display 30 is, for example, an HUD. The sub-display 30 projects an image at a display position on the front glass to display the image as a virtual image that is seen as if the image were present at a position on an extension of line of sight when the display position is viewed from the eyepoint of a driver.
[0030] The entire sub-display 30 is, for example, accommodated in a recess formed on the upper surface of the instrument panel. The sub-display 30 includes a liquid crystal display, a reflector, and the like. The liquid crystal display is arranged on the back side of a combination meter and emits display light toward the traveling direction of the vehicle. The reflector reflects light emitted from the liquid crystal „, ...„
PCT/IB2010/002532
8
display to thereby project the reflected light on a windshield.
[0031] The night vision support ECU 40 is, for example, a microcomputer in which a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), and the like, are connected to one another via a bus. The night vision support ECU 40 further includes a storage device, such as a hard disc drive (HDD), a digital versatile disk (DVD) drive, a compact disc-recordable (CD-R) drive and an electronically erasable and programmable read only memory (EEPROM), an I/O port, a timer, a counter, and the like. The storage device stores programs and data executed by the CPU.
[0032] In-vehicle sensors, such as a vehicle speed sensor 50 and a steering angle sensor 52, are connected to the night vision support ECU 40 via an in-vehicle communication line, such as a multiplex communication line. The vehicle speed sensor 50 includes, for example, a skid control computer and a wheel speed sensor attached to each wheel of the vehicle. The skid control computer converts a wheel speed pulse signal, output from each wheel speed sensor, into a vehicle speed rectangular wave pulse signal (a vehicle speed signal), and outputs the vehicle speed signal. Each wheel speed sensor, for example, includes a magnetic rotor and an active sensor. In the magnetic rotor, rubber is filled with magnetic powder and then positive electrodes and negative electrodes are alternately arranged in the circumferential direction. The active sensor detects a variation in magnetic field due to rotation of the magnetic rotor. The steering angle sensor 52 is, for example, arranged inside a steering column, and outputs a steering angle signal. Communication among the devices via the multiplex communication line is carried out using an appropriate communication protocol. The communication protocol, for example, includes a low-speed body communication protocol, a multimedia communication protocol and a FlexRay. The low-speed body communication protocol is typically a controller area network (CAN) or a local interconnect network (LIN). The multimedia communication protocol is typically a media oriented systems transport (MOST).
[0033] In addition, the night vision support ECU 40 includes a pedestrian recognition unit 42, a. pedestrian estimation unit 44 and a display control unit 46 as major functional blocks. The major functional blocks function in such a manner that the CPU executes the programs stored in the storage device. Note that these functional blocks need not be based on distinctly different programs; one program may contain portions that implement a plurality of functional blocks.
[0034] Here, the vehicle night vision support system 1 is activated by a command from a user through, for example, a GUI switch prepared on the display surface of the main display 20 or a switch attached around the main display 20. Instead, the vehicle night vision support system 1 may be automatically activated when the outside illuminance detected by an illuminance sensor is lower than a predetermined value.
[0035] As the vehicle night vision support system 1 is activated, an image captured by the infrared camera 12 is displayed on the main display 20.
[0036] The pedestrian recognition unit 42 determines whether there is a pedestrian, for example, in the image captured by the infrared camera 12 (in the present embodiment, the captured image is the same as the image displayed on the main display 20). The recognition of a pedestrian in the pedestrian recognition unit 42 is carried out by, for example, a step of extracting feature points from the image captured by the infrared camera 12 and a step of applying pattern matching to a series of feature points indicating the outline of an object to determine whether the object is a pedestrian. The feature points may be simply defined as points of which the luminance difference between the adjacent pixels is larger than or equal to a predetermined value; instead, characteristic points may be extracted using a specific filter for this kind of image processing.
[0037] When the pedestrian recognition unit 42 determines that there is a pedestrian in the image captured by the infrared camera 12, the display control unit 46 controls the main display 20 so as to superimpose a rectangular outline (just an example) for emphasizing the recognized pedestrian, on the image of the recognized pedestrian. Superimposing the rectangular outline is a so-called overlay display, and is carried out in such a manner that part of pixel values that indicate the image captured by the infrared camera 12 are replaced with a desired pixel value. Hereinafter, such a image is termed emphasized image.
[0038] In addition, when the pedestrian recognition unit 42 determines that there is a pedestrian in the image captured by the infrared camera 12, the display control unit 46 controls the sub-display 30 so as to display information for notifying a driver of the presence of the pedestrian, such as an icon that indicates a pedestrian, at a desired position. The desired position may be always constant; however, it may be changed depending on, for example, whether a pedestrian is recognized on the right side or left side of the vehicle. Hereinafter, the above information, such as an icon that indicates a pedestrian, is termed warning information.
[0039] Through the above controls, when there is a pedestrian ahead of the vehicle, the driver initially sees the sub-display 30 to recognize the presence of the pedestrian, and then looks at the main display 20 to check the position and behavior of the pedestrian. By so doing, it is possible for the driver to safely drive the vehicle so as not to have a collision, or the like. FIG. 3 is a view that shows a state where the infrared image ahead of the vehicle and the emphasized image are displayed on the main display 20 and the warning information is displayed on the sub-display 30.
[0040] However, as described above, the image capturing range of the infrared camera 12 has an angle narrower than that of the irradiation range of the head lamps, so even a pedestrian that has disappeared from the image displayed on the main display 20 may possibly remain in the field of view of the driver (which is assumed to be substantially equal to the irradiation range of the head lamps in the nighttime). FIG. 5 is a view that shows the relationship between the image capturing range of the infrared camera 12 and the field of view of a driver.
[0041] Thus, if the display control unit 46 displays warning information on the sub-display 30 only when the pedestrian recognition unit 42 determines that there is a pedestrian in the image captured by the infrared camera 12 (in the case of (A) in FIG. 5), the warning information disappears from the sub-display 30 notwithstanding the pedestrian that has disappeared from the image displayed on the main display 20 remains T/IB2010/002532
11
in the field of view of the driver. Therefore, the driver may experience a feeling of strangeness.
• [0042] Then, the vehicle night vision support system 1 according to the present embodiment includes the pedestrian estimation unit 44 that estimates (determines) whether a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver. Even when the pedestrian estimation unit 44 estimates that a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver (in the case of (B) in FIG. 5), the warning information is still displayed on the sub-display 30.
[0043] The pedestrian estimation unit 44, for example, performs estimation in the following steps. First, a relative position of a pedestrian, recognized by the pedestrian recognition unit 42, with respect to the vehicle is calculated on the basis of the position of the pedestrian on the image (actually, the pedestrian has a predetermined size, so the feet, or the like, are used as representative points). The relative position is, for example, calculated in the form of a combination of a distance and a bearing angle with reference to the central axis of the vehicle. Subsequently, the velocity vector of the vehicle is calculated on the basis of the vehicle speed signal input from the vehicle speed sensor 50 and the steering angle signal input from the steering angle sensor 52. Then, by integrating the velocity vector of the vehicle in stages (by varying the time range of integration) with the position of the pedestrian invariant, a plurality of future positions of the vehicle are calculated to thereby calculate a period of time that is taken until the vehicle reaches the position at which the pedestrian falls outside the field of view of the driver for the first time. It may be estimated that the pedestrian is present in the field of view of the driver during a period of time from the timing the pedestrian recognition unit 42 recognizes the pedestrian to the timing the vehicle reaches the position at which the pedestrian falls outside the field of view of the driver.
[0044] With the above control, when a pedestrian that has disappeared from the image displayed on the main display 20 remains in the field of view of the driver, the sub-display 30 continuously provides the warning information. Thus, it is possible to prevent the driver from experiencing a feeling of strangeness. In addition, such a situation that a pedestrian remains in the field of view of the driver still requires attention to the pedestrian when driving. Thus, by continuing the warning information display of the sub-display 30, it is possible to contribute to further safe driving.
[0045] FIG. 4 is a view that shows a state where the emphasized image disappears from the main display 20 but the warning information is still displayed on the sub-display 30.
[0046] Then, when the period of time from the timing the pedestrian recognition unit 42 recognizes a pedestrian to the timing the vehicle reaches the position at which the pedestrian falls outside the field of view of the driver has elapsed, the pedestrian estimation unit 44 determines that no pedestrian is present in the field of view of the driver. When the pedestrian estimation unit 44 determines that no pedestrian is present in the field of view of the driver, the display control unit 46 controls the sub-display 30 so as to turn off the warning information display. FIG. 6 is a view that shows a state where it is determined that the pedestrian has reached the outside of the field of view of the driver and then the warning information display is turned off.
[0047] Hereinafter, the flow of the above described processes will be described using a flowchart. FIG. 7 is a flowchart that shows the flow of characteristic processes executed by the night vision support ECU 40 according to the first embodiment. This flow is repeatedly executed while the vehicle night vision support system 1 is in operation.
[0048] First, the pedestrian recognition unit 42 determines whether there is a pedestrian in an image captured by the infrared camera 12 (S100).
[0049] When the pedestrian recognition unit 42 determines that there is a pedestrian in the image captured by the infrared camera 12, the display control unit 46 controls the main display 20 so as to emphasize the pedestrian on the image (to display an emphasized image) and controls the sub-display 30 so as to display warning information (S102).
[0050] When the pedestrian recognition unit 42 determines that there is no 2
13
pedestrian in the image captured by the infrared camera 12, the pedestrian estimation unit 44 estimates (determines) whether a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver (S104).
[0051] When the pedestrian estimation unit 44 determines that a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver, the display control unit 46 controls the sub-display 30 so as to display the warning information (S106).
[0052] On the other hand, when the pedestrian estimation unit 44 determines that no pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver, neither emphasized image nor warning information is displayed (S108).
[0053] With the vehicle night vision support system 1 according to the above described embodiment, when a pedestrian that has disappeared from an image displayed on the main display 20 remains in the field of view of the driver, the sub-display 30 continuously provides warning information, so it is possible to prevent the driver from experiencing a feeling of strangeness. In addition, by continuing the warning information display of the sub-display 30 in such a situation that requires attention when driving, it is possible to contribute to further safe driving.
[0054] Hereinafter, a vehicle night vision support system 2 according to a second embodiment of the invention will be described. The hardware configuration of the vehicle night vision support system 2 is the same as that of the vehicle night vision support system 1 according to the first embodiment, so the description of the basic configuration, or the like, is omitted and the difference from the first embodiment will be specifically described.
[0055] As the vehicle night vision support system 2 according to the present embodiment is activated, only the center portion, excluding the right and left ends, of an image captured by the infrared camera 12 is displayed on the main display 20. Such a display is intended to, for example, avoid hunting of emphasized image.
[0056] Thus, even when the pedestrian recognition unit 42 determines that there is a pedestrian in the image captured by the infrared camera 12, there is a possibility that neither pedestrian nor emphasized image is displayed on the main display 20.
[0057] In the present embodiment, an emphasized image is displayed on the main display 20 and warning information is provided on the sub-display 30 when there is a pedestrian in the center portion, excluding the right and left ends, of the image captured by the infrared camera 12.
[0058] On the other hand, no emphasized image is displayed on the main display 20 but warning information is provided on the sub-display 30 when there is a pedestrian in the right or left end of the image captured by the infrared camera 12 or when the pedestrian recognition unit 42 estimates that a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver.
[0059] That is, when there is a pedestrian in the center portion, excluding the right and left ends, of the image captured by the infrared camera 12, both an emphasized image and warning information are displayed. In addition, when there is a pedestrian in the right or left end of the image captured by the infrared camera 12 (when the presence of a pedestrian is detected through image recognition) or when the pedestrian recognition unit 42 estimates that a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver, only warning inforamtion is displayed.
[0060] Hereinafter, the flow of the above described processes will be described using a flowchart. FIG. 8 is a flowchart that shows the flow of characteristic processes executed by the night vision support ECU 40 according to the second embodiment. This flow is repeatedly executed while the vehicle night vision support system 2 is in operation.
[0061] First, the pedestrian recognition unit 42 determines whether there is a pedestrian in an image captured by the infrared camera 12 (S200).
[0062] When the pedestrian recognition unit 42 determines that there is a pedestrian in the image captured by the infrared camera 12, the pedestrian recognition unit 42 further determines whether there is a pedestrian in the center portion of the captured image (S202).
[0063] When it is determined that there is a pedestrian in the center portion of the captured image, the display control unit 46 controls the main display 20 so as to display an emphasized image of the pedestrian and controls the sub-display 30 so as to display warning information (S204).
[0064] In addition, when it is determined that the pedestrian recognition unit 42 determines that there is a pedestrian in the right or left end of the captured image, the display control unit 46 controls the sub-display 30 so as to display the warning information (S206).
[0065] On the other hand, when the pedestrian recognition unit 42 determines that there is no pedestrian in the image captured by the infrared camera 12, the pedestrian estimation unit 44 estimates (determines) whether a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver (S208).
[0066] When the pedestrian estimation unit 44 determines that a pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver, the display control unit 46 controls the sub-display 30 so as to display the warning information (S206).
[0067] When the pedestrian estimation unit 44 determines that no pedestrian that has disappeared from the image displayed on the main display 20 is present in the field of view of the driver, neither emphasized image nor warning information is displayed (S210).
[0068] . With the vehicle night vision support system 2 according to the above described embodiment, when a pedestrian that has disappeared from an image displayed on the main display 20 remains in the field of view of the driver, the sub-display 30 continuously provide warning information, so it is possible to prevent the driver from experiencing a feeling of strangeness. In addition, by continuing the warning information display of the sub-display 30 in such a situation that requires attention when driving, it is possible to contribute to further safe driving.
[0069] The aspect of the invention is not limited to the embodiments described above; it may be modified or replaced in various forms without departing from the scope of the invention.
[0070] For example, a warning target object intended for emphasized image or warning information may be an object other than a pedestrian. In addition, specific embodiments of the main display 20 and sub-display 30 are not limited to the ones described in the above embodiments; instead, for example, a display unit provided in a combination meter may be used as the sub-display 30.
[0071] In addition, in consideration of a period of time from when the driver sees warning information on the sub-display 30 to when the driver looks at the main display 20, the period of time may be reflected on control.
[0072] The aspect of the invention is useable in automobile manufacturing, automobile parts manufacturing, and the like.

Claims

CLAIMS:
1. A vehicle night vision support system comprising:
an infrared camera that captures an image ahead of a vehicle;
a main display that displays the image captured by the infrared camera;
a sub-display; and
a control unit that determines whether there is an object in the displayed image on the main display and, when the- control unit determined that there is the object in the displayed image, controls the sub-display so as to display warning information on the object, and the control unit estimates or detects whether an object that has disappeared from the displayed image is present in the field of view of the driver and, when the control unit estimates or detects that the object that has disappeared from the displayed image is present in the field of view of the driver, controls the sub-display so as to display the warning information.
2. The vehicle night vision support system according to claim 1, wherein the sub-display is a head up display.
3. The vehicle night vision support system according to claim 1 or 2, further comprising:
a vehicle speed detector that detects a vehicle speed; and
a steering angle detector that detects a steering angle, wherein
the control unit calculates a relative position of the object with respect to the vehicle on the basis of a position of the object in the displayed image, and
the control unit estimates whether the object that has disappeared from the displayed image is present in the field of view of the driver on the basis of the relative position and a driving state of the vehicle, calculated on the basis of the vehicle speed detected by the vehicle speed detector and the steering angle detected by the steering angle detector.
4. The vehicle night vision support system according to claim 3, wherein the control unit calculates a future position of the vehicle on the basis of the driving state of the vehicle, and
the control unit estimates whether the object that has disappeared from the displayed image is present in the field of view of the driver on the basis of a position of the vehicle which is calculated on the basis of the future position of the vehicle and at which the object falls outside the field of view of the driver for the first time.
5. The vehicle night vision support system according to claim 4, wherein until the vehicle reaches the position of the vehicle at which the object falls outside the field of view of the driver for the first time, the control unit estimates that the object that has disappeared from the displayed image is present in the field of view of the driver.
6. The vehicle night vision support system according to any one of claims 1 to 5, wherein
the main display displays a center portion of the image captured by the infrared camera with respect to a horizontal direction of the captured image.
7. The vehicle night vision support system according to claim 6, wherein
the control unit determines whether there is an object in the captured image other than the center portion and, when the control unit determines that there is the object in the captured image other than the center portion, controls the sub-display so as to display the warning information.
8. The vehicle night vision support system according to any one of claims 1 to 7, wherein
when the control unit determines that there is the object in the displayed image, the control unit controls the main display so as to emphasize the object on the displayed image.
9. The vehicle night vision support system according to any one of claims 1 to 8, wherein
the object is a pedestrian.
10. A control method for a vehicle night vision support system that includes an infrared camera that captures an image ahead of a vehicle, a main display that displays the image captured by the infrared camera and a sub-display, the control method comprising: determining whether there is an object in the displayed image on the main display; when it is determined that there is the object in the displayed image, controlling the sub-display so as to display warning information on the object;
estimating or detecting whether an object that has disappeared from the displayed image is present in the field of view of the driver; and
when it is estimated or detected that the object that has disappeared from the displayed image is present in the field of view of the driver, controlling the sub-display so as to display the warning information.
PCT/IB2010/002532 2009-10-21 2010-10-07 Vehicle night vision support system and control method for the same WO2011048454A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US13/391,157 US9061632B2 (en) 2009-10-21 2010-10-07 Vehicle night vision support system and control method for the same
DE112010004133.4T DE112010004133B4 (en) 2009-10-21 2010-10-07 Night vision support system for a vehicle and control method therefor
CN201080047902.0A CN102596643B (en) 2009-10-21 2010-10-07 Vehicle night vision support system and control method for the same

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2009242481A JP4952765B2 (en) 2009-10-21 2009-10-21 Vehicle night vision support device
JP2009-242481 2009-10-21

Publications (2)

Publication Number Publication Date
WO2011048454A1 true WO2011048454A1 (en) 2011-04-28
WO2011048454A8 WO2011048454A8 (en) 2011-06-30

Family

ID=43432085

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2010/002532 WO2011048454A1 (en) 2009-10-21 2010-10-07 Vehicle night vision support system and control method for the same

Country Status (5)

Country Link
US (1) US9061632B2 (en)
JP (1) JP4952765B2 (en)
CN (1) CN102596643B (en)
DE (1) DE112010004133B4 (en)
WO (1) WO2011048454A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR2985073A1 (en) * 2011-12-22 2013-06-28 Carlos Leal Method for controlling traffic using satellite image, involves utilizing satellite for detection of weather conditions, and providing information indicating approach of human or animal in selected radius to driver
EP2759999A1 (en) * 2011-09-21 2014-07-30 Honda Motor Co., Ltd. Apparatus for monitoring surroundings of vehicle
DE102017100880A1 (en) 2017-01-18 2018-07-19 Visteon Global Technologies, Inc. Display device and method for display in a vehicle
FR3062742A1 (en) * 2017-02-07 2018-08-10 Peugeot Citroen Automobiles Sa AID DEVICE FOR NIGHT ROAD NAVIGATION

Families Citing this family (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5582008B2 (en) * 2010-12-08 2014-09-03 トヨタ自動車株式会社 Vehicle information transmission device
JP2012155655A (en) * 2011-01-28 2012-08-16 Sony Corp Information processing device, notification method, and program
US9143703B2 (en) * 2011-06-10 2015-09-22 Flir Systems, Inc. Infrared camera calibration techniques
JP5547159B2 (en) * 2011-10-18 2014-07-09 本田技研工業株式会社 Vehicle periphery monitoring device
JP5604011B2 (en) * 2011-10-31 2014-10-08 本田技研工業株式会社 Vehicle periphery monitoring device
JP5703192B2 (en) * 2011-11-04 2015-04-15 本田技研工業株式会社 Vehicle periphery monitoring device
CN104602982B (en) * 2012-09-04 2017-04-26 丰田自动车株式会社 Collision avoidance assistance device and collision avoidance assistance method
JP6076017B2 (en) * 2012-09-28 2017-02-08 矢崎総業株式会社 Vehicle display device
US9481301B2 (en) * 2012-12-05 2016-11-01 Magna Electronics Inc. Vehicle vision system utilizing camera synchronization
CN103395393B (en) * 2013-08-13 2016-03-16 北京汽车研究总院有限公司 A kind of front windshield projection system thereof and automobile
JP6287237B2 (en) * 2014-01-17 2018-03-07 株式会社リコー Information processing apparatus, information processing method, and program
JP6252365B2 (en) * 2014-06-11 2017-12-27 株式会社デンソー Safety confirmation support system, safety confirmation support method
US9262924B2 (en) * 2014-07-09 2016-02-16 Toyota Motor Engineering & Manufacturing North America, Inc. Adapting a warning output based on a driver's view
US9469248B2 (en) * 2014-10-10 2016-10-18 Honda Motor Co., Ltd. System and method for providing situational awareness in a vehicle
US9588340B2 (en) 2015-03-03 2017-03-07 Honda Motor Co., Ltd. Pedestrian intersection alert system and method thereof
US9505346B1 (en) 2015-05-08 2016-11-29 Honda Motor Co., Ltd. System and method for warning a driver of pedestrians and other obstacles
KR101683509B1 (en) * 2015-05-20 2016-12-07 현대자동차 주식회사 For preventing glaring by the head lamp and method for preventing glaring using the same
US11195026B2 (en) 2016-06-29 2021-12-07 Kyocera Corporation Object detection and display apparatus, moveable body, and object detection and display method
WO2018066307A1 (en) * 2016-10-06 2018-04-12 富士フイルム株式会社 Projection display apparatus, display control method therefor, and program
WO2018110724A1 (en) * 2016-12-13 2018-06-21 주식회사 토비스 Night vision system for displaying thermal energy information and control method therefor
EA037887B1 (en) * 2017-03-30 2021-06-01 Агк Гласс Юроп Glass trim element for a motor vehicle
JP6925848B2 (en) * 2017-04-13 2021-08-25 アルパイン株式会社 Display control device, display control method and camera monitoring system
WO2019039201A1 (en) * 2017-08-23 2019-02-28 日本精機株式会社 Display device for vehicles
US10666880B2 (en) 2017-10-31 2020-05-26 Adasky, Ltd. Infrared camera assembly for a vehicle
US10175112B1 (en) 2017-12-14 2019-01-08 Adasky, Ltd Compact infrared camera for automotive safety and driving systems
US11432375B2 (en) 2017-10-31 2022-08-30 Adasky, Ltd. Protective window for resistive heating
CN108055476B (en) * 2017-11-23 2020-03-27 中国航空工业集团公司洛阳电光设备研究所 Method for eliminating vision errors of front-view infrared equipment and head-up display
JP7006235B2 (en) * 2017-12-18 2022-01-24 トヨタ自動車株式会社 Display control device, display control method and vehicle
CN108437891A (en) * 2018-03-10 2018-08-24 佛山杰致信息科技有限公司 A kind of intelligent driving system and method suitable for night-environment
US10958830B2 (en) 2018-05-24 2021-03-23 Magna Electronics Inc. Vehicle vision system with infrared LED synchronization
US11153950B2 (en) * 2019-02-14 2021-10-19 Orlaco Products B.V. Replacement mirror system with IR LED overheating management
CN109795415A (en) * 2019-02-19 2019-05-24 上海理工大学 A kind of Intelligent night vision head-up display device
IL265495B (en) * 2019-03-19 2022-09-01 Rober Ohrenstein Traffic allowance method
JP7228472B2 (en) * 2019-06-07 2023-02-24 本田技研工業株式会社 Recognition device, recognition method, and program
US11706380B2 (en) 2020-09-17 2023-07-18 Adasky, Ltd. Radiometric camera with black body elements for screening infectious disease carriers and method for calibrating a thermal camera having internal black body elements
US11968639B2 (en) 2020-11-11 2024-04-23 Magna Electronics Inc. Vehicular control system with synchronized communication between control units
CN113071412A (en) * 2021-05-17 2021-07-06 浙江工业大学 Vehicle-mounted infrared monitoring system and method

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE10253509A1 (en) * 2002-11-16 2004-06-17 Robert Bosch Gmbh Method and device for warning the driver of a motor vehicle
JP2005081860A (en) 2003-09-04 2005-03-31 Nissan Motor Co Ltd Display device for vehicle
JP2005088630A (en) 2003-09-12 2005-04-07 Nissan Motor Co Ltd Noctovision system for vehicle, interlocked with headlight for curved road
US20060006331A1 (en) * 2004-07-12 2006-01-12 Siemens Aktiengesellschaft Method for representing a front field of vision from a motor vehicle
WO2007028630A1 (en) * 2005-09-08 2007-03-15 Johnson Controls Gmbh Driver assistance system for a vehicle and method for visualising the environment of a vehicle
DE102007045932A1 (en) * 2007-09-26 2009-04-02 Daimler Ag Warning method for driver of vehicle, involves indicating optical warning signal in instantaneous viewing field of driver during detection of danger source, where signal contains reference to direction in which source is located

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3864406B2 (en) * 1999-01-26 2006-12-27 マツダ株式会社 Vehicle display device
US7852462B2 (en) * 2000-05-08 2010-12-14 Automotive Technologies International, Inc. Vehicular component control methods based on blind spot monitoring
JP4156214B2 (en) * 2001-06-13 2008-09-24 株式会社デンソー Vehicle periphery image processing apparatus and recording medium
JP3999025B2 (en) * 2002-04-12 2007-10-31 本田技研工業株式会社 Alarm display device
JP3982504B2 (en) * 2004-01-23 2007-09-26 日産自動車株式会社 Night-vision camera device and display device for vehicle
DE102006003538B3 (en) * 2006-01-24 2007-07-19 Daimlerchrysler Ag Image acquisitions merging method for bird`s eye perspective, involves selecting image sections such that shadowing effects produced by moved objects are projected during transition in overall image from image section to other section
JP5022609B2 (en) * 2006-02-27 2012-09-12 日立オートモティブシステムズ株式会社 Imaging environment recognition device
WO2007135856A1 (en) * 2006-05-19 2007-11-29 Pioneer Corporation Photographing control apparatus, photographing control method, photographing control program and recording medium
JP2008040974A (en) * 2006-08-09 2008-02-21 Denso Corp Driving assistance system
DE102006059033A1 (en) * 2006-12-14 2008-06-19 Volkswagen Ag Method and system for recognizing a road user and for generating a warning
JP4852159B2 (en) * 2010-02-24 2012-01-11 株式会社東芝 Image processing apparatus, image processing method, and air conditioning control apparatus
JP5491235B2 (en) * 2010-03-02 2014-05-14 東芝アルパイン・オートモティブテクノロジー株式会社 Camera calibration device

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE10253509A1 (en) * 2002-11-16 2004-06-17 Robert Bosch Gmbh Method and device for warning the driver of a motor vehicle
JP2005081860A (en) 2003-09-04 2005-03-31 Nissan Motor Co Ltd Display device for vehicle
JP2005088630A (en) 2003-09-12 2005-04-07 Nissan Motor Co Ltd Noctovision system for vehicle, interlocked with headlight for curved road
US20060006331A1 (en) * 2004-07-12 2006-01-12 Siemens Aktiengesellschaft Method for representing a front field of vision from a motor vehicle
WO2007028630A1 (en) * 2005-09-08 2007-03-15 Johnson Controls Gmbh Driver assistance system for a vehicle and method for visualising the environment of a vehicle
DE102007045932A1 (en) * 2007-09-26 2009-04-02 Daimler Ag Warning method for driver of vehicle, involves indicating optical warning signal in instantaneous viewing field of driver during detection of danger source, where signal contains reference to direction in which source is located

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2759999A1 (en) * 2011-09-21 2014-07-30 Honda Motor Co., Ltd. Apparatus for monitoring surroundings of vehicle
EP2759999A4 (en) * 2011-09-21 2015-04-15 Honda Motor Co Ltd Apparatus for monitoring surroundings of vehicle
FR2985073A1 (en) * 2011-12-22 2013-06-28 Carlos Leal Method for controlling traffic using satellite image, involves utilizing satellite for detection of weather conditions, and providing information indicating approach of human or animal in selected radius to driver
DE102017100880A1 (en) 2017-01-18 2018-07-19 Visteon Global Technologies, Inc. Display device and method for display in a vehicle
FR3062742A1 (en) * 2017-02-07 2018-08-10 Peugeot Citroen Automobiles Sa AID DEVICE FOR NIGHT ROAD NAVIGATION
WO2018146401A1 (en) * 2017-02-07 2018-08-16 Psa Automobiles Sa Device for assisting night-time road navigation

Also Published As

Publication number Publication date
US9061632B2 (en) 2015-06-23
JP4952765B2 (en) 2012-06-13
US20120320207A1 (en) 2012-12-20
WO2011048454A8 (en) 2011-06-30
DE112010004133B4 (en) 2017-11-02
CN102596643B (en) 2015-03-04
DE112010004133T5 (en) 2012-10-11
CN102596643A (en) 2012-07-18
JP2011091549A (en) 2011-05-06

Similar Documents

Publication Publication Date Title
US9061632B2 (en) Vehicle night vision support system and control method for the same
US10523904B2 (en) Vehicle data recording system
US10397451B2 (en) Vehicle vision system with lens pollution detection
US10147323B2 (en) Driver assistance system with path clearance determination
US10692380B2 (en) Vehicle vision system with collision mitigation
EP3184365B1 (en) Display device for vehicle and control method thereof
JP6307895B2 (en) Vehicle periphery monitoring device
US8933797B2 (en) Video-based warning system for a vehicle
JP3201518U (en) System for detecting obstacles in the field of image sensor
US11858424B2 (en) Electronic device for displaying image by using camera monitoring system (CMS) side display mounted in vehicle, and operation method thereof
JP5629521B2 (en) Obstacle detection system and method, obstacle detection device
US10423842B2 (en) Vehicle vision system with object detection
US10974649B2 (en) Vehicular parking assist system using backup camera
EP2487907B1 (en) Vehicle peripheral monitoring device
US11081008B2 (en) Vehicle vision system with cross traffic detection
KR101601500B1 (en) System and Method for assisting vehicle driving for using smartphone
KR102552490B1 (en) Image processing apparatus and method for vehicle
KR102559534B1 (en) Method and apparatus for supporting driving lines of vehicle in bad weather
KR101592874B1 (en) Detecting system and method for blinking light at night
JP2020141155A (en) Peripheral image display device
KR20170019843A (en) Apparatus and method for monitoring environment

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 201080047902.0

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10773706

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 13391157

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 112010004133

Country of ref document: DE

Ref document number: 1120100041334

Country of ref document: DE

122 Ep: pct application non-entry in european phase

Ref document number: 10773706

Country of ref document: EP

Kind code of ref document: A1