JP2013100057A - Driving support device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a bird's eye view image display type driving support device that can cope with a change of an outline of an own vehicle caused by opening of a door, and suppresses generation of blind corner region from a driver.SOLUTION: The device includes: a bird's eye view image generation part that performs projective transformation of a shot image obtained by an on-vehicle camera for shooting a peripheral region of a vehicle in a virtual top view to generate a shot bird's eye view image; a vehicle body image output part that outputs a vehicle body bird's eye view image showing a vehicle body; a door image output part that outputs a transparent door bird's eye view image in which a region in the door bird's eye view image showing an open door other than a part of outside outline is made transparent or semitransparent; and an image synthesis part that generates a driving support image in which the shot bird's eye view image, the vehicle body bird's eye view image, and the transparent door bird's eye view image are synthesized as a monitor display image.

Description

  The present invention relates to a driving support apparatus that monitors and displays a bird's-eye view image generated by projective conversion of a captured image acquired by an in-vehicle camera that captures a peripheral region of a vehicle from an upper virtual viewpoint.

  A bird's-eye view image in a conventional driving support device is a bird's-eye view from directly above by projective transformation in which a captured image acquired by an in-vehicle camera is projected onto a projection plane parallel to the road surface, in particular, by projective transformation with a virtual viewpoint positioned vertically upward. An image is generated. At that time, a photographic image or an illustration image of the vehicle in plan view is prepared in advance, and the vehicle image is synthesized with the position of the vehicle in the overhead image. By displaying the bird's-eye view image on the monitor, the driver can grasp the road surface condition around the vehicle in a bird's-eye view. However, in a vehicle such as a passenger car, the outer contour of the vehicle changes considerably by opening a door, particularly a back door that swings upward about a horizontal axis. It is important to consider such changes in the outer contour of the vehicle when parked with loading and unloading.

  Patent Document 1 discloses a camera that is attached to a vehicle and that captures an image that indicates an open / closed range of a target door that is mounted on the vehicle, and the target door that is mounted on the vehicle and is captured by the camera. A peripheral monitoring and monitoring system including a display device that synthesizes and displays an image indicating an opening / closing range in the vehicle width direction is described. Further, the image indicating the opening / closing range of the target door displayed on the display device is converted into an image taken from directly above, or the trajectory of opening / closing the target door is obtained, and the image displayed on the display device is displayed. It is also described that the trajectory is superimposed on an image showing the opening / closing range of the target door. As a result, the driver can check the opening / closing range of the door in the form of an overhead view with the opening / closing locus line, but the door is opened from the opening / closing locus line displayed on the small monitor screen in a form superimposed on the captured image. It is not easy to intuitively grasp the change in the external shape of the vehicle.

  Patent Document 2 stores an image capturing unit that captures an image of the surroundings of a vehicle, a video converting unit that converts video captured by the image capturing unit, a vehicle state detecting unit that detects a vehicle state, and an illustration image of the vehicle. A storage unit that superimposes an illustration image of the vehicle stored in the storage unit on a video output from the video conversion unit, and a display unit that displays the video output from the superimposition processing unit. A vehicle information display device is described in which the superimposition processing unit changes and superimposes the illustration image of the vehicle according to the vehicle state detected by the vehicle state detection unit. For example, in this vehicle information display device, by using a vehicle illustration in a door open state, it is possible to follow a change in the outer contour of the vehicle due to the opened door. However, in such a display mode, there arises a problem that the photographing area hidden by the door becomes a blind spot for the driver due to the opened door.

JP 2007-118762 (paragraph number [0005-0010], FIG. 1) JP 2009-023471 A (paragraph number [0006-0022], FIGS. 1 and 10)

  In view of the above situation, there is a demand for a driving support device of a bird's-eye view image display type that can cope with a change in the outer contour of the vehicle due to the opening of the door and suppress the generation of a blind spot area from the driver.

  According to the present invention, a driving support device for a vehicle having a door that opens to the outside of the vehicle body is characterized in that a captured image acquired by an in-vehicle camera that captures a peripheral region of the vehicle is projected and converted from an upper virtual viewpoint. The region other than the outer contour portion in the door overhead image showing the door in the open state is made transparent or semi-transparent. A door image output unit for outputting a transparent door bird's-eye view image, an image composition unit for generating, as a monitor display image, a driving support image obtained by combining the shooting bird's-eye view image, the vehicle body bird's-eye view image, and the transparent door bird's-eye view image; It is in the point with.

  According to this configuration, when displaying a bird's-eye view image around the vehicle, an image showing an open door can be added, and at that time, only the outer contour of the door is clearly displayed, The region other than the outer contour portion of the door, that is, the inner portion of the door is made transparent or translucent. Thereby, an object such as an obstacle on the road surface is not obscured by the inner portion of the door in the overhead view image. For example, during reverse parking for cargo handling, the driver can intuitively grasp how much the rear door pops out of the vehicle body when fully opened. In addition, at that time, the inside of the outer contour of the specified door is transparent, so the road surface below the door can be visually confirmed.

  Although it is necessary to display the outer contour of the door in the open state as clearly as possible, if the outer contour is made too wide, there is an inconvenience that more parts are covered. In order to make this contradictory relationship appropriate, in one preferred embodiment of the present invention, the transparent door overhead image has a transparent gradation that increases in transparency from the outside to the center of the inside. In this configuration, even if the opaque outer contour portion is narrowed, the transparency increases in gradation from the inside to the inside, so that it is easy to grasp the contour, and the area covered by the door (dead angle) The objective of minimizing as much as possible is achieved.

  When the bird's-eye view image around the vehicle is clearly overwritten by the open door outline, the continuity of the original bird's-eye view image may be lost in the overwritten portion, making it difficult to see. In consideration of such inconvenience, it is preferable that the outer contour portion of the transparent door overhead image is also made transparent that is weaker than the transparency of the region other than the outer contour portion. That is, by synthesizing the outer contour portion weakly and transparently, the composite harmony between the overhead image and the transparent door overhead image is ensured.

  Also, in another preferred embodiment of the present invention, the transparency in the transparent door overhead image is configured to change at predetermined time intervals. According to this configuration, the entire door shown in the open state is displayed on the monitor while changing the transparency. Therefore, the display form in which the overall shape and shape of the door are clearly shown or the door is substantially invisible and the door is not visible. Since the display form in which the periphery of the vehicle that is not obstructed is displayed is repeated, the driver can confirm both the state where the released door occupies the space and the state around the vehicle during a predetermined time.

It is a schematic diagram explaining the basic concept of this invention which synthesize | combines the door image of an open state with the bird's-eye view image produced | generated from the periphery picked-up image. It is a functional block diagram which shows a part of driving electronic system of the vehicle carrying the driving assistance apparatus by this invention. It is a functional block diagram of the image processing module which comprises a driving assistance device. It is the schematic diagram which illustrated the process of synthesize | combining a door image with a bird's-eye view image. It is a flowchart which shows a bird's-eye view image display routine. It is a monitor screen figure which shows the transfer from the normal display overhead image to the display overhead image which added the door of the open state.

  Before describing a specific embodiment of the present invention, its basic concept will be described with reference to the schematic diagram of FIG. Cameras are mounted on the front, rear, left and right 1a to 1d in order to photograph the entire periphery of the vehicle. In order to monitor and display a bird's-eye view image as a vehicle periphery monitoring screen, first, a captured image of the surrounding area of the vehicle is acquired by the camera (# 1). If four cameras 1a to 1d are arranged on the front, rear, left, and right of the vehicle so as to cover the entire periphery of the vehicle, and an overhead view image is generated from these cameras based on the captured image, the overhead image is centered on the own vehicle. It becomes an all-around bird's-eye view image, allowing you to glance at the entire surroundings of your vehicle. However, the present invention is not limited to the generation of such an all-around overhead image, and a specific direction overhead image generated based on a captured image in a specific direction is also an object of the invention.

  When captured images from the respective cameras 1a to 1d are acquired, here, a mapping table in which projection transformation with a projection plane parallel to the road surface, that is, a viewpoint transformation in which a virtual viewpoint is set directly above is set. (# 2). By combining the image segment for the overhead view image obtained from each captured image through this projective transformation process (mapping) so that the overlapping areas overlap each other, an all-around photographed overhead view image of the periphery of the vehicle viewed from directly above the vehicle ( (Hereinafter simply referred to as a photographic overhead view image) is obtained (# 3). In the illustrated bird's-eye view image, a border line that divides the photographed bird's-eye view image into a front image region, left and right side image regions, and a rear image region is drawn. Further, in FIG. 1, in order to make the explanation easy to understand, a mapping table is prepared for each captured image, but a single mapping table corresponding to the generated captured overhead image is used. It is common. This mapping table describes which pixel of which captured image is applied to each pixel constituting the captured bird's-eye view image in consideration of a camera installation error or the like.

  Since the vehicle overhead view image representing the own vehicle (hereinafter simply referred to as a vehicle body image) is incorporated into the photographed overhead view image, the vehicle body image to be incorporated is read from the vehicle body image storage unit (# 4). The body image storage unit may be a simple ROM, but when a plurality of vehicle body images are stored, a storage unit having a structure that is extracted based on a predetermined extraction condition is used.

In addition, a door overhead image, which is an overhead image of an open state of a door (for example, a rear door) protruding outward from the vehicle body in the opened state, is incorporated at a corresponding position in the vehicle overhead image as necessary. It is stored in the door image storage unit in advance. This door bird's-eye view image is not a bird's-eye view image of the door that is simply opened, but is a transparent door bird's-eye view image in which at least the region other than the outer contour portion is made transparent. However, this transparent door overhead image will be simply referred to as a door overhead image unless otherwise specified. In other words, even if this door overhead image is incorporated so that it protrudes from the shooting overhead image showing the periphery of the vehicle, the shooting overhead image can be seen through in the inner part other than the outline, so the shooting overhead image that is covered by the door overhead image The part is slight. A door overhead image in which regions other than the outer contour portion are generated with various degrees of transparency may be created in advance and stored in the door image storage unit, or the door overhead image in a form in which the transparency can be adjusted is a door image. You may store in a storage part. Further, not only a fully opened door, but also a door overhead image of a partially opened door or a door overhead image from a different viewpoint may be prepared and selectively used. Moreover, you may prepare the door overhead image created so that an outer side outline part also has a certain amount of transparency.
In any case, when it is requested to add a door overhead image, an appropriate door overhead image is read from the door image storage unit (# 5).

  Next, the read vehicle overhead view image and door overhead image are combined with the captured overhead image generated from the captured image (# 6). Alternatively, the photographing overhead image and the door overhead image are combined with the template on which the vehicle body overhead image is arranged. In any case, an image in which a vehicle overhead view image with or without a door overhead image is placed at the center of the shooting overhead image is transferred to the monitor as a monitor display image, so that the transparent door overhead view is displayed on the monitor screen. A shooting overhead image is displayed together with the vehicle body overhead image to which the image is attached (# 7).

Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 2 is a functional block diagram showing a part of a driving electronic system of a vehicle equipped with a driving support device according to the present invention.
A vehicle equipped with this driving support apparatus is equipped with four in-vehicle cameras 1, that is, a front camera 1a, a back camera 1d, a left side camera 1b, and a right side camera 1c. An all-around bird's-eye view image is created from these captured images from the in-vehicle camera 1. In the following description, these in-vehicle cameras 1a, 1b, 1c, and 1d may be simply referred to as the camera 1 as appropriate. When the vehicle periphery monitoring control is operated, an image captured by the camera 1 or an overhead image generated using the captured image is displayed on the monitor. The camera 1 is a digital camera that is obtained by photographing a vehicle periphery in time series, digitally converts a photographed image, and outputs it in real time. The camera 1 includes a wide-angle lens or a fisheye lens.

  The processing module 23 of the driving electronic system is connected to a vehicle state detection sensor group that detects a driving operation and a vehicle running state, and outputs a signal input from the vehicle state detection sensor group as it is or after evaluation. Although not shown, the vehicle state detection sensor group includes a steering sensor that measures the steering operation direction (steering direction) and the operation amount (steering amount), a shift position sensor that determines the shift position of the shift lever, and an accelerator pedal. An accelerator sensor that measures the operation amount, a brake sensor that detects the operation amount of the brake pedal, a distance sensor that detects the travel distance of the host vehicle, and the like are included.

  As shown in FIG. 2, the driving support image processing unit 20 that forms the core of the driving support device includes various functional units constructed in the form of hardware and / or software. Related functional units include an image processing module 50, a communication interface 70, a display module 71, an audio processing module 72, and the like. The communication interface 70 is used as an input / output interface for the in-vehicle LAN, and is connected to a control unit such as the processing module 23, the touch panel 21T, the power steering unit PS, the transmission mechanism T, and the brake device BK so as to be able to transmit data. . In addition, a monitor 21 is mounted as a visual information output device, and a speaker 22 is mounted as an audio information output device. The monitor display image generated by the image processing module 50 is converted into a video signal by the display control unit 71 and sent to the monitor 21. The voice guide generated by the voice processing module 72, an emergency warning sound, and the like are played by the speaker 22.

  FIG. 3 is a functional block diagram of the image processing module 50 of the driving support image processing unit 20. The image processing module 50 has a function of generating an image such as an overhead image converted by projective transformation from a captured image acquired by the camera 1 that captures the periphery of the vehicle.

  The image processing module 50 includes a captured image memory 51, a preprocessing unit 52, an image generation unit 53, an image composition unit 54, and a frame memory 56. The captured image acquired by the camera 1 is developed in the captured image memory 51, and the preprocessing unit 52 adjusts the luminance balance, color balance, and the like between the captured images individually acquired by the camera 1.

The image generation unit 53 includes a normal image generation unit 60, an overhead image generation unit 61, a mapping table 62, and a vehicle image output unit 64.
The normal image generation unit 60 adjusts the captured image to an image quality suitable for monitor display as a vehicle peripheral image as it is. The vehicle peripheral image displayed on the monitor may be one selected by the driver from images captured by the front camera 1a, the left and right side cameras 1b and 1c, and the back camera 1d, or a combination of a plurality of captured images.

  The bird's-eye view image generation unit 61 generates a bird's-eye view image by projectively transforming a shot image from a specific camera 1 or a shot image from all the cameras 1 developed in the shot image memory 51 from the upper virtual viewpoint. It has a function. Specifically, an overhead image is generated from the captured image by mapping using the mapping table 62. In the mapping table 62, various maps (mapping data group) for projective transformation used here are stored in advance so as to be selectable or calculated as necessary. Each map stored in or calculated in the mapping table 62 can be constructed in various forms. Here, pixel data of a captured image for each camera and pixels of a projective transformation image (usually a bird's-eye view image). It is constructed as a map that describes the correspondence with data. The overhead image generation unit 61 also has a function of selecting an appropriate map from the mapping table 62 based on an external command or an internal command.

  The own vehicle image output unit 64 includes a vehicle body image output unit 64a, a door image storage unit 64b, a door image output unit 64c, and an own vehicle image generation unit 64d. The vehicle body image output unit 64a outputs a vehicle body overhead image indicating the vehicle body from the upper viewpoint to the host vehicle image generation unit 64d. This vehicle overhead view image may be a photograph or an illustration. Also, one type may be used, or a plurality of types may be prepared and selected. When a photographic overhead view image from a specific photographed camera 1 image is also adopted as the combined photographic overhead view image, a vehicle body image that matches the photographic overhead view image, for example, an image of only the rear part of the vehicle body, etc. It is convenient to prepare.

  The door image storage unit 64b stores a door overhead image of the open state of the door equipped on the vehicle as viewed from above. This door bird's-eye view image is a transparent door bird's-eye view image in which the region other than the outer contour portion is made transparent or semi-transparent. The degree of transparency can be arbitrarily set, but a transparent door overhead image created with a plurality of different degrees of transparency may be stored and freely selectable. Further, the transparent door overhead image is stored in the door image storage unit 64b as an image file structure in which the transparency degree of the region other than the outer contour portion can be arbitrarily set, and the transparency specified by the door image output unit 64c is specified. You may make it produce | generate and output the transparent door overhead image transparentized by the degree. Although illustrated in FIG. 4, not only simply changing the degree of transparency, but also a transparent door overhead image with a transparent gradation that increases in transparency from the outside to the center of the inside may be generated. However, some degree of transparency may be applied. In any case, the door image output unit 64c outputs the transparent door overhead image read from the door image storage unit 64b to the own vehicle image generation unit 64d after adjusting the image orientation and image size, if necessary. To do.

  As illustrated in FIG. 4, the host vehicle image generation unit 64d converts the transparent door overhead image output from the door image output unit 64c into a corresponding door portion in the vehicle body overhead image output from the vehicle body image output unit 64a. By compositing, a self-viewing image of the host vehicle with the opened door added is generated. The own vehicle overhead view image generated by the own vehicle image generation unit 64d is combined with the photographic overhead view image generated by the overhead view image generation unit 61 by the image composition unit 54. The synthesized bird's-eye view image is inserted into a predetermined display template and transferred to the frame memory 56 as a monitor display image for vehicle driving assistance. The monitor display image transferred to the frame memory 56 is displayed on the monitor 21 via the display control unit 71.

Next, the flow of the driving support overhead image display by the driving support apparatus configured as described above will be described with reference to the flowchart of FIG.
When the bird's-eye view image display routine for the purpose of vehicle periphery monitoring starts, first, the display type of the bird's-eye view image is set manually or automatically according to the driver's request (# 01). Here, the display type of the overhead image defines the captured image and the virtual viewpoint position used when generating the overhead image around the vehicle, the layout of the generated overhead image on the monitor screen, and the like. Mapping table setting (map setting) for projective transformation used in the overhead image generation unit 61 is performed according to the display type of the read overhead image (# 02). Here, if all-around bird's-eye view image display is selected, all captured images acquired from the four in-vehicle cameras 1 are processed (# 03). An all-around photographic overhead view image around the vehicle is generated using each captured image using the set mapping table (# 04). A vehicle body overhead image arranged at the center of the all-around shooting overhead image is read (# 05).

  Here, it is checked whether or not the door image in the open state, that is, the transparent door overhead image is required to be added to the vehicle body overhead image (# 06). The addition of the transparent door overhead image may be determined by a manual operation requested by the driver, may be automatically determined on the apparatus side, or both of them may be employed. As for automatic determination on the device side, it is decided to add a transparent overhead view image for the rear door when moving backward, or the obstacle side door based on obstacle detection information from an obstacle detection module (not shown) It is preferable to determine the addition of the transparent door overhead image for.

  When a door image is requested (# 06 Yes branch), a suitable transparent door overhead image is generated and output by the door image generation unit 64c (# 07). Further, the vehicle image in which the transparent door overhead image is added to the vehicle body overhead image is generated and output by the vehicle image generation unit 64d (# 08). When the door image is not requested (# 06 No branch), the process of adding the transparent door overhead image is omitted, and the vehicle image generation unit 64d outputs the vehicle body overhead image as it is as the vehicle image.

  Next, a display template used for the set driving support bird's-eye view image display is extracted (# 09), and an image composition process for fitting the photographed bird's-eye view image and the vehicle image into the display template is performed by the image composition unit 54. (# 10). A monitor display image that is a bird's-eye view image for driving support generated by this image composition processing is transferred to the frame memory 56 (# 20) and displayed on the monitor 21 (# 21).

  Unless there is an instruction to end this display routine (# 22 No branch), the routine returns to step # 01 again, and this routine is repeated. In this routine, even when a door image is requested, the transparency of the transparent door overhead image may be changed at predetermined time intervals. For example, as an extreme example, a non-transparent door image (transparency 0%) and a transparent door image may be alternately displayed at predetermined time intervals.

[Another embodiment]
(1) In the description of the above-described embodiment, an image obtained by projective transformation from a captured image using a vehicle upper viewpoint and a projection plane parallel to the road surface is used as the captured overhead image. It is not limited to the photographic overhead view image obtained by such projective transformation. For example, you may use the bird's-eye view image using a concave curved surface and a bending surface as a projection surface.
(2) In the above-described embodiment, the functions of the image generation unit 53 and the image synthesis unit 54 built in the image processing module 50 are blocked for easy understanding of the description. The form of conversion can be arbitrarily changed.

  The present invention can be used for all driving support technologies for monitoring the periphery of a vehicle using a photographed bird's-eye view image.

1: Camera 21: Monitor 50: Image processing module 53: Image generation unit 54: Image composition unit 60: Normal image generation unit 61: Overhead image generation unit 62: Mapping table 64: Own vehicle image output unit 64a: body image output unit 64b: door image storage unit 64c: door image output unit 64d: own vehicle image generation unit

Claims (4)

A driving support device for a vehicle having a door that opens to the outside of the vehicle body,
A bird's-eye view image generation unit that generates a shooting bird's-eye view image by projectively transforming a photographed image acquired by an in-vehicle camera that images a peripheral region of the vehicle from an upper virtual viewpoint;
A vehicle body image output unit for outputting a vehicle body overhead image showing the vehicle body;
A door image output unit for outputting a transparent door overhead image in which a region other than the outer contour portion in the door overhead image showing the door in an open state is made transparent or translucent;
An image compositing unit that generates, as a monitor display image, a driving assistance image in which the shooting overhead image, the vehicle body overhead image, and the transparent door overhead image are combined;
A driving assistance device comprising:   The driving support device according to claim 1, wherein the transparent door overhead image is provided with a transparent gradation in which transparency increases from the outside to the center of the interior.   The driving assistance device according to claim 1 or 2, wherein the outer contour portion of the transparent door overhead image is also made transparent less than the transparency of the region other than the outer contour portion.   The driving assistance device according to any one of claims 1 to 3, wherein the transparency of the transparent door overhead image changes at predetermined time intervals.

Images