JP2020086884A - Lane marking estimation device, display control device, method and computer program - Google Patents

Abstract

To provide drive assistance useful for a driver by estimating a position of a lane marking of a cruising lane even in such an environment that it is hard for the driver to visually recognize the lane marking.SOLUTION: A processor acquires information on a position of a real lane marking 320 which is based on at least an image captured by a camera 401 provided in a self-vehicle and information on a position of a stationary object 330 which is provided along a cruising lane of the self-vehicle and is based on detection by one or more sensors 403, 405 provided in the self-vehicle and having a different detection method from that of the camera 401, calculates an interval D between the real lane marking 320 and the stationary object 330 on the basis of the information on the position of the real lane marking 320 and the information on the position of the stationary object 330, estimates a position of a virtual lane marking EL on the basis of the calculated interval D and the position of the stationary object detected by the sensors 403, 405 and then displays a line image 200 along the virtual lane marking EL.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a marking line estimation device, a display control device, a method, and a computer program.

In Patent Document 1, a white line (compartment line) drawn on a road surface is detected by a camera, and an interval (road width) between the left and right white lines of a vehicle driving lane is sequentially updated, and only one white line is detected by a camera. If it is not possible, using the stored road width data, a partition line recognition device that estimates the position of the other white line based on the one white line detected by the camera is described. There is.

JP, 2003-44836, A

However, in Patent Document 1, it is assumed that either one of the left and right white lines can be detected, but it is difficult to detect both the left and right white lines in bad weather such as rain, fog, and snow. Is assumed.

A summary of the specific embodiments disclosed herein is provided below. It should be understood that these aspects are presented only to provide an overview of the reader to these particular embodiments and are not intended to limit the scope of this disclosure. Indeed, the present disclosure may include various aspects not described below.

The outline of the present disclosure relates to estimating the position of a lane marking even in an environment where the lane marking of a driving lane is difficult to see, and providing useful driving assistance to a driver. More specifically, it also relates to accurately recognizing the position of the lane markings in the traveling lane.

Therefore, the display control device described in the present specification provides information regarding the position of the real lane markings based on at least the image captured by the camera provided in the own vehicle, and a detection method different from that of the camera provided in the own vehicle. The information about the position of the stationary target provided along the traveling lane of the own vehicle based on the detection by the one or more sensors having the information, and the information about the position of the real marking line and the position of the stationary target. The distance between the real marking line and the stationary target is calculated from the information, the position of the virtual marking line is estimated from the calculated distance and the information about the position of the stationary target, and the image along the virtual marking line is estimated. Is displayed. In some embodiments, information regarding positions of a plurality of stationary targets provided along a traveling lane of the vehicle based on detection by one or more sensors is acquired, and reliability of the plurality of stationary targets is acquired. May be calculated, and the position of the virtual lane marking may be estimated from the stationary target having high reliability and the interval.

It is a figure which shows the application example of the display system for vehicles which concerns on some embodiment. FIG. 3 is a block diagram of a vehicular display system according to some embodiments. It is a figure explaining the space|interval of the target and real lane markings which a lane marking estimating device calculates concerning some embodiments. It is a figure explaining the position of the virtual lane marking which a lane marking estimating device estimates concerning some embodiments. It is a figure which shows the foreground which a driver|operator visually recognizes when facing a front from a driver|operator's seat, and the line image which the vehicle display system visually recognizes by overlapping with this foreground according to some embodiments. It is a figure which shows the foreground which a driver|operator visually recognizes when facing a front from a driver|operator's seat, and the line image which the vehicle display system visually recognizes by overlapping with this foreground according to some embodiments. It is a figure which shows the foreground which a driver|operator visually recognizes when facing a front from a driver|operator's seat, and the line image which the vehicle display system visually recognizes by overlapping with this foreground according to some embodiments. It is a flowchart of the process which accumulates the space|interval of the target and an actual lane marking which a lane marking estimating device performs according to some embodiments. It is a flow figure showing processing of display control of a line picture which a display control device performs concerning some embodiments.

Embodiments according to the present invention will be described below with reference to the drawings. The present invention is not limited to the following embodiments (including the contents of the drawings). Of course, changes (including deletion of components) can be added to the following embodiments. Further, in the following description, in order to facilitate understanding of the present invention, description of known technical matters will be appropriately omitted.

In the following, FIGS. 1 and 2 provide a description of the configuration of an exemplary vehicle display system. An exemplary control processing flow will be described with reference to FIGS. 3, 4, 6, and 7. 5A, 5B, 5C provide example displays.

Please refer to FIG. The image display unit 11 that displays the line image 200 in the vehicular display system 10 is a head-up display (HUD: Head-Up Display) device provided in the dashboard 5 of the vehicle 1. The HUD device emits the display light 11a toward the front windshield 2 (which is an example of a member to be projected) and displays a line image 200 in the virtual display area 100, so that the display light 11a is transmitted through the front windshield 2. The line image 200 is visually recognized by superimposing it on the foreground 300 which is the real space visually recognized.

Further, the image display unit 11 may be a head mounted display (hereinafter, HMD) device. The driver 4 mounts the HMD device on his/her head and sits on the seat of the host vehicle 1 to superimpose the displayed line image 200 on the foreground 300 through the front windshield 2 of the host vehicle 1. To see. The display area 100 in which the vehicle display system 10 displays a predetermined line image 200 is fixed at a specific position with the coordinate system of the host vehicle 1 as a reference, and when the driver 4 faces in that direction, the specific position. The line image 200 displayed in the display area 100 fixed to can be visually recognized.

The image display unit 11 is an obstacle (pedestrian, bicycle, motorcycle) existing in the foreground 300 which is a real space visually recognized through the front windshield 2 of the vehicle 1 under the control of the display control device 13. , Other vehicles), road surfaces, road signs (lane markings), buildings, etc., near real objects (an example of the positional relationship between images and real objects), positions that overlap the real objects (positions between images and real objects) By displaying the line image 200 at a position (an example of the positional relationship between the image and the real object) set with reference to the real object, thereby making a visual augmented reality (AR). It can also be formed.

FIG. 2 is a block diagram of a vehicular display system 10 according to some embodiments. The vehicle display system 10 includes an image display unit 11 and a display control device 13 that controls the image display unit 11. The display controller 13 includes one or more I/O interfaces 14, one or more processors 16, one or more storage units 18, and one or more image processing circuits 20. .. The various functional blocks depicted in FIG. 2 may be implemented in hardware, software, or a combination of both. FIG. 2 is only one example of an embodiment, and the illustrated components may be combined into fewer components or there may be additional components. For example, image processing circuitry 20 (eg, a graphics processing unit) may be included in one or more processors 16. Further, in the following description, the display control device 13 of the present embodiment also includes a function as a marking line estimation device, but the marking line estimation device may be configured independently of the display control device 13.

As illustrated, the processor 16 and the image processing circuit 20 are operably connected to the storage unit 18. More specifically, the processor 16 and the image processing circuit 20 execute the program stored in the storage unit 18 to operate the vehicle display system 10, such as generating or/and transmitting image data. It can be carried out. The processor 16 or/and the image processing circuit 20 may be at least one general purpose microprocessor (eg, central processing unit (CPU)), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA). , Or any combination thereof. The storage unit 18 includes any type of magnetic medium such as a hard disk, any type of optical medium such as CD and DVD, any type of semiconductor memory such as volatile memory, and non-volatile memory. Volatile memory may include DRAM and SRAM, and non-volatile memory may include ROM and NVROM.

As shown, the processor 16 is operably coupled to the I/O interface 14. For example, the I/O interface 14 configures the vehicle display system 10 as a personal area network (PAN) such as a Bluetooth (registered trademark) network or a local area network (LAN) such as an 802.11x Wi-Fi (registered trademark) network. A wireless communication interface for connecting to a wide area network (WAN) such as a 4G or LTE cellular network can be included. The I/O interface 14 may also include a wired communication interface such as, for example, a USB port, a serial port, a parallel port, an OBDII, and/or any other suitable wired communication port.

As shown in the figure, the processor 16 is interoperably connected to the I/O interface 14 so as to be informed of various other electronic devices connected to the vehicle display system 10 (I/O interface 14). Can be given and received. The I/O interface 14 includes, for example, a camera 401, a front sensor 403 (first sensor), a side sensor 405 (second sensor), an eye position detection unit 407, a road information database 409 provided in the vehicle 1. Also, the vehicle position detection unit 411 and the like are operably connected. The image display unit 11 is operably connected to the processor 16 and the image processing circuit 20. Therefore, the image displayed by the image display unit 11 may be based on the image data received from the processor 16 and/or the image processing circuit 20. The processor 16 and the image processing circuit 20 control the image displayed by the image display unit 11 based on the information obtained from the I/O interface 14. The I/O interface 14 may include a function of processing (converting, calculating, analyzing) information received from another electronic device or the like connected to the vehicle display system 10.

The host vehicle 1 includes one or more cameras 401 that capture at least the front of the host vehicle 1. In addition to this, the camera 401 may include a camera that images the side and/or the rear side of the vehicle 1. The camera 401 is a monocular or compound-eye CCD camera, an infrared camera, or another imaging device attached to one or several locations of the vehicle 1, and may transmit the captured image to the processor 16. The processor 16 may have a function of analyzing the acquired captured image by a known pattern matching method or the like to extract a specific real object (real lane marking of the driving lane), but the camera 401 has an analysis function. May have. In this case, the camera 401 may send the analysis result of the captured image (real object related information) to the processor 16.

It should be noted that the camera 401 can send the real object related information to the processor 16 together with its reliability. For example, the camera 401 is in an environment (such as bad weather such as rain, fog, or snow) in which the detection accuracy decreases, or the camera 401 is covered with mud or the detection accuracy is detected only in a part of the detection area. If the lane markings (an example of real objects) of the traveling lane that is the detection target are partially worn due to aging, the camera 401 determines that part or all of the real object related information is present. The reliability may be output as a low value, and the reliability of the real object related information may be output as a high value when the lane markings of the traveling lane can be clearly confirmed.

The host vehicle 1 includes one or more front sensors 403 (first sensor) that perform distance measurement to a target existing in front of the host vehicle 1. The front sensor 403 is a radar device such as a millimeter wave radar, a laser radar, an ultrasonic radar, or a distance measuring device such as a sonar, which is attached to one or several places of the vehicle 1. The front sensor 403 may transmit the reception signal of the front sensor 403 to the processor 16. The processor 16 may have a function of analyzing the acquired reception signal of the front sensor 403 and detecting the presence or absence of a target, the position of the target, the distance to the target, and the type of the target. The front sensor 403 may have an analysis function. In this case, the front sensor 403 may transmit the analysis result (target information) of the reception signal of the front sensor 403 to the processor 16.

In addition, the host vehicle 1 may include one or more side sensors 405 (second sensor) that perform distance measurement to a target existing on the side of the host vehicle 1. The configuration of the side sensor 405 is substantially the same as that of the front sensor 403, and the description thereof will be omitted. The second sensor may measure the distance to the target existing behind the vehicle 1.

The host vehicle 1 may include an eye position detection unit 407 including an infrared camera that detects the position of the eyes of the driver 4. The processor 16 can specify the eye position of the driver 4 by acquiring an image captured by the infrared camera (an example of information that can estimate the eye position) and analyzing the captured image. The processor 16 may acquire the information on the position of the eyes of the driver 4 identified from the image captured by the infrared camera from the I/O interface 14. The method for acquiring the position of the eyes of the driver 4 of the vehicle 1 or the information capable of estimating the position of the eyes of the driver 4 is not limited to these, and known eye position detection (estimation) It may be acquired using a technology. The processor 16 adjusts at least the position of the line image 200 based on the position of the eyes of the driver 4 so that the line image 200 superimposed on a desired position of the foreground 300 is detected by the viewer (driver). You may make it visible to 4).

The host vehicle 1 may include a road information database 409 including a navigation system or the like. The road information database 409 includes road information (running lanes, lane markings, stop lines) on which the vehicle 1 travels, which is an example of actual object-related information, based on the position of the vehicle 1 acquired from a vehicle position detection unit 411 described later. Lines, pedestrian crossings, road width, number of driving lanes, intersections, curves, branches, traffic restrictions, etc.), presence/absence of feature information (buildings, bridges, rivers, etc.), position (including distance to own vehicle 1) , Direction, shape, type, detailed information, etc. can be read and transmitted to the processor 16. In particular, in the present embodiment, the processor 16 determines whether or not there is a stationary target existing beside the traveling lane, the shape of the stationary target, the type of the stationary target, the position of the stationary target, and each section from the stationary target to the traveling lane. The distance to the line or/and the distance between the respective division lines may be acquired from the road information database 409.

The processor 16 may acquire the above-described information acquired from the road information database 409 from a portable information terminal (not shown) connected to the I/O interface 14 or a communication connection device outside the vehicle. The mobile information terminal is, for example, a smartphone, a laptop computer, a smart watch, or another information device that can be carried by the driver 4 (or another occupant of the vehicle 1). The outside-vehicle communication connection device is, for example, another vehicle connected to the own vehicle 1 by vehicle-to-vehicle communication (V2V: Vehicle To Vehicle), or a pedestrian (pedestrian) connected by vehicle-to-vehicle communication (V2P: Vehicle To Pedestal). Mobile communication terminal), and network communication equipment connected by road-to-vehicle communication (V2I: vehicle to load infrastructure), and in a broad sense, all network devices connected by communication (V2X: vehicle to everything). Including things.

The host vehicle 1 may include a host vehicle position detection unit 411 including a GNSS (Global Navigation Satellite System) or the like. The road information database 409, the portable information terminal, and/or the communication device outside the vehicle obtains the position information of the own vehicle 1 from the own vehicle position detection unit 411 continuously, intermittently, or for each predetermined event, Information around the host vehicle 1 can be selected and generated and transmitted to the processor 16.

The software components stored in the storage unit 18 include the target detection module 502, the actual lane marking detection module 504, the road shape estimation module 506, the target-division line interval calculation module 508, the reference target determination module 510, and the actual division. A line detection accuracy determination module 512, a marking line position estimation module 514, an eye position detection module 516, an image determination module 518, and a graphics module 520 are included.

The target detection module 502 acquires a plurality of reflection points (an example of detection points) detected by the front sensor 403 and the side sensor 405, and sets the acquired plurality of reflection points as the same stationary target based on a predetermined condition. Group. For example, the target detection module 502 can group a step, a wall, and a pole that are intermittently present on the side of the driving lane as the same stationary target. When it is determined as a stationary target, the position of the stationary target is partially estimated by linearly or polynomial interpolating other reflective points at the part where there are no reflective points. Good. The target detection module 502 acquires the presence/absence of a stationary target existing on the side of the traveling lane, the shape of the stationary target, the type of the stationary target, and the position of the stationary target from the road information database 409, and the same. It may be used as a reference when grouping as a stationary target.

In addition, the target detection module 502 calculates the reliability of each grouped stationary target from the data of a plurality of reflection points (an example of detection points) detected by the front sensor 403 and the side sensor 405. Good. For example, the target detection module 502 includes a stationary target with a low detection signal level at a detection point, a stationary target with a low signal-to-noise ratio (S/N ratio), a stationary target of a low reliability type, or / And a static target having low consistency with the static target acquired from the road information database 409 may be determined to have low reliability (the method of calculating the reliability of the position of the static target is limited to these). Not).

The real lane marking detection module 504 detects the position of the real lane marking by analyzing the image captured by the camera 401 by pattern matching or the like. The real marking line detection module 504 may detect the position of the real marking line by analyzing the detection points detected by the front sensor 403 and the side sensor 405, or the real marking line detected by the camera 401. The position of the real marking line may be detected by fusing the front marking sensor 403 and the real marking line detected by the side sensor 405.

The road shape estimation module 506 estimates the road shape based on the real lane markings detected by the real lane marking detection module 504. The road shape estimation module 506 may estimate the road shape based on the position (shape) of a stationary target continuously or intermittently beside the traveling lane detected by the target detection module 502. The estimation may be performed in combination with the position (shape) of the actual marking line detected by the marking line detection module 504.

The target-partition line interval calculation module 508 continuously calculates the interval between the stationary target and the actual lane markings, and stores the interval data. Specifically, the target-lane line spacing calculation module 508 refers to the road shape estimated by the road shape estimation module 506, and determines the predetermined point of the stationary target detected by the target detection module 502 and the actual point. An interval D (see D11 to D13 and D21 to D23 in FIG. 3) between a predetermined point of the actual lane marking detected by the lane marking detection module 504 is calculated, and the data of the distance D is accumulated in the storage unit 18. You may. The data of the distance D includes the distance between the actual marking line on the left side of the host vehicle 1 and the target, and the distance between the marking line on the right side and the target. Further, when there are a plurality of stationary targets that are detected continuously or intermittently, the target-partition line interval calculation module 508 calculates the interval with the marking line for each target that is detected continuously or intermittently. Then, the data of the interval D is accumulated.

FIG. 3 is a diagram for explaining the distance D between the target and the actual marking line, which is accumulated by the target-marking line distance calculation module 508. The traveling lanes illustrated in FIG. 3 are the traveling lane 311 of the host vehicle 1 and the traveling lane 312 on the right of the traveling lane 311. The actual lane markings illustrated in FIG. 3 are the actual lane markings 321 on the left side of the traveling lane 311 of the host vehicle 1 and the actual lane markings on the right side of the driving lane 311 of the subject vehicle 1 (on the left of the adjacent traveling lane 312 on the right). 322 and an actual partition line 323 on the right side of the traveling lane 312 on the right side. The stationary targets shown in FIG. 3 are a stationary target 331 existing outside the left side of the traveling lane 311 of the host vehicle 1 and a stationary target 332 existing outside of the right side of the traveling lane 312 on the right side. ,. The target-partition line interval calculation module 508 calculates a distance D11 between the stationary target 331 and the closest real partition line 321 and a distance D12 between the stationary target 331 and the second closest real partition line 322, and a third closest real partition line 323. The distance D13 and the distance D23 from the stationary target 332 to the closest real marking line 323, the distance D22 to the second closest real marking line 322, and the distance D21 to the third closest real marking line 321 are continuous or intermittent. Calculated and stored (accumulated) in the storage unit 18.

Referring back to FIG. The reference target determination module 510 determines a stationary target capable of accurately estimating the marking line. The reference target determination module 510 includes distances D11 and D21 between the stationary targets 331 and 332 and the left lane marking 321 and distances D12 and D22 between the stationary targets 331 and 332 and the right lane marking 322, respectively. And parameters (for example, standard deviation) of the respective normal distributions of the intervals D13 and D23 between the stationary targets 331 and 332 and the actual lane markings 323 of the adjacent traveling lanes are calculated as the reliability of the data of the intervals D. , A stationary target with high reliability of the distances D11 and D21 from the left real lane marking 321 and a stationary target with high reliability of distances D12 and D22 from the right lane marking 322, and/or of adjacent traveling lanes. A stationary target having a high reliability of the distances D13 and D23 from the real marking line 323 is determined. Note that the reference target determination module 510 replaces the reliability of the distance D between the stationary target and the marking line, or in addition to this, with the detected position of each stationary target calculated by the target detection module 502. The reliability may be used to determine a highly reliable stationary target. The method of determining a stationary target that can accurately estimate the marking line is not limited to these.

The real lane marking detection accuracy determination module 512 determines whether the environment is such that the accuracy of detecting the real lane markings decreases. In the camera 401 or the front sensor 403, there is a part where the marking line cannot be detected, the contrast of the captured image (reception signal) is reduced, the reception signal is attenuated, the detection noise is increased (S/N ratio is reduced), or In addition, when a decrease in reliability at the time of analysis is detected, the actual lane marking detection accuracy determination module 512 may determine that the environment is one in which the lane marking detection accuracy decreases (the determination method is not limited to these. ). In addition, the real lane marking detection accuracy determination module 512 determines bad weather such as rain, fog, and snow by acquiring weather information of a position where the vehicle travels from a portable information terminal (not shown), an external communication connection device, and the like, It may be determined whether or not the environment is one in which the accuracy of lane marking detection is reduced.

The marking line position estimation module 514 estimates the marking line position based on the stationary target. Specifically, the lane marking position estimation module 514 uses the stationary target determined by the reference target determination module 510 as a reference, based on the average value of the distance D calculated by the target-lane marking distance calculation module 508, Estimate the position of the line.

FIG. 4 is a diagram illustrating the position of the virtual lane markings estimated by the lane marking position estimation module 514. The traveling lanes illustrated in FIG. 4 are the traveling lane 311 of the host vehicle 1 and the traveling lane 312 on the right of the traveling lane 311. In addition, the actual lane markings shown in FIG. 4 are the actual lane markings 321 on the left side of the traveling lane 311 of the host vehicle 1 and the actual lanes on the right side of the traveling lane 311 of the subject vehicle 1 (on the left of the adjacent traveling lane 312 on the right). A line 322 and an actual lane marking 323 on the right side of the traveling lane 312 on the right side. In FIG. 4, the accuracy of detection of the actual lane markings of the traveling lane is lowered, or the actual lane markings of the traveling lane cannot be detected, and the estimated lane markings (virtual lane markings) are indicated by dotted lines. The partition line position estimation module 514 uses the virtual partition line EL1 ahead (far) of the real partition line 321 on the left side, the virtual partition line EL2 ahead (far) of the real partition line 322 on the right side, and the traveling lane 312 on the right side. The virtual lane marking EL3 ahead (far) of the real lane marking 323 on the right side of is estimated.

For example, the reference target determination module 510 determines the distance between the left stationary target 331 and the actual lane marking 321 accumulated in the storage unit 18 when determining the virtual lane marking EL1 on the left side of the traveling lane 311 of the vehicle 1. The standard deviation of D11 and the standard deviation of the distance D21 between the stationary object 332 on the right side and the actual marking line 321 accumulated in the storage unit 18 are compared to determine a stationary object with higher reliability (here. Then, it is assumed that the stationary target 331 has high reliability). Next, based on the stationary target 331 detected by the target detection module 502, based on the average value E11 of the distance D11 between the stationary target 331 and the real marking line 321 accumulated in the storage unit 18, the virtual marking line EL1 To estimate. Similarly, when the reference target determination module 510 determines the virtual lane marking EL2 on the right side of the traveling lane 311 of the host vehicle 1, the reference target 331 and the real lane marking 322 on the left side accumulated in the storage unit 18 are stored. The standard deviation of the distance D12 and the standard deviation of the distance D22 between the stationary object 332 on the right side and the actual lane marking 322 accumulated in the storage unit 18 are compared, and a stationary object having higher reliability is determined ( Here, the reliability of the stationary target 332 is assumed to be high). Next, based on the stationary target 332 detected by the target detection module 502, the virtual marking line EL2 is calculated based on the average value E22 of the distance D22 between the stationary target 332 and the real marking line 322 accumulated in the storage unit 18. To estimate. In addition, when the reliability of the stationary target is determined to be low without satisfying the predetermined condition stored in the storage unit 18, the distance Da between the actual lane markings that can be continuously detected in the vicinity of the vehicle 1 is set. The other virtual lane marking may be estimated from one virtual lane marking estimated with reference to the stationary target.

The eye position detection module 516 detects the position of the eyes of the driver 4 of the vehicle 1. The eye position detection module 516 determines where the eye height of the driver 4 is in the height region provided in a plurality of stages, detects the eye height of the driver 4 (position in the Y-axis direction), Detection of the eye height (position in the Y-axis direction) and depth position (position in the Z-axis direction) of the driver 4, detection of eye position (position in the X, Y, Z-axis directions) of the driver 4 , And various software components for performing various operations related to. The eye position detection module 516 can acquire the eye position of the driver 4 from the eye position detection unit 407, or can estimate the eye position including the eye height of the driver 4 from the eye position detection unit 407, for example. Information is received and the eye position including the eye height of the driver 4 is estimated. The information capable of estimating the position of the eyes includes, for example, the position of the driver's seat of the vehicle 1, the position of the driver's 4 face, the height of the sitting height, the input value by the driver 4 on the operation unit (not shown), and the like. Good.

The image determination module 518 determines the type, position, and size of the line image 200 visually recognized by the driver 4 along the virtual lane marking that is the estimated lane marking. The image determination module 518 can determine the type of the line image 200 based on the type of the actual lane marking 320, the notification necessity of the line image 200, and/or the degree of deterioration of the lane marking detection accuracy. Further, the image determination module 518 arranges the position of the line image 200 along the virtual lane marking EL according to the position of the virtual lane marking EL estimated by the lane marking position estimation module 514. The image determination module 518 may set the position of the line image 200 to a position overlapping with the virtual lane marking EL, or a position offset from the virtual lane marking EL. Further, the image determination module 518 determines the size by generating the line image 200 by the perspective method. That is, the line image 200 overlapping the distant foreground 300 is adjusted to be small, and the line image 200 overlapping the nearby foreground 300 is adjusted to be large.

The graphics module 520 may include visual effects (eg, brightness, transparency, saturation, contrast, or other visual characteristic), size, display position, display distance (from driver 4 to line image 200) of the displayed line image 200. Various known software components for changing the distance to. The graphic module 520 includes the type and coordinates set by the image determination module 518 (the left-right direction (X-axis direction) when the driver 4 views the display area 100 from the driver's seat of the vehicle 1 and the vertical direction (Y). The line image 200 is displayed so as to be visually recognized by the driver 4 in the image size (including at least the axial direction). The graphic module 520 follows the foreground 300 so that the relative position of the line image 200 with respect to the foreground 300 does not change when the foreground 300 seen by the driver 4 approaches as the vehicle 1 advances. The position of may be adjusted continuously, or the image size may be adjusted to gradually increase.

Further, when the graphic module 520 can determine that the line image 200 should not be displayed (or should not be displayed) based on the information acquired from the I/O interface 14, the visibility of the line image 200 is reduced. You may allow it (including not displaying it). Specifically, the graphic module 520 may reduce the visibility of the line image 200 if the speed acquired from the ECU of the host vehicle 1 (not shown) is equal to or higher than a predetermined threshold. When the line image 200 is displayed even in an environment with poor visibility, it is assumed that the speed may excessively increase with peace of mind, but when the speed of the host vehicle 1 increases, the visibility of the line image 200 may be increased. It is also possible to suppress such an excessive speed by reducing Further, the graphic module 520 may reduce the visibility of the line image 200 when it is determined that the vehicle 1 is near the intersection based on the information acquired from the road information database 409. In the vicinity of the intersection, it is assumed that there will be no target along the driving lane, and it will be impossible to estimate the virtual lane markings with sufficient reliability. However, when the own vehicle 1 detects that the vehicle approaches the intersection. By reducing the visibility of the line image 200, it is possible to make it difficult to visually recognize the line image 200 along the virtual partition line with low reliability.

5A, 5B, and 5C are diagrams showing a foreground 300 and a line image 200 overlapping the foreground 300 that are visually recognized when the driver 4 faces forward from the driver's seat of the vehicle 1. The processor 16 and the image processing circuit 20 are, as shown in FIG. 5A, a line image 201 along the virtual lane marking line EL1 on the left side of the driving lane 311 of the host vehicle 1 and a line image 202 along the virtual lane marking line EL2 on the right side. And may be displayed on the image display unit 11. Further, as shown in FIG. 5B, when the reliability of the estimated virtual lane markings EL is low, the processor 16 and the image processing circuit 20 show a line image along one of the left and right virtual lane markings EL with high reliability. Only 201 may be displayed. Further, as shown in FIG. 5C, the processor 16 and the image processing circuit 20 may display the line image 203 near the center of the traveling lane 311 of the host vehicle 1 along the virtual lane markings EL.

FIG. 6 is a flow diagram of a process of accumulating the distance between the target and the real marking line according to some embodiments. This processing is repeatedly executed from the start of the own vehicle 1 to the end of the traveling. However, in order to reduce the processing load, the accuracy of detection of the actual lane markings is required to display the line image 200. Is not required, or the line image 200 is not required to be displayed, but this process may be performed after the accuracy of detecting the actual marking line is reduced.

First, the processor 16 executes the target detection module 502 to acquire at least the position of the target existing continuously or intermittently beside the traveling lane from the front sensor 403 and/or the side sensor 405, and obtains the data. The data is stored in the storage unit 18 (step S11), the real lane marking detection module 504 is executed, at least the positions of the real lane markings on the left and right of the traveling lane are acquired from the camera 401, and the data is stored in the storage unit 18 (step S12). ), the target-partition line interval calculation module 508 is executed, the interval between the stationary target and the actual lane marking is calculated, and the data is accumulated in the storage unit 18.

FIG. 7 is a flowchart showing a process of display control of the line image 200 according to some embodiments. This process is repeatedly executed from the time the host vehicle 1 starts running to the end of running.

The processor 16 executes the real lane marking detection accuracy determination module 512 to determine whether the environment is such that the detection accuracy of part or all of the real lane markings decreases (step S21), and the real lane markings If the accuracy of all detections has not deteriorated (No in step S21), step S21 is repeated. If the accuracy of detection of part or all of the real marking lines has decreased (Yes in step S21), step S22. Move to.

Next, the processor 16 determines whether the host vehicle 1 is less than the threshold value (step S22), and determines whether the host vehicle 1 is near the intersection (step S23), and the host vehicle 1 is equal to or more than the threshold value (step S23). If No in step S22) or if the vehicle 1 is near the intersection (No in step S23), the graphic module 520 is executed to reduce the visibility of the line image 200.

When the host vehicle 1 is less than the threshold value (Yes in step S22) and the host vehicle 1 is near the intersection (Yes in step S23), the processor 16 executes the reference target determination module 510 to execute a plurality of stationary objects. A stationary target that can accurately estimate the virtual marking line is determined from the targets (step S24), the position of the virtual marking line is estimated from the determined stationary target, and the line image 200 is arranged along the virtual marking line. Is displayed (step S25).

The operations of the processing processes described above can be performed by executing one or more functional modules of an information processing device such as a general-purpose processor or an application-specific chip. All of these modules, a combination of these modules, and/or a combination with general hardware capable of substituting their functions are included in the scope of protection of the present invention.

The functional blocks of the vehicular display system 10 are optionally implemented by hardware, software, or a combination of hardware and software to implement the principles of the various described embodiments. The functional blocks described in FIG. 2 may optionally be combined or one functional block may be separated into two or more sub-blocks to implement the principles of the described embodiments. As will be appreciated by those skilled in the art. Thus, the description herein optionally supports any possible combination or division of the functional blocks described herein.

As described above, in some embodiments, the one or more processors 16 are imaged by the camera 401 provided in the vehicle 1 from the one or more I/O interfaces 14. Information regarding the position of the real lane marking 320 based on at least the image and the traveling lane of the host vehicle 1 based on the detection by one or more sensors 403 and 405 having a detection method different from that of the camera 401 provided in the host vehicle 1. The information about the position of the stationary target 330 provided along the information is acquired, and the information about the position of the real marking line 320 and the information about the position of the stationary target 330 are acquired. The distance D is calculated, the position of the virtual lane marking EL is estimated from the calculated distance D and the information about the position of the stationary target 330, and the image display 11 displays the line image 200 along the estimated virtual lane marking EL. To do. As a result, even in an environment where the actual marking line is difficult to detect with the camera, the virtual marking line based on the stationary target can be accurately estimated.

Further, in some embodiments, information regarding the positions of the plurality of stationary targets 330 provided along the traveling lane of the vehicle 1 based on the detection by the one or more sensors 403 and 405 is acquired, and The reliability of the stationary target 330 may be calculated, and the position of the virtual lane marking EL may be estimated from the stationary target 330 having high reliability and the calculated interval D. The reliability of the target can be calculated from the detection results of the one or more sensors 403 and 405, and the virtual marking line EL can be accurately estimated based on the highly reliable stationary target.

Also, in some embodiments, one or more processors 16 may determine the position of the virtual lane marking EL closer to the highly reliable stationary target 330 to be spaced from the more reliable stationary target 330. Alternatively, the image display 11 may display only the line image 200 along the virtual lane marking EL that is estimated from D and that is closer to the highly reliable stationary target 330. By estimating a virtual marking line close to the highly reliable stationary target from the stationary target determined to have high reliability, the error is small and the virtual marking line can be estimated.

Further, in some embodiments, the one or more processors 16 are configured so that the real lane markings on the left and right of the traveling lane of the host vehicle 1 are based on at least the image captured by the camera 401 provided in the host vehicle 1. The information about the travel lane width Da1 between 321 and 322 is acquired, and the position of the virtual lane marking EL closer to the highly reliable stationary target 330 is calculated as the calculated highly reliable stationary target 330 and the calculated distance D. And the other position of the virtual lane marking EL is estimated from the position of the virtual lane marking EL closer to the highly reliable stationary target 330 and the traveling lane width Da1. The image display 11 may display the line images 200 along the virtual lane markings EL1 and EL2. According to this, it is possible to accurately estimate the positions of the plurality of virtual lane markings from one highly reliable stationary target.

In some embodiments, the first sensor 403 that detects the stationary target 330 in front of the host vehicle 1 and the second sensor 405 that detects the stationary target 330 on the side or the rear side of the host vehicle 1. And the camera 401 images the side or rear side of the host vehicle 1, and the one or more processors 16 include information regarding the position of the real marking line 320 and the second sensor 405. The distance D is calculated from the acquired information about the position of the stationary target 330, and the position of the virtual lane marking EL is calculated from the calculated interval D and the information about the position of the stationary target 330 acquired from the first sensor 403. It may be estimated. According to this, the position of the target detected by the second sensor 405 on the side or the rear side of the host vehicle 1 and the actual position detected by the camera 401 in the vicinity (side or rear side) of the host vehicle 1. Based on the position of the marking line 320, the distance D between the stationary target 330 and the actual marking line 320 is calculated, and the calculated distance D and the stationary target detected by the first sensor 403 in front of the vehicle 1. The virtual lane marking EL is estimated at the position of 330. That is, if the vehicle is on the side or the rear side of the host vehicle 1, the actual lane marking 320 exists near the host vehicle 1, so that even in an environment where it is difficult for the camera 401 to detect the actual lane marking in front of the vehicle 1. , It is possible to continue to acquire the distance between the real marking line and the stationary target.

Also, in some embodiments, if one or more of the processors 16 determines that the information about the position of the real marking line 320 is partially unobtainable or that the position information is partially inaccurate, The image display 11 may display a line image 200 along at least the virtual demarcation line EL of the determined region. That is, it is possible to display the line image 200 on the part of the real marking line that is difficult to visually recognize, and prevent the line image 200 from being excessively displayed on the part where the real marking line is visible.

Further, in some embodiments, the one or more processors 16 obtain information regarding the speed of the host vehicle 1 from the one or more I/O interfaces 14, and the speed of the host vehicle 1 is predetermined. The image display 11 may stop displaying the estimated line image 200 along the virtual lane markings EL1, EL2 when it is determined to be faster than the threshold value.

Further, in some embodiments, the one or more processors 16 obtain information from the one or more I/O interfaces 14 that it can estimate whether the vehicle 1 is near an intersection, When it is determined that the vehicle 1 is near the intersection, the image display 11 may stop displaying the estimated line image 200 along the virtual lane markings EL1, EL2.

1 Own vehicle, 2... Front windshield, 4... Driver, 5... Dashboard, 10... Vehicle display system, 11... Image display, 11... Image display part, 11a... Display light, 13... Display control device, 14 ... I/O interface, 16... Processor, 18... Storage section, 20... Image processing circuit, 100... Display area, 200 (201-203)... Line image, 300... Foreground, 311... Travel lane, 312... Travel lane, 320 (321 to 323)... Real marking line, 330 (331, 332))... Stationary target, 401... Camera, 403... Front sensor (first sensor), 405... Side sensor (second sensor), 407... Eye position detection unit, 409... Road information database, 411... Own vehicle position detection unit, 413... Portable information terminal, 420... External communication connection device, 502... Target detection module, 504... Real marking line detection module, 506... Road Shape estimation module, 508... marking line interval calculation module, 510... reference target determination module, 512... actual marking line detection accuracy determination module, 514... marking line position estimation module, 516... eye position detection module, 518... image determination module 520... Graphic module, Da1... Running lane width, E11... Average value, E22... Average value, EL (EL1 to EL3)... Virtual partition line

Claims (20)

In the marking line estimation device for estimating the position of the actual marking line (320) that divides the traveling lane of the host vehicle (1),
One or more I/O interfaces (14),
One or more processors (16),
A memory (18),
One or more computer programs stored in the memory (18) and configured to be executed by the one or more processors (16),
Said one or more processors (16)
From the one or more I/O interfaces (14),
Information on the position of the real marking line (320) based at least on the image taken by the camera (401) provided in the vehicle (1);
Provided along the traveling lane of the own vehicle (1) based on detection by one or more sensors (403, 405) having a detection method different from that of the camera (401) provided in the own vehicle (1) And the information about the position of the stationary target (330) to be acquired,
An interval (D) between the real marking line (320) and the stationary target (330) is calculated from information about the position of the real marking line (320) and information about the position of the stationary target (330). Then
Executing a command to estimate the position of the virtual lane markings (EL1, EL2) from the calculated distance (D) and information about the position of the stationary target (330),
Marking line estimation device. Said one or more processors (16)
From the one or more I/O interfaces (14),
Acquiring information on the positions of a plurality of stationary targets (330) provided along the traveling lane of the vehicle (1) based on the detection by the one or more sensors (403, 405),
Calculating the reliability of the plurality of stationary targets (330),
Executing a command to estimate the position of the virtual lane markings (EL1, EL2) from the stationary target (330) with high reliability and the distance (D).
The marking line estimation device according to claim 1. Said one or more processors (16)
From the one or more I/O interfaces (14),
A traveling lane width between the left and right real marking lines (321, 322) of the traveling lane of the vehicle (1) based on at least an image captured by a camera (401) provided in the vehicle (1). Get information about (Da1),
The position of the virtual marking line (EL1, EL2) closer to the highly reliable stationary target (330) is defined as the highly reliable stationary target (330) and the interval (D). Estimated from
The other position of the virtual lane markings (EL1, EL2) is set to the position of the virtual lane markings (EL1, EL2) closer to the highly reliable stationary target (330), and the travel lane width ( Execute the instruction, inferred from Da1),
The marking line estimation device according to claim 2. The one or more sensors (403, 405) are
A first sensor (403) for detecting the stationary target (330) in front of the host vehicle (1);
A second sensor (405) for detecting the stationary target (330) on the side or the rear side of the host vehicle (1),
Said one or more processors (16)
The distance (D) is calculated from information about the position of the real marking line (320) and information about the position of the stationary target (330) acquired from the second sensor (405),
A command for estimating the position of the virtual lane markings (EL1, EL2) from the calculated distance (D) and information about the position of the stationary target (330) acquired from the first sensor (403) is issued. Run,
The marking line estimation device according to claim 1. The position of the real lane markings (320) of the traveling lane of the host vehicle (1) is estimated, and a line image (200) along the virtual lane markings (EL1, EL2) that is the estimated real lane markings (320) is displayed. In a display control device for controlling the image display unit (11),
One or more I/O interfaces (14),
One or more processors (16),
A memory (18),
One or more computer programs stored in the memory (18) and configured to be executed by the one or more processors (16),
Said one or more processors (16)
From the one or more I/O interfaces (14),
Information about the position of the real marking line (320) based at least on the image captured by the camera (401) provided in the vehicle (1);
Provided along the traveling lane of the own vehicle (1) based on detection by one or more sensors (403, 405) having a detection method different from that of the camera (401) provided in the own vehicle (1) Information about the position of the stationary target (330) to be acquired,
An interval (D) between the real marking line (320) and the stationary target (330) is calculated from information about the position of the real marking line (320) and information about the position of the stationary target (330). Then
Estimating the position of the virtual lane markings (EL1, EL2) from the calculated distance (D) and information about the position of the stationary target (330),
A command is executed so that the image display (11) displays the line image (200) along the estimated virtual lane markings (EL1, EL2),
Display controller. Said one or more processors (16)
From the one or more I/O interfaces (14),
Acquiring information on the positions of a plurality of stationary targets (330) provided along the traveling lane of the vehicle (1) based on the detection by the one or more sensors (403, 405),
Calculating the reliability of the plurality of stationary targets (330),
Executing a command to estimate the position of the virtual lane markings (EL1, EL2) from the stationary target (330) with high reliability and the distance (D).
The display control device according to claim 5. Said one or more processors (16)
The position of the virtual marking line (EL1, EL2) closer to the stationary target (330) with high reliability is the stationary target (330) with high reliability, and the interval (D), Estimated from
The image display (11) displays only the estimated line image (200) along the virtual lane marking (EL1, EL2) closer to the stationary target (330) with high reliability, Execute instructions,
The display control device according to claim 6. Said one or more processors (16)
From the one or more I/O interfaces (14),
A traveling lane width between the left and right real marking lines (321, 322) of the traveling lane of the vehicle (1) based on at least an image captured by a camera (401) provided in the vehicle (1). Get information about (Da1),
The position of the virtual marking line (EL1, EL2) closer to the highly reliable stationary target (330) is defined as the highly reliable stationary target (330) and the interval (D). Estimated from
The other position of the virtual lane markings (EL1, EL2) is set to the position of the virtual lane markings (EL1, EL2) closer to the highly reliable stationary target (330), and the travel lane width ( Da1) and
A command is executed so that the image display (11) displays the estimated line image (200) along each of the left and right virtual lane markings (EL1, EL2).
The display control device according to claim 6. The one or more sensors (403, 405) are
A first sensor (403) for detecting the stationary target (330) in front of the host vehicle (1);
A second sensor (405) for detecting the stationary target (330) on the side or the rear side of the host vehicle (1),
The camera (401) images the side or rear side of the vehicle (1),
Said one or more processors (16)
The distance (D) is calculated from information about the position of the real marking line (320) and information about the position of the stationary target (330) acquired from the second sensor (405),
A command for estimating the position of the virtual lane markings (EL1, EL2) from the calculated distance (D) and information about the position of the stationary target (330) acquired from the first sensor (403) is issued. Run,
The display control device according to claim 5. Said one or more processors (16)
When it is determined that the information regarding the position of the real lane markings (320) cannot be partially acquired or the accuracy of the information regarding the position is partially low, at least the virtual lane markings (EL1, EL2) in the determined area are displayed. Execute instructions such that the image display (11) displays the line image (200) along
The display control device according to claim 5. Said one or more processors (16)
From the one or more I/O interfaces (14),
Obtaining information about the speed of the own vehicle (1),
When it is determined that the speed of the host vehicle (1) is faster than a predetermined threshold,
Executing a command so that the image display (11) stops displaying the estimated line image (200) along the virtual lane markings (EL1, EL2),
The display control device according to claim 5. Said one or more processors (16)
From the one or more I/O interfaces (14),
Acquiring information that can estimate whether the vehicle (1) is near an intersection,
When it is determined that the host vehicle (1) is near the intersection,
Executing a command so that the image display (11) stops displaying the estimated line image (200) along the virtual lane markings (EL1, EL2),
The display control device according to claim 5. The position of the real lane markings (320) of the traveling lane of the host vehicle (1) is estimated, and a line image (200) along the virtual lane markings (EL1, EL2) that is the estimated real lane markings (320) is displayed. In the method of controlling the image display unit (11),
Acquiring information about the position of the real marking line (320) based at least on the image captured by the camera (401) provided in the vehicle (1);
Provided along the traveling lane of the own vehicle (1) based on detection by one or more sensors (403, 405) having a detection method different from that of the camera (401) provided in the own vehicle (1) Obtaining information about the position of the stationary target (330) to be captured,
An interval (D) between the real marking line (320) and the stationary target (330) is calculated from information about the position of the real marking line (320) and information about the position of the stationary target (330). What to do
Estimating the position of the virtual lane markings (EL1, EL2) from the calculated distance (D) and information about the position of the stationary target (330);
Displaying the line image (200) along the estimated virtual lane markings (EL1, EL2) on the image display (11).
Method. Acquiring information about the positions of a plurality of stationary targets (330) provided along the traveling lane of the vehicle (1) based on detection by the one or more sensors (403, 405);
Calculating the reliability of the plurality of stationary targets (330);
Estimating the position of the virtual lane markings (EL1, EL2) from the stationary target (330) with high reliability and the distance (D).
The method of claim 13. The position of the virtual marking line (EL1, EL2) closer to the stationary object (330) with high reliability is calculated from the stationary object (330) with high reliability and the interval (D). Estimating,
The image display (11) displays only the estimated line image (200) along the virtual lane markings (EL1, EL2) closer to the stationary target (330) with high reliability. Including,
The method according to claim 14. A traveling lane width between the left and right real marking lines (321, 322) of the traveling lane of the vehicle (1) based on at least an image captured by a camera (401) provided in the vehicle (1). Acquiring information about (Da1),
The position of the virtual marking line (EL1, EL2) closer to the highly reliable stationary target (330) is defined as the highly reliable stationary target (330) and the interval (D). From the
The other position of the virtual lane markings (EL1, EL2) is set to the position of the virtual lane markings (EL1, EL2) closer to the highly reliable stationary target (330), and the travel lane width ( Estimating from Da1) and
The image display (11) displaying the estimated line image (200) along each of the left and right virtual lane markings (EL1, EL2).
The method according to claim 14. The one or more sensors (403, 405) are
A second sensor (405) for detecting the stationary target (330) on the side or the rear side of the host vehicle (1);
A first sensor (403) for detecting the stationary target (330) in front of the host vehicle (1),
The camera (401) images the side or rear side of the vehicle (1),
Calculating the interval (D) from information about the position of the real marking line (320) and information about the position of the stationary target (330) acquired from the first sensor (403);
Estimating the position of the virtual lane markings (EL1, EL2) from the calculated distance (D) and information about the position of the stationary target (330) acquired from the first sensor (403); including,
The method of claim 13. When it is determined that the information regarding the position of the real lane markings (320) cannot be partially acquired or the accuracy of the information regarding the position is partially low, at least the virtual lane markings (EL1, EL2) in the determined area are displayed. Displaying the line image (200) along the image display (11).
The method of claim 13. Obtaining information about the speed of the vehicle (1),
When it is determined that the speed of the host vehicle (1) is faster than a predetermined threshold,
Stopping the display of the line image (200) along the estimated virtual lane markings (EL1, EL2) by the image display (11).
The method of claim 13. A computer program comprising instructions for performing the method according to any of claims 13-19.

Images