CN110962746B - Driving assisting method, system and medium based on sight line detection - Google Patents
Driving assisting method, system and medium based on sight line detection Download PDFInfo
- Publication number
- CN110962746B CN110962746B CN201911275693.4A CN201911275693A CN110962746B CN 110962746 B CN110962746 B CN 110962746B CN 201911275693 A CN201911275693 A CN 201911275693A CN 110962746 B CN110962746 B CN 110962746B
- Authority
- CN
- China
- Prior art keywords
- driver
- sight line
- driving
- line information
- area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
- B60R2300/802—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for monitoring and displaying vehicle exterior blind spot views
Abstract
The invention provides a driving assistance method, a system and a medium based on sight line detection, wherein the driving assistance method based on sight line detection comprises the following steps: acquiring sight line information of a driver; the sight line information comprises fixation points of the driver and fixation time of each fixation point; determining the driving state of the driver according to the sight line information; and selecting corresponding driving information to display according to the driving state of the driver. The auxiliary driving method based on the sight line detection can predetermine the driving state of the driver according to the sight line information of the driver and display the corresponding driving information, and is beneficial to improving the driving safety.
Description
Technical Field
The invention belongs to the field of vehicle auxiliary devices, relates to an auxiliary driving method, and particularly relates to an auxiliary driving method, an auxiliary driving system and a driving medium based on sight line detection.
Background
With the development of technology and the progress of society, the number of automobiles kept in China is increased year by year, and the automobiles become indispensable transportation tools for many families. The convenience is brought to resident's trip, and the traffic accident also leads to the emergence of traffic accident more and more frequently to the automobile quantity that constantly increases, and wherein, many traffic accidents all take place in the car blind area outside the driver's field of vision. The automobile blind area is an area which cannot be directly observed because the sight line of a driver is shielded by an automobile body when the driver is positioned at a normal driving position.
Referring to fig. 1, the blind areas of the car mainly include: the left column A blind area 11 is generally about 17 degrees; the dead zone 12 of the column A on the right side is about 8 degrees generally; a dead zone 13 of the right B column is about 64 degrees generally; the right tail dead zone 14 is generally about 5 degrees; the left tail blind area 15 is generally about 5 degrees; the left B-pillar blind area 16 is generally about 42 degrees. These blind areas are caused by the structure of the vehicle and the driving position of the driver and are difficult to eliminate in practice. The influence of different blind areas is different when the driver is in different driving states, for example, when the driver turns left, the driver needs to pay more attention to the left A column blind area 11, the left B column blind area 16 and the left vehicle tail blind area 15; more attention is required to be paid to the left side dead zone 15 and the right side dead zone 14 when backing. The prior art is difficult to provide corresponding blind area information for a driver according to different driving states of the driver.
Disclosure of Invention
In view of the above-mentioned shortcomings of the prior art, it is an object of the present invention to provide a driving assistance method, system and medium based on gaze detection, which can solve the problem that it is difficult to provide corresponding blind area information to a driver according to different driving states in the prior art.
To achieve the above and other related objects, the present invention provides a driving assistance method based on sight line detection, including: acquiring sight line information of a driver; the sight line information comprises fixation points of a driver and fixation time of each fixation point; determining the driving state of the driver according to the sight line information; and selecting corresponding driving information to display according to the driving state of the driver.
In an embodiment of the present invention, an implementation method for determining a driving state of a driver according to the sight line information includes: dividing a driver's observation area into a front area, a left area, a right area, and a central rearview mirror area; and determining the driving state of the driver according to the area to which the fixation point belongs and the fixation time of the fixation point of each area, wherein the area to which the fixation point belongs is contained in the sight line information.
In an embodiment of the present invention, an implementation method for determining a driving state of a driver according to an area to which a gaze point belongs and gaze time of a gaze point in each area, which are included in the gaze information, includes: if the sight line information contains a plurality of fixation points of the left area within a period of time, and the fixation time of the fixation points of the left area is greater than a first threshold value, determining that the driver is in a left-turning or left lane-changing state; if the sight line information contains a plurality of fixation points of the right area within a period of time, and the fixation time of the fixation points of the right area is greater than a second threshold value, determining that the driver is in a right turning or right lane changing state; and if the sight line information contains the fixation points of a plurality of central rearview mirror areas within a period of time and the fixation time of the fixation points on the central rearview mirrors is greater than a third threshold value, determining that the driver is in a reversing state.
In an embodiment of the present invention, an implementation method for determining a driving state of a driver according to an area to which a gaze point belongs and gaze time of a gaze point in each area, which are included in the gaze information, includes: if the watching time of the watching point of the front area contained in the sight line information is larger than a fourth threshold value within a period of time, determining that the driver is in a forward driving state; and if the watching time of the front area watching point contained in the sight line information in a period of time is less than a fifth threshold, determining that the driver is in a dangerous driving state.
In an embodiment of the present invention, an implementation method for determining a driving status of a driver according to the sight line information includes: selecting a matched driving state from a database according to the sight line information; the database stores historical sight line information of the driver and a driving state matched with the historical sight line information.
In an embodiment of the present invention, an implementation method for determining a driving state of a driver according to the sight line information includes: the sight line information is used as the input of a machine learning model, and the output of the machine learning model is the driving state of the driver; the machine learning model is trained by using the historical sight line information of the driver and the driving state matched with the historical sight line information.
In an embodiment of the present invention, an implementation method for selecting a corresponding image or video to display according to a driving state of a driver includes: selecting a corresponding blind area image or video to display according to the driving state of the driver; the blind area image or the video is acquired by a camera arranged on the outer side of the automobile.
In an embodiment of the present invention, the driving assistance method based on gaze detection further includes: the sight line information of the driver is tracked and the content observed by the driver is recorded.
The present invention also provides a computer-readable storage medium having stored thereon a computer program; the computer program, when invoked, executes the gaze detection based driver assistance method of the present invention.
The invention also provides a driving assistance system based on sight line detection, which comprises: the sight line acquisition equipment is used for acquiring sight line information of a driver; the sight line information comprises the fixation points of the driver and the stay time of the sight line at each fixation point; the processor is arranged on a center console or a cloud end of the automobile, is in communication connection with the sight line acquisition equipment, and is used for determining the driving state of the driver according to the sight line information; and the display is in communication connection with the processor and is used for selecting the corresponding image or video to display according to the driving state of the driver.
As described above, the driving assistance method, system, and medium based on gaze detection according to the present invention have the following advantageous effects: the auxiliary driving method based on the sight line detection can determine the driving state of the driver according to the sight line information of the driver, and select and display corresponding images or videos according to different driving states of the driver, so that the driving safety is improved, and traffic accidents are reduced.
Drawings
Fig. 1 is a schematic view illustrating a blind area of an automobile according to an embodiment of the driving assistance method based on gaze detection.
Fig. 2 is a flowchart illustrating a driving assistance method based on gaze detection according to an embodiment of the present invention.
Fig. 3 is a flowchart illustrating the driving assistance method based on gaze detection according to step S21 in an embodiment of the invention.
Fig. 4 is a view illustrating a point of regard of the driving assistance method based on gaze detection according to an embodiment of the invention.
Fig. 5 is a block diagram of a driving assistance system based on visual line detection according to an embodiment of the present invention.
Fig. 6 is a structural diagram of a driving assistance method based on gaze detection according to an embodiment of the invention.
Description of the element reference
11 left A column blind zone
12 right side A column blind area
13 right B column blind area
14 right tail blind zone
15 left side rear dead zone
16 left B column blind area
41 left rear-view mirror
42 right rear view mirror
43 center rearview mirror
44 front window
45 sight line acquisition equipment
46 display
47 Right A column
48 left A column
49 rotatable camera
S21-S23
S211 to S217
Detailed Description
The following embodiments of the present invention are provided by way of specific examples, and other advantages and effects of the present invention will be readily apparent to those skilled in the art from the disclosure herein. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the components related to the present invention are only shown in the drawings rather than drawn according to the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.
The automobile A column is a connecting column for connecting the roof and the front cabin with the front left and the front right, the automobile B column is a column between the front window and the rear window, the existence of the automobile A column and the automobile B column enables the bearing type automobile body structure to have higher stability and automobile body rigidity, the cabin is protected to reduce deformation when the automobile collides, and the automobile A column and the automobile B column play an important role in the safety of drivers and passengers, so that the A column and the B column belong to an essential part in the automobile structure. However, the presence of the a-pillar and the B-pillar may cause the driver to have corresponding a-pillar and B-pillar blind areas. In addition, since the driver sits in the front row of the vehicle and the rear view thereof is mainly from the rear view mirror, there are left and right rear dead zones in places where it is difficult to see through the rear view mirror.
The existence of blind areas brings great potential safety hazards to the driving behaviors of drivers, and the blind areas are difficult to avoid by changing the structure of the automobile or adjusting the driving posture. In some embodiments, the influence of the blind area on the driving behavior is reduced by collecting the blind area information through the camera and presenting the blind area information to the user. However, the driver's demand for blind spot information is also different in different driving states, such as: when turning left or changing lanes to the left, a driver needs to pay more attention to a left column A blind area, a left column B blind area and a left vehicle tail blind area; when turning right or changing lanes to the right, a driver needs to pay more attention to a right A column blind area, a right B column blind area and a right tail blind area; when backing a car, a driver needs to pay more attention to the left tail dead zone and the right tail dead zone. The prior art is difficult to provide the most required blind area information for the driver according to different driving states of the driver, so that the driver cannot obtain the required information at the first time. In view of the above problem, the present invention provides a driving assistance method based on sight line detection, including: acquiring sight line information of a driver; the sight line information comprises fixation points of a driver and fixation time of each fixation point; determining the driving state of the driver according to the sight line information; the auxiliary driving method based on the sight line detection can determine the driving state of the driver according to the sight line information of the driver and provide the driver with the most required image or video according to the driving state, so that the driver can obtain corresponding blind area information quickly, and the driving safety is improved.
Referring to fig. 2, in an embodiment of the present invention, the driving assistance method based on gaze detection includes:
s21, obtaining sight line information of the driver; the sight line information includes a gaze point of the driver and gaze time of each gaze point. And the fixation time of the fixation point is the time of the driver sight line staying at the fixation point.
And S22, determining the driving state of the driver according to the sight line information. The driving state of the driver includes: forward driving, left turning, right turning, reversing, left lane changing, right lane changing, left overtaking, dangerous driving, etc.
And S23, selecting and displaying corresponding driving information according to the driving state of the driver. The driving information includes: navigation information, blind area images or video information, automobile driving condition information, alarm information and the like.
With the development of technology, many automobile interiors are provided with display screens such as: the display screen of the vehicle-mounted computer, the display screen of the center console, the display screen of the vehicle-mounted terminal and the like. The technical solution of this embodiment can be implemented by any in-vehicle display device that can display an image or a video according to the driving state of the driver, and the specific display device is not limited in this embodiment.
Referring to fig. 3, in the present embodiment, the sight line collecting device may be an eye tracker. The method for acquiring the sight line information of the driver by using the sight line acquisition equipment 45 comprises the following steps:
and S211, acquiring pupil coordinates of the driver by using the sight line acquisition equipment.
S212, the coordinates of the Purkinje spot on the outer surface of the cornea of the driver are collected by the sight line collecting equipment. Purkinje's spot is a bright spot on the cornea of the eye produced by reflection of light entering the pupil from the outer surface of the cornea.
And S213, acquiring the coordinates of the current fixation point of the driver.
And S214, repeating the steps S211 to S213 when the gazing point is changed by the driver, and obtaining a plurality of groups of data of pupil coordinates, Purkinje coordinates and gazing point coordinates.
And S214, fitting by using the multiple groups of data of the pupil coordinate, the Purkinje coordinate and the fixation point coordinate, and establishing a mapping relation between the fixation point coordinate and the pupil coordinate and the Purkinje coordinate.
S215, processing the obtained pupil coordinate and the Purkinje coordinate by using the mapping relation, and obtaining the current fixation point of the driver.
S216, timing the current fixation point of the driver, and acquiring fixation time of the driver on the current fixation point.
In an embodiment of the present invention, an implementation method for determining a driving state of a driver according to the sight line information includes: dividing a driver's observation area into a front area, a left area, a right area, and a central mirror area; and determining the driving state of the driver according to the area to which the fixation point belongs and the fixation time of the fixation point of each area, wherein the area to which the fixation point belongs is contained in the sight line information.
Referring to fig. 4, in the present embodiment, the observation area of the driver is divided into eight areas, i.e., an area (i) to an area (i). Wherein, the region I corresponds to the right front, the region II corresponds to the left front, the region II corresponds to the right front, and the region I, the region II and the region II form a front region; the area (c) corresponds to a left vehicle window, the area (c) corresponds to a left rearview mirror, and the area (c) form a left area; the region (c) corresponds to a right side vehicle window, the region (b) corresponds to a right rearview mirror, and the region (c) and the region (b) form a right side region; the area (v) corresponds to the area of the right rearview mirror.
In an embodiment of the present invention, an implementation method for determining a driving state of a driver according to the sight line information includes: and if the sight line information contains a plurality of fixation points of the left area within a period of time and the fixation time of the fixation points of the left area is greater than a first threshold value, determining that the driver is in a left-turning or left lane-changing state.
In fig. 4, if the line-of-sight information in the period of time includes a plurality of gaze points in the region (c) and the region (c), and the sum of the gaze times of the gaze points in the region (c) and the region (c) is greater than the first threshold, it may be determined that the driver is in a left turn state or a left lane change state. The period of time may be set according to the driving habits of the driver, for example: 10 seconds, 15 seconds, or 30 seconds. The first threshold is, for example, 10%, 15%, or 20% of the period of time. When the driver is in a left turning or lane changing state, images or videos of a left column A blind area, a left column B blind area and/or a left vehicle tail blind area are selected to be displayed, and the images or videos can be acquired by cameras arranged at corresponding positions of a vehicle body.
In fig. 4, if the line of sight information in the period of time includes a plurality of regions (c) and gaze points in the region (b), and the sum of gaze times of the gaze points in the regions (c) and (d) is greater than a second threshold, it is determined that the driver is in a right turn state or a right lane change state, and the period of time may be set according to driving habits of the driver, for example: 10 seconds, 15 seconds, or 30 seconds. The second threshold is, for example, 10%, 15%, or 20% of the period of time. When the driver is determined to be in a right turning state, images or videos of a right A column blind area, a right B column blind area and/or a right vehicle tail blind area are selected to be displayed, and the images or videos can be acquired by cameras arranged at corresponding positions of a vehicle body.
In this embodiment, it may also be determined whether the driver is ready to overtake by combining the gazing duration of the injection point in the region ii and the gazing duration of the injection point in the region on the left. And when the situation that the driver is ready to overtake is determined, selecting images or videos of the left A column blind area, the left B column blind area and/or the left vehicle tail blind area for displaying.
In this embodiment, whether the driver needs to turn or change lanes can be judged in advance according to the sight line information of the driver, and the information of the corresponding blind area is displayed to the driver in time, so that traffic accidents caused by the existence of the blind area when the driver turns or changes lanes are avoided.
In an embodiment of the present invention, an implementation method for determining a driving state of a driver according to the sight line information includes: and if the sight line information contains the gazing points on the plurality of central rearview mirrors within a period of time and the gazing time of the gazing points on the central rearview mirrors is greater than a third threshold value, determining that the driver is in a reversing state.
Referring to fig. 4, in this embodiment, if the sight line information collected within the period of time includes a plurality of gaze points of the region(s), and the gaze time of the gaze point of the region(s) is greater than a third threshold, it is determined that the driver is in the reverse state; the value of the third threshold depends on the driving habits of the driver, for example 10%, 15% or 20% of the period of time. When the driver is determined to be in the reversing state, images or videos of the left vehicle tail dead zone or the right vehicle tail dead zone are selected to be displayed, and the images or videos can be acquired by a camera arranged at the corresponding position of a vehicle body.
In this embodiment, whether the driver needs to back a car or not can be judged in advance according to the sight line information of the driver, and the information of the corresponding blind area is displayed to the driver in time, so that traffic accidents caused by the existence of the blind area when the driver backs a car are avoided.
In an embodiment of the present invention, an implementation method for determining a driving state of a driver according to the sight line information includes: and if the watching time of the watching point of the front area contained in the sight line information in a period of time is greater than a fourth threshold value, determining that the driver is in a forward driving state. And if the watching time of the front area watching point contained in the sight line information in a period of time is less than a fifth threshold, determining that the driver is in a dangerous driving state.
Referring to fig. 4, if the gaze point included in the sight line information of the driver is mostly in the regions of (i) and (ii), the value of the fourth threshold depends on the driving habit of the driver, for example, 70%, 80%, or 85% of the period of time. At this time, corresponding blind area information does not need to be displayed, and in order to facilitate driving, information such as navigation and automobile driving conditions can be selected to be displayed, wherein the automobile driving conditions comprise information such as vehicle speed, oil quantity/electric quantity and temperature in the automobile.
In an embodiment of the present invention, referring to fig. 4, if the sum of the gaze times of the gaze points of the region i, the region ii, and the region |, in the gaze points included in the sight line information of the driver within the period of time is less than a fifth threshold, it may be determined that the sight line of the driver deviates from the road ahead for a long time, and the driver is in a dangerous driving state. The value of the fifth threshold depends on the driving habits of the driver, for example, 50% and 60% of the period of time, and at this time, warning information needs to be displayed to remind the driver of paying attention to safe driving, so as to avoid traffic accidents.
In this embodiment, if the sum of the gaze durations of the gaze points in the first region to the second region is less than the sixth threshold value in the gaze points included in the gaze information of the driver within the period of time, it is determined that the driver's gaze deviates from the reason for a long time and is in a dangerous driving state. The sixth threshold is, for example, 95% or 90% of the period of time. At the moment, warning information needs to be displayed to remind a driver of paying attention to safe driving, so that traffic accidents are avoided.
In this embodiment, if the gazing duration of the driver to a certain gazing point is too long, for example, the gazing duration is greater than 30 seconds, it may be comprehensively determined whether the driver is in the distracted state according to the blinking frequency of the driver and the opening and closing degree of the glasses, and when the driver is in the distracted state, a warning message needs to be displayed to remind the driver to concentrate on attention.
As described above, the driving assistance method based on sight line detection according to the embodiment can determine whether the driver is in a dangerous driving state according to the sight line information of the driver, and display corresponding warning information in time when the driver is in the dangerous driving state to remind the driver of paying attention to safe driving, thereby reducing traffic accidents.
In the invention, the sight line information of the driver and the related information of other parts of the automobile can be used as the judgment basis together, so that the driving state of the driver can be accurately judged. For example, when the driver's sight line information includes a plurality of left gaze points and the driver turns the steering wheel to the left and lowers the vehicle speed, it is possible to accurately determine that the driving state of the driver is a left turn; when the sight line information of the driver comprises a plurality of left-side fixation points and the driver turns a steering wheel to the left and the vehicle speed is increased, the driving state of the driver can be accurately judged to be left-side overtaking. The method can judge the driving state of the driver more accurately and timely.
The driving habits of different drivers are different, and the sight line activities of the same driver are basically similar when the same driver faces the same scene, so that the current driving state of the driver can be determined according to the historical sight line information of the driver and the corresponding historical driving state. In an embodiment of the present invention, an implementation method for determining a driving state of a driver according to the sight line information includes: selecting a matched driving state from a database according to the sight line information; the database stores historical sight line information of the driver and a driving state matched with the historical sight line information.
In the present embodiment, a plurality of pairs of "historical sight line information-historical driving state" data are stored in the database in advance, each pair of "historical sight line information-historical driving state" data including sight line information of the driver and a driving state corresponding to the sight line information in a past period of time. And finding out historical sight line information matched with the current sight line information from the database by taking the current sight line information collected in the step S21 as a basis, and determining the current driving state of the driver according to the driving state corresponding to the historical sight line information.
In the embodiment, the driving state of the driver is determined by directly searching the database, and the algorithm is simple and efficient, but has a small application range. For the sight line information that has not occurred in the past, the present embodiment cannot find the matching historical sight line information from the database, and thus cannot determine the current driving state of the driver. The operations such as data storage and query can be performed locally in the automobile, for example, by a vehicle machine configured in the automobile, or can be performed in the cloud in a wireless communication manner.
In an embodiment of the present invention, an implementation method for determining a driving state of a driver according to the sight line information includes: the sight line information is used as the input of a machine learning model, and the output of the machine learning model is the driving state of the driver; the machine learning model is trained by using the historical sight line information of the driver and the driving state matched with the historical sight line information.
With the continuous development of artificial intelligence technology, machine learning technology is becoming more mature and widely applied to various industries. The machine learning utilizes a computer to simulate a human learning mode and effectively improves the learning efficiency by dividing the existing content into knowledge structures. Common machine learning models are, for example, decision trees, random forests, artificial neural networks, bayesian learning models, and the like.
In this embodiment, a plurality of pairs of "historical sight line information — historical driving state" data are obtained in advance and used as training data to train a machine learning model, so as to obtain a stable machine learning model. And taking the current sight line information of the driver collected in the step S21 as the input of the stable machine learning model, wherein the output of the stable machine learning model is the current driving state of the driver.
In the embodiment, the machine learning model is adopted to process the sight line information of the driver and obtain the driving state of the driver, and the algorithm has relatively high complexity, but has stronger robustness and applicability. With sufficient training, the current driving state of the driver can be determined from the current gaze information of the driver even if the current gaze information of the driver has never appeared in the past.
The machine learning model can establish a model framework suitable for a driver through training and learning and can be updated by self to improve the accuracy of driving state judgment. The training of the machine learning model and the processing of the video information thereof can be performed locally, for example, by a vehicle machine configured by a vehicle, or can be performed in the cloud.
In an embodiment of the present invention, an implementation method for selecting a corresponding image or video to display according to a driving state of a driver includes: selecting a corresponding blind area image or video to display according to the driving state of the driver; the blind area image or the video is acquired by a camera arranged on the outer side of the automobile.
In an embodiment of the present invention, the driving assistance method based on gaze detection further includes: the sight line information of the driver is tracked and the content observed by the driver is recorded.
In this embodiment, a rotatable camera is used to track the sight line information of the driver, and the focus of the rotatable camera is the same as the gaze point of the driver, and the focus changes with the change of the gaze point of the driver. The rotatable camera can be used for completely recording the content observed in the sight of the driver. The record can be stored in the local automobile or uploaded to a cloud server in real time. The said record contains complete driving route, scenery along the way, hit-and-run information, etc.
In an embodiment of the present invention, the driving assistance method based on gaze detection further includes: and selectively extracting corresponding information from the records and generating the driving record. For example: extracting scenes with the driver gazing time larger than a seventh threshold value from the records, and making the scenes into the driving record so that the driver can review or view the driving record when parking; the seventh threshold value is, for example, 0.5 second, 1 second, 2 seconds, or the like.
The present invention also provides a computer-readable storage medium having stored thereon a computer program; the computer program, when invoked, executes the gaze detection based driver assistance method of the present invention.
Referring to fig. 5, the present invention further provides a driving assistance system 5 based on line-of-sight detection; the line-of-sight detection-based driver assistance system 5 includes:
the line of sight collection device 51 is configured to execute step S21, that is: acquiring sight line information of a driver; the sight line information comprises the fixation points of the driver and the stay time of the sight line at each fixation point;
the processor 52 is disposed at a center console or a cloud end of the vehicle, is in communication with the sight line acquisition device 51, and is configured to execute step S22, that is: determining the driving state of the driver according to the sight line information;
a display 53 communicatively coupled to the processor 52 for performing step S23, namely: and selecting a corresponding image or video to display according to the driving state of the driver.
Referring to fig. 6, in an embodiment of the present invention, the automobile includes a left rear-view mirror 41, a right rear-view mirror 42, a center rear-view mirror 43, a front window 44, a video capture device 45, a display 46, a right a-pillar 47, and a left a-pillar 48.
The processor is realized by a vehicle machine or a motion processor of an automobile center console, and is in communication connection with the video acquisition device 45 and the display 46 through a CAN bus or a wireless communication mode. The video capture device 45 is disposed above the central rear view mirror 43, and can effectively capture the sight line information of the driver without blocking the sight line of the driver. In addition, the video capture device 45 may also be located on the left a-pillar 48, the roof, etc. The display 46 is disposed on the center console of the vehicle.
In an embodiment of the present invention, an implementation method of the processor determining the driving state of the driver according to the sight line information includes: if the sight line information contains the fixation points of a plurality of left areas within a period of time and the fixation time of the fixation points of the left areas is greater than a first threshold value, determining that the driver is in a left-turning or left-lane-changing state; and if the sight line information contains a plurality of fixation points of the right area within a period of time and the fixation time of the fixation points of the right area is greater than a second threshold value, determining that the driver is in a right turning or right lane changing state.
In an embodiment of the present invention, an implementation method for determining the driving state of the driver according to the sight line information by the processor includes: and if the sight line information contains the fixation points on the plurality of central rearview mirrors within a period of time and the fixation time of the fixation points on the central rearview mirrors is greater than a third threshold value, determining that the driver is in a reversing state.
In an embodiment of the present invention, an implementation method for determining the driving state of the driver according to the sight line information by the processor includes: if the watching time of the watching point of the front area contained in the sight line information is larger than a fourth threshold value within a period of time, determining that the driver is in a forward driving state; and if the watching time of the front area watching point contained in the sight line information in a period of time is less than a fifth threshold, determining that the driver is in a dangerous driving state.
In an embodiment of the present invention, the driving assistance system 5 based on the line of sight detection further includes a memory 54. The memory 54 is located at a center console or a cloud of the vehicle, and is connected to the processor 52 for storing a database; the database stores historical sight line information of the driver and driving states matching the historical sight line information, and the processor 52 can select a matching driving state from the database according to the sight line information.
In an embodiment of the present invention, an implementation method for determining the driving state of the driver according to the sight line information by the processor includes: the sight line information is used as the input of a machine learning model, and the output of the machine learning model is the driving state of the driver; the machine learning model is trained by using the historical sight line information of the driver and the driving state matched with the historical sight line information.
In an embodiment of the present invention, an implementation method for displaying the corresponding image or video selected by the display according to the driving state of the driver includes: selecting a corresponding blind area image or video to display according to the driving state of the driver; the blind area image or the video is acquired by a camera arranged on the outer side of the automobile.
In an embodiment of the present invention, the driving assistance system 6 based on gaze detection further includes a rotatable camera 49, and the rotatable camera 49 can track the gaze information of the driver and record the content observed by the driver. Specifically, the rotatable camera is in communication connection with the video acquisition device and the processor respectively; the processor controls the rotatable camera 49 to rotate and automatically focus to the current fixation point of the driver according to the driver sight line information collected by the video collecting device 51. The technology of controlling the camera to rotate and focus through the processor can be realized through the existing monitoring technology, and is not described herein again.
The protection scope of the driving assistance method based on the sight line detection is not limited to the execution sequence of the steps listed in the embodiment, and all the solutions of the prior art, such as the addition, the subtraction, and the step replacement according to the principle of the present invention, are included in the protection scope of the present invention.
The invention also provides a driving assistance system based on sight line detection, which can realize the driving assistance method based on sight line detection, but the device for realizing the driving assistance method based on sight line detection of the invention comprises but is not limited to the structure of the driving assistance system based on sight line detection, and all structural modifications and substitutions of the prior art made according to the principle of the invention are included in the protection scope of the invention.
The driving assisting method based on sight line detection can judge the driving state of the driver in advance according to the sight line information of the driver and display corresponding blind area information to the driver, and can effectively avoid traffic accidents caused by blind areas;
the auxiliary driving method based on sight line detection can determine whether the driver is in a dangerous driving state or not according to the sight line information of the driver and send corresponding warning information, so that traffic accidents caused by the fact that the sight line is separated from a road ahead for a long time are effectively avoided;
the driving assistance method based on sight line detection can be matched with other automobile parts and the like to accurately determine the driving state of a driver, so that a basis is provided for an automobile to take corresponding measures. Such other automotive parts are for example: steering wheel, accelerator pedal, brake pedal, etc.
In conclusion, the present invention effectively overcomes various disadvantages of the prior art and has high industrial utilization value.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.
Claims (10)
1. A driving assistance method based on sight line detection is characterized by comprising the following steps:
acquiring sight line information of a driver; the sight line information comprises fixation points of the driver and fixation time of each fixation point;
determining a driving state of a driver according to the sight line information, wherein the driving state of the driver comprises: forward driving, left turning, right turning, backing, left lane changing, right lane changing, left overtaking and dangerous driving;
and selecting corresponding driving information to display according to the driving state of the driver, wherein the driving information comprises navigation information, blind area images or video information, automobile running condition information and alarm information.
2. The sight-line detection-based driving assistance method according to claim 1, wherein one implementation method for determining the driving state of the driver from the sight-line information comprises:
dividing a driver's observation area into a front area, a left area, a right area, and a central rearview mirror area;
and determining the driving state of the driver according to the area to which the fixation point belongs and the fixation time of the fixation point of each area, wherein the area to which the fixation point belongs is contained in the sight line information.
3. The driving assistance method based on sight line detection according to claim 2, wherein one implementation method for determining the driving state of the driver according to the area to which the gaze point belongs and the gaze time of the gaze point in each area, which are included in the sight line information, comprises:
if the sight line information contains a plurality of fixation points of the left area within a period of time, and the fixation time of the fixation points of the left area is greater than a first threshold value, determining that the driver is in a left-turning or left lane-changing state;
if the sight line information contains a plurality of fixation points of the right area within a period of time, and the fixation time of the fixation points of the right area is greater than a second threshold value, determining that the driver is in a right turning or right lane changing state;
and if the sight line information contains the fixation points of a plurality of central rearview mirror areas within a period of time and the fixation time of the fixation points on the central rearview mirrors is greater than a third threshold value, determining that the driver is in a reversing state.
4. The driving assistance method based on visual line detection according to claim 2, wherein one implementation method for determining the driving state of the driver according to the area to which the gaze point belongs and the gaze time of the gaze point in each area, which are included in the visual line information, comprises:
if the fixation time of the fixation point of the front area contained in the sight line information in a period of time is greater than a fourth threshold value, determining that the driver is in a forward driving state;
and if the watching time of the front area watching point contained in the sight line information in a period of time is less than a fifth threshold, determining that the driver is in a dangerous driving state.
5. The sight-line detection-based driving assistance method according to claim 1, wherein one implementation method for determining the driving state of the driver from the sight-line information comprises:
selecting a matched driving state from a database according to the sight line information; the database stores historical sight line information of the driver and a driving state matched with the historical sight line information.
6. The sight-line detection-based driving assistance method according to claim 1, wherein one implementation method for determining the driving state of the driver from the sight-line information comprises:
the sight line information is used as the input of a machine learning model, and the output of the machine learning model is the driving state of the driver; the machine learning model is trained by using the historical sight line information of the driver and the driving state matched with the historical sight line information.
7. The driving assistance method based on sight line detection according to claim 1, wherein the blind area image or video is acquired by a camera provided outside the automobile.
8. The sight-line detection-based driving assist method according to claim 1, further comprising: the sight line information of the driver is tracked and the content observed by the driver is recorded.
9. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when invoked, performs a gaze detection based assisted driving method of any one of claims 1-8.
10. A driving assistance system based on sight line detection, characterized in that the driving assistance system based on sight line detection includes:
the sight line acquisition equipment is used for acquiring sight line information of a driver; the sight line information comprises the fixation points of the driver and the stay time of the sight line at each fixation point;
the processor is arranged on a central control station or a cloud end of the automobile, is in communication connection with the sight line acquisition equipment and is used for determining the driving state of the driver according to the sight line information, and the driving state of the driver comprises: forward driving, left turning, right turning, backing, left lane changing, right lane changing, left overtaking and dangerous driving;
and the display is in communication connection with the processor and is used for selecting corresponding blind area images or video information, navigation information, automobile running condition information and alarm information to display according to the driving state of the driver.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911275693.4A CN110962746B (en) | 2019-12-12 | 2019-12-12 | Driving assisting method, system and medium based on sight line detection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911275693.4A CN110962746B (en) | 2019-12-12 | 2019-12-12 | Driving assisting method, system and medium based on sight line detection |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110962746A CN110962746A (en) | 2020-04-07 |
| CN110962746B true CN110962746B (en) | 2022-07-19 |
Family
ID=70033967
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911275693.4A Active CN110962746B (en) | 2019-12-12 | 2019-12-12 | Driving assisting method, system and medium based on sight line detection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110962746B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114379461A (en) * | 2020-10-20 | 2022-04-22 | 采埃孚汽车科技(上海)有限公司 | Driving assistance apparatus and method for vehicle |
| CN112380935B (en) * | 2020-11-03 | 2023-05-26 | 深圳技术大学 | Man-machine collaborative sensing method and system for automatic driving |
| CN113276775B (en) * | 2021-04-26 | 2023-04-14 | 睿联汽车电子(芜湖)有限公司 | Intelligent adjusting method for outer rearview mirror |
| CN113421458A (en) * | 2021-06-16 | 2021-09-21 | 吉林大学 | Method for detecting dynamic blind area of driver of heavy commercial vehicle |
| CN113837027A (en) * | 2021-09-03 | 2021-12-24 | 东风柳州汽车有限公司 | Driving assistance sensing method, device, equipment and storage medium |
| CN113989466B (en) * | 2021-10-28 | 2022-09-20 | 江苏濠汉信息技术有限公司 | Beyond-the-horizon assistant driving system based on situation cognition |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102963301B (en) * | 2012-10-25 | 2015-07-08 | 浙江吉利汽车研究院有限公司杭州分公司 | Automobile visual angle auxiliary device |
| CN106143307B (en) * | 2016-07-11 | 2019-07-19 | 京东方科技集团股份有限公司 | A kind of driving assistance system and vehicle |
| CN107953827A (en) * | 2016-10-18 | 2018-04-24 | 杭州海康威视数字技术股份有限公司 | A kind of vehicle blind zone method for early warning and device |
| CN108501809A (en) * | 2018-03-26 | 2018-09-07 | 京东方科技集团股份有限公司 | Vehicular display control device, display system based on Eye-controlling focus and display methods |
| CN108657184A (en) * | 2018-07-06 | 2018-10-16 | 京东方科技集团股份有限公司 | Vehicle DAS (Driver Assistant System), method, apparatus and storage medium |
| CN108515912B (en) * | 2018-08-06 | 2018-11-02 | 江苏中科朗恩斯车辆科技有限公司 | Vehicle, vehicle mirrors and its mirror driver and minute surface driving method |
-
2019
- 2019-12-12 CN CN201911275693.4A patent/CN110962746B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110962746A (en) | 2020-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110962746B (en) | Driving assisting method, system and medium based on sight line detection | |
| US10777083B2 (en) | Driving assistance systems and methods | |
| CN112965504B (en) | Remote confirmation method, device and equipment based on automatic driving and storage medium | |
| CN107176165B (en) | Vehicle control device | |
| CN102673473B (en) | With the read out instrument of display commercial vehicle directly perceived | |
| DE102018108200A1 (en) | Systems and methods for reducing attentiveness while maintaining readiness by augmented reality in a vehicle | |
| DE102019208663B4 (en) | Method for supporting a driver with regard to objects and motor vehicle relevant to a traffic situation | |
| WO2015134840A2 (en) | Vehicular visual information system and method | |
| CN108604413B (en) | Display device control method and display device | |
| CN110544368B (en) | Fatigue driving augmented reality early warning device and early warning method | |
| US20210053586A1 (en) | Systems and methods for improving visual scanning behavior associated with controlling a vehicle | |
| DE102018105577A1 (en) | Parking assistance method and parking aid device, automatic driving control device and program that use the parking assistance method | |
| CN112084232A (en) | Vehicle driving risk assessment method and device based on visual field information of other target vehicles | |
| DE102018127605A1 (en) | INFORMATION PROCESSING DEVICE, DRIVER MONITORING SYSTEM, INFORMATION PROCESSING METHOD AND INFORMATION PROCESSING PROGRAM | |
| CN114872713A (en) | Device and method for monitoring abnormal driving state of driver | |
| DE102018127600A1 (en) | Information processing apparatus, driver monitoring system, information processing method and information processing program | |
| DE112016003438T5 (en) | DRIVING ASSISTANCE DEVICE | |
| CN116674468A (en) | Image display method and related device, vehicle, storage medium, and program | |
| DE112019007195T5 (en) | DISPLAY CONTROL DEVICE, DISPLAY CONTROL METHOD AND DISPLAY CONTROL PROGRAM | |
| JP5223289B2 (en) | Visual information presentation device and visual information presentation method | |
| Mizuno et al. | Detecting driver's visual attention area by using vehicle-mounted device | |
| US11908208B2 (en) | Interface sharpness distraction mitigation method and system | |
| CN113212451A (en) | Rearview auxiliary system for intelligent driving automobile | |
| CN114202965B (en) | Driving assistance method and device, vehicle-mounted terminal and storage medium | |
| CN115134491B (en) | Image processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |