CN107909037B - Information output method and device - Google Patents

Information output method and device Download PDF

Info

Publication number
CN107909037B
CN107909037B CN201711136211.8A CN201711136211A CN107909037B CN 107909037 B CN107909037 B CN 107909037B CN 201711136211 A CN201711136211 A CN 201711136211A CN 107909037 B CN107909037 B CN 107909037B
Authority
CN
China
Prior art keywords
video
sample
fatigue driving
driver
time period
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201711136211.8A
Other languages
Chinese (zh)
Other versions
CN107909037A (en
Inventor
贾巍
商兴奇
李宏言
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to CN201711136211.8A priority Critical patent/CN107909037B/en
Publication of CN107909037A publication Critical patent/CN107909037A/en
Application granted granted Critical
Publication of CN107909037B publication Critical patent/CN107909037B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B60W2040/0818—Inactivity or incapacity of driver
    • B60W2040/0827—Inactivity or incapacity of driver due to sleepiness

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Automation & Control Theory (AREA)
  • Mathematical Physics (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Image Analysis (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

The embodiment of the application discloses an information output method and device. One embodiment of the method comprises: acquiring a target video recorded with a current driving process of a target driver; inputting the target video into a pre-trained deep learning model to obtain the predicted fatigue driving occurrence time period of the target driver, wherein the deep learning model is used for representing and recording the corresponding relation between the video of the current driving process of the driver and the predicted fatigue driving occurrence time period of the driver; determining whether the predicted fatigue driving occurrence time period of the target driver is within a preset time period range; in response to determining that the predicted fatigue driving occurrence period of the target driver is within the preset period range, a predicted fatigue driving occurrence period of the target driver is output. The embodiment can predict the fatigue driving occurrence time period of the driver and output the predicted fatigue driving occurrence time period, and is helpful for reducing traffic accidents caused by fatigue driving.

Description

Information output method and device
Technical Field
The embodiment of the application relates to the technical field of computers, in particular to the technical field of internet, and particularly relates to an information output method and device.
Background
The driving fatigue refers to a phenomenon that a driver has a disorder of physiological and psychological functions after driving for a long time, and the driving skill is objectively reduced. The driver has poor or insufficient sleeping quality, and is easy to fatigue when driving the vehicle for a long time. Driving fatigue affects the driver's attention, feeling, perception, thinking, judgment, consciousness, decision and movement.
The judgment ability is reduced, the reaction is slow and the operation error is increased when the driver is tired. When a driver is in slight fatigue, untimely and inaccurate gear shifting can occur; when the driver is in moderate fatigue, the operation action is dull, and sometimes even the driver forgets the operation; when a driver is severely tired, the driver is often conscious of operation or sleeps for a short time, and the control capability of the vehicle is lost in severe cases. When a driver is tired, the phenomena of blurred vision, soreness and pain of the waist and back, stiff movements, fullness in hands and feet, or lack of concentration of energy, slow reaction, poor thinking, distraction, anxiety, impatience and the like can occur. If the vehicle is still being driven barely, a traffic accident may occur.
Disclosure of Invention
The embodiment of the application provides an information output method and device.
In a first aspect, an embodiment of the present application provides an information output method, where the method includes: acquiring a target video recorded with a current driving process of a target driver; inputting the target video into a pre-trained deep learning model to obtain the predicted fatigue driving occurrence time period of the target driver, wherein the deep learning model is used for representing and recording the corresponding relation between the video of the current driving process of the driver and the predicted fatigue driving occurrence time period of the driver; determining whether the predicted fatigue driving occurrence time period of the target driver is within a preset time period range; in response to determining that the predicted fatigue driving occurrence period of the target driver is within the preset period range, a predicted fatigue driving occurrence period of the target driver is output.
In some embodiments, the method further comprises: and acquiring and outputting prompt information including a mode of relieving fatigue driving.
In some embodiments, the deep learning model includes a convolutional neural network, a cyclic neural network, and a fully-connected layer.
In some embodiments, inputting the target video into a pre-trained deep learning model to obtain the predicted fatigue driving occurrence time period of the target driver comprises: inputting the target video into a convolutional neural network to obtain the feature vectors of each frame of image of the target video, wherein the convolutional neural network is used for representing the corresponding relationship between the video and the feature vectors of each frame of image of the video; inputting the feature vectors of each frame of image of the target video into a recurrent neural network to obtain the feature vectors of the target video, wherein the recurrent neural network is used for representing the corresponding relation between the feature vectors of each frame of image of the video and the feature vectors of the video, and the feature vectors of the video are used for representing the incidence relation between the feature vectors of each frame of image of the video; and inputting the characteristic vector of the target video into a full-connection layer to obtain the predicted fatigue driving occurrence time period of the target driver, wherein the full-connection layer is used for representing the corresponding relation between the characteristic vector of the video and the predicted fatigue driving occurrence time period of the driver.
In some embodiments, the deep learning model is trained by: the method comprises the steps of obtaining a plurality of sample videos recorded with driving processes of sample drivers and fatigue driving occurrence time periods of the sample drivers corresponding to each sample video in the plurality of sample videos; and taking each sample video in the plurality of sample videos as input, taking the fatigue driving occurrence time period of the sample driver corresponding to each sample video in the plurality of sample videos as output, and training to obtain the deep learning model.
In some embodiments, training a deep learning model by taking each sample video of the plurality of sample videos as an input and taking a fatigue driving occurrence time period of a sample driver corresponding to each sample video of the plurality of sample videos as an output includes: the following training steps are performed: sequentially inputting each sample video in the plurality of sample videos to the initialized deep learning model to obtain a predicted fatigue driving occurrence time period of a sample driver corresponding to each sample video in the plurality of sample videos, comparing the predicted fatigue driving occurrence time period of the sample driver corresponding to each sample video in the plurality of sample videos with the predicted fatigue driving occurrence time period of the sample driver corresponding to the sample video to obtain a predicted accuracy of the initialized deep learning model, determining whether the predicted accuracy is greater than a preset accuracy threshold, and if the predicted accuracy is greater than the preset accuracy threshold, taking the initialized deep learning model as the trained deep learning model.
In some embodiments, training a deep learning model by using each sample video of the plurality of sample videos as an input and using a fatigue driving occurrence time period of a sample driver corresponding to each sample video of the plurality of sample videos as an output further includes: and responding to the condition that the accuracy is not larger than the preset accuracy threshold value, adjusting the parameters for initializing the deep learning model, and continuing to execute the training step.
In a second aspect, an embodiment of the present application provides an information output apparatus, including: the target video acquisition unit is configured for acquiring a target video recorded with a current driving process of a target driver; the fatigue driving occurrence time period prediction unit is configured to input a target video to a pre-trained deep learning model to obtain a predicted fatigue driving occurrence time period of a target driver, wherein the deep learning model is used for representing a corresponding relation between a video recorded with a current driving process of the driver and the predicted fatigue driving occurrence time period of the driver; a fatigue driving occurrence period determination unit configured to determine whether a predicted fatigue driving occurrence period of the target driver is within a preset period range; a fatigue driving occurrence period output unit configured to output the predicted fatigue driving occurrence period of the target driver in response to the determination that the predicted fatigue driving occurrence period is within the preset period range.
In some embodiments, the apparatus further comprises: and the prompt information acquisition unit is configured to acquire and output prompt information including a mode of relieving fatigue driving.
In some embodiments, the deep learning model includes a convolutional neural network, a cyclic neural network, and a fully-connected layer.
In some embodiments, the fatigue driving occurrence period prediction unit includes: the image feature vector acquisition subunit is configured to input the target video to a convolutional neural network to obtain feature vectors of each frame of image of the target video, wherein the convolutional neural network is used for representing the corresponding relationship between the video and the feature vectors of each frame of image of the video; the video feature vector acquisition subunit is configured to input feature vectors of each frame of image of the target video to a recurrent neural network to obtain feature vectors of the target video, wherein the recurrent neural network is used for representing a corresponding relationship between the feature vectors of each frame of image of the video and the feature vectors of the video, and the feature vectors of the video are used for representing an association relationship between the feature vectors of each frame of image of the video; and the fatigue driving occurrence time period predicting subunit is configured to input the feature vector of the target video to a full connection layer to obtain the predicted fatigue driving occurrence time period of the target driver, wherein the full connection layer is used for representing the corresponding relation between the feature vector of the video and the predicted fatigue driving occurrence time period of the driver.
In some embodiments, the apparatus further comprises a deep learning model training unit comprising: the system comprises a sample acquisition subunit, a display unit and a control unit, wherein the sample acquisition subunit is configured to acquire a plurality of sample videos in which the driving process of a sample driver is recorded and a fatigue driving occurrence time period of the sample driver corresponding to each sample video in the plurality of sample videos; and the deep learning model training subunit is configured to take each sample video in the plurality of sample videos as input, take a fatigue driving occurrence time period of a sample driver corresponding to each sample video in the plurality of sample videos as output, and train to obtain the deep learning model.
In some embodiments, the deep learning model training subunit comprises: a deep learning model training module configured to perform the following training steps: sequentially inputting each sample video in the plurality of sample videos to the initialized deep learning model to obtain a predicted fatigue driving occurrence time period of a sample driver corresponding to each sample video in the plurality of sample videos, comparing the predicted fatigue driving occurrence time period of the sample driver corresponding to each sample video in the plurality of sample videos with the predicted fatigue driving occurrence time period of the sample driver corresponding to the sample video to obtain a predicted accuracy of the initialized deep learning model, determining whether the predicted accuracy is greater than a preset accuracy threshold, and if the predicted accuracy is greater than the preset accuracy threshold, taking the initialized deep learning model as the trained deep learning model.
In some embodiments, the deep learning model training subunit further comprises: and the parameter adjusting module is configured to respond to the condition that the accuracy is not greater than a preset accuracy threshold value, adjust the parameters of the initialized deep learning model and continue to execute the training step.
In a third aspect, an embodiment of the present application provides an electronic device, including: one or more processors; storage means for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any implementation of the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the method as described in any implementation manner of the first aspect.
According to the information output method and the information output device, the target video recorded with the current driving process of the target driver is input into the pre-trained deep learning model, so that the predicted fatigue driving occurrence time period of the target driver is obtained; then determining whether the predicted fatigue driving occurrence time period of the target driver is within a preset time period range; and finally outputting the predicted fatigue driving occurrence time period of the target driver in the case where it is determined that the predicted fatigue driving occurrence time period is within the preset time period range. Therefore, the occurrence time period of fatigue driving of the driver can be predicted, the predicted occurrence time period of fatigue driving is output, and the target driver can take corresponding measures in the predicted occurrence time period of fatigue driving, so that traffic accidents caused by fatigue driving can be reduced.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is an exemplary system architecture diagram to which embodiments of the present application may be applied;
FIG. 2 is a flow diagram of one embodiment of an information output method according to the present application;
FIG. 3 is a flow chart of yet another embodiment of an information output method according to the present application;
FIG. 4 is a flow diagram of one embodiment of a deep learning model training method according to the present application;
FIG. 5 is a schematic block diagram of an embodiment of an information output apparatus according to the present application;
FIG. 6 is a schematic block diagram of a computer system suitable for use in implementing an electronic device according to embodiments of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Fig. 1 shows an exemplary system architecture 100 to which an information output method or an information output apparatus of an embodiment of the present application can be applied.
As shown in fig. 1, system architecture 100 may include a car 101, a network 102, and a server 103. Network 102 is the medium used to provide a communication link between automobile 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The car 101 may interact with a server 103 via a network 102 to receive or send messages or the like. The automobile 101 may be equipped with a video capture device (e.g., a camera, video camera, etc.) for capturing video of the driver's driving process.
The server 103 may be an in-vehicle server installed in the automobile 101, or may be a backend server for controlling the automobile 101. The server 103 may provide various services, for example, the server 103 may perform processing such as analysis on the acquired data such as the target video in which the current driving process of the target driver is recorded, and output the processing result (for example, the predicted fatigue driving occurrence time period of the target driver).
It should be noted that the information output method provided in the embodiment of the present application is generally executed by the server 103, and accordingly, the information output apparatus is generally disposed in the server 103.
It should be understood that the number of cars, networks, and servers in fig. 1 is merely illustrative. There may be any number of cars, networks, and servers, as desired for implementation.
With continued reference to FIG. 2, a flow 200 of one embodiment of an information output method according to the present application is shown. The information output method comprises the following steps:
step 201, a target video recorded with a current driving process of a target driver is acquired.
In this embodiment, the electronic device (for example, the server 103 shown in fig. 1) on which the information output method operates may acquire the target video in which the current driving process of the target driver is recorded from a video capture device installed on an automobile (for example, the automobile 101 shown in fig. 1) by a wired connection manner or a wireless connection manner. The video acquisition device can be a camera, a video camera and the like, is usually arranged in front of an automobile, and the lens of the video acquisition device is opposite to the main driving position so as to acquire the video of the driving process of a driver. Here, the video acquisition device can acquire a video of a driving process of the target driver in the current time period in real time and transmit the acquired video to the electronic device in real time.
Step 202, inputting the target video into a pre-trained deep learning model to obtain the predicted fatigue driving occurrence time period of the target driver.
In this embodiment, based on the target video acquired in step 201, the electronic device may input the target video to a deep learning model trained in advance, so as to obtain a predicted fatigue driving occurrence time period of the target driver. The fatigue driving phenomenon can occur after a period of time when the driver has the phenomena of long-time eye closure, frequent yawning, frequent nodding, long-time sight line deviation and the like in the driving process. The different phenomena may correspond to different fatigue driving occurrence periods, for example, the fatigue driving occurrence period corresponding to the long-time eye-closing phenomenon may be within 5 to 10 minutes, and the fatigue driving occurrence period corresponding to the frequent yawning phenomenon may be within 10 to 20 minutes. When the driver does not have the fatigue driving phenomenon in the driving process, the fatigue driving occurrence time period of the driver can be determined according to the current continuous driving time period of the driver, for example, if the driver has driven for 1 hour continuously, the corresponding fatigue driving occurrence time period can be within 2.5 to 3 hours, and if the driver has driven for 3.5 hours continuously, the corresponding fatigue driving occurrence time period can be within 0 to 30 minutes.
In this embodiment, the deep learning model may be an artificial neural network, which abstracts the human brain neuron network from the information processing perspective, establishes a simple model, and forms different networks according to different connection modes. Usually, the system is composed of a large number of nodes (or neurons) connected to each other, each node representing a specific output function, called a stimulus function. The connection between each two nodes represents a weighted value, called weight (also called parameter), for the signal passing through the connection, and the output of the network varies according to the connection mode, the weight value and the excitation function of the network. The deep learning model generally includes a plurality of layers, each layer includes a plurality of nodes, and in general, the weight of the node in the same layer may be the same, and the weight of the node in different layers may be different, so the parameters of the plurality of layers of the deep learning model may also be different. Here, the electronic device may input the target video from the input side of the deep learning model, sequentially undergo processing (for example, multiplication, convolution, or the like) of parameters of each layer in the deep learning model, and output from the output side of the deep learning model, the information output from the output side being the predicted fatigue driving occurrence period of the target driver.
In this embodiment, the deep learning model may be used to represent a correspondence between a video in which a current driving process of the driver is recorded and a predicted time period during which fatigue driving occurs of the driver, and the electronic device may train the deep learning model in various ways, which may represent a correspondence between a video in which a current driving process of the driver is recorded and a predicted time period during which fatigue driving occurs of the driver.
As an example, the electronic device may generate a correspondence table storing a plurality of correspondences between videos in which the driving course of the driver is recorded and the fatigue driving occurrence periods of the driver, based on statistics of a large number of videos in which the driving course of the driver is recorded and the fatigue driving occurrence periods of the driver, and use the correspondence table as the deep learning model. In this way, the electronic device may sequentially compare the target video with the videos in the correspondence table, in which the driving processes of the drivers are recorded, and if one of the videos in the correspondence table is the same as or similar to the driving process in the target video, the fatigue driving occurrence time period of the driver corresponding to the video in the correspondence table is used as the predicted fatigue driving occurrence time period of the target driver.
As another example, the electronic device may first obtain a plurality of sample videos in which driving processes of sample drivers are recorded and a fatigue driving occurrence time period of the sample driver corresponding to each of the plurality of sample videos; and then, taking each sample video in the plurality of sample videos as input, taking the fatigue driving occurrence time period of the sample driver corresponding to each sample video in the plurality of sample videos as output, and training to obtain the deep learning model. Here, the electronic device may obtain a plurality of sample videos in which driving processes of the sample drivers are recorded and play the sample videos for a person skilled in the art, and the person skilled in the art may label a fatigue driving occurrence period of the sample driver for each of the plurality of sample videos based on experience. The electronic equipment can train an initialized deep learning model, the initialized deep learning model can be an untrained deep learning model or an untrained deep learning model, each layer of the initialized deep learning model can be provided with initial parameters, and the parameters can be continuously adjusted in the training process of the deep learning model. The initialized deep learning model can be various types of untrained or untrained artificial neural networks or a model obtained by combining various types of untrained or untrained artificial neural networks, for example, the initialized deep learning model can be an untrained convolutional neural network, an untrained cyclic neural network, or a model obtained by combining an untrained convolutional neural network, an untrained cyclic neural network and an untrained full-connected layer. In this way, the electronic device can input the target video from the input side of the deep learning model, sequentially process the parameters of each layer in the deep learning model, and output the target video from the output side of the deep learning model, wherein the information output by the output side is the predicted fatigue driving occurrence time period of the target driver.
In step 203, it is determined whether the predicted fatigue driving occurrence period of the target driver is within a preset period range.
In this embodiment, based on the predicted fatigue driving occurrence time period of the target driver obtained in step 202, the electronic device may compare the minimum value of the predicted fatigue driving occurrence time period of the target driver with the minimum value of the preset time period, compare the maximum value of the predicted fatigue driving occurrence time period of the target driver with the maximum value of the preset time period, and determine that the predicted fatigue driving occurrence time period of the target driver is within the preset time period range if the minimum value of the predicted fatigue driving occurrence time period of the target driver is not less than the minimum value of the preset time period and the maximum value of the predicted fatigue driving occurrence time period of the target driver is not greater than the maximum value of the preset time period; otherwise, determining that the predicted fatigue driving occurrence time period of the target driver is not within the preset time period range. The preset time period may be a time period preset by a person skilled in the art or a driver according to experience, and the person skilled in the art or the driver may arbitrarily adjust the preset time period. For example, the preset time period may be within 0-30 minutes.
In response to determining that the predicted fatigue driving occurrence period is within the preset period range, step 204 outputs a predicted fatigue driving occurrence period for the target driver.
In the present embodiment, the electronic device may output the predicted fatigue driving occurrence period of the target driver in a case where it is determined that the predicted fatigue driving occurrence period of the target driver is within the preset period range. For example, the electronic device may send the predicted time period of occurrence of fatigue driving of the target driver to an audio playing device installed on the automobile, the audio playing device may play the predicted time period of occurrence of fatigue driving of the target driver, and the target driver may take corresponding measures within the predicted time period of occurrence of fatigue driving, thereby reducing traffic accidents caused by fatigue driving. For example, stimulating the face with cool air or cold water, drinking a cup of hot tea or coffee, listening to light music or turning the sound up appropriately or activating an autopilot system.
In some optional implementation manners of the embodiment, the electronic device may further obtain prompt information including a manner of alleviating fatigue driving, and output the prompt information. For example, the electronic device may send the prompt information including the manner of alleviating fatigue driving to an audio playing device installed on the automobile, the audio playing device may play the prompt information including the manner of alleviating fatigue driving, and the target driver may take corresponding measures according to the prompt of the prompt information, thereby reducing traffic accidents caused by fatigue driving.
According to the information output method provided by the embodiment of the application, the target video recorded with the current driving process of the target driver is input into the pre-trained deep learning model, so that the predicted fatigue driving occurrence time period of the target driver is obtained; then determining whether the predicted fatigue driving occurrence time period of the target driver is within a preset time period range; and finally outputting the predicted fatigue driving occurrence time period of the target driver in the case where it is determined that the predicted fatigue driving occurrence time period is within the preset time period range. Therefore, the occurrence time period of fatigue driving of the driver can be predicted, the predicted occurrence time period of fatigue driving is output, and the target driver can take corresponding measures in the predicted occurrence time period of fatigue driving, so that traffic accidents caused by fatigue driving can be reduced.
With further reference to fig. 3, a flow 300 of yet another embodiment of an information output method according to the present application is shown. In this embodiment, the deep learning model may include a convolutional neural network, a cyclic neural network, and a fully-connected layer, and the flow 300 of the information output method includes the following steps:
step 301, a target video recorded with a current driving process of a target driver is obtained.
In this embodiment, the electronic device (for example, the server 103 shown in fig. 1) on which the information output method operates may acquire the target video in which the current driving process of the target driver is recorded from a video capture device installed on an automobile (for example, the automobile 101 shown in fig. 1) by a wired connection manner or a wireless connection manner. The video acquisition device can be a camera, a video camera and the like, is usually arranged in front of an automobile, and the lens of the video acquisition device is opposite to the main driving position so as to acquire the video of the driving process of a driver. Here, the video acquisition device can acquire a video of a driving process of the target driver in the current time period in real time and transmit the acquired video to the electronic device in real time.
Step 302, inputting the target video into the convolutional neural network to obtain the feature vector of each frame of image of the target video.
In this embodiment, based on the target video obtained in step 301, the electronic device may input the target video to the convolutional neural network, so as to obtain a feature vector of each frame of image of the target video. The video is generally composed of a plurality of frames of images, and the feature vector of each frame of image of the target video can be used for describing features of each frame of image, such as action features of a driver, expression features of the driver, and the like.
In this embodiment, the convolutional neural network may be a feedforward neural network whose artificial neurons may respond to a portion of the coverage of surrounding cells, which may perform well for large image processing. In general, the basic structure of a convolutional neural network includes two layers, one of which is a feature extraction layer, and the input of each neuron is connected to the local acceptance domain of the previous layer and extracts the features of the local acceptance domain. Once the local feature is extracted, the position relation between the local feature and other features is determined; the other is a feature mapping layer, each calculation layer of the network is composed of a plurality of feature mappings, each feature mapping is a plane, and the weights of all neurons on the plane are equal. Here, the electronic device may input the target video from an input side of the convolutional neural network, sequentially perform processing on parameters of each layer in the convolutional neural network, and output the target video from an output side of the convolutional neural network, where information output by the output side is a feature vector of each frame image of the target video.
In this embodiment, the convolutional neural network may be used to represent the correspondence between the feature vectors of the video and the respective frame images of the video, and the electronic device may train the convolutional neural network that can represent the correspondence between the feature vectors of the video and the respective frame images of the video in various ways.
As an example, the electronic device may generate a correspondence table storing correspondences between a plurality of videos and feature vectors of respective frame images of the videos based on counting the feature vectors of a large number of videos and respective frame images of the videos, and treat the correspondence table as a convolutional neural network. In this way, the electronic device may sequentially compare the target video with the plurality of videos in the correspondence table, and if one video in the correspondence table is the same as or similar to the target video, use the feature vector of each frame image of the video in the correspondence table as the feature vector of each frame image of the target video.
As another example, the electronic device may first obtain a sample video and a feature vector of each frame image of the sample video; and then taking the sample video as input, taking the feature vector of each frame image of the sample video as output, and training to obtain a convolutional neural network capable of representing the corresponding relation between the video and the feature vector of each frame image of the video. In this way, the electronic device can input the target video from the input side of the convolutional neural network, sequentially process the parameters of each layer in the convolutional neural network, and output the target video from the output side of the convolutional neural network, where the information output by the output side is the feature vector of each frame of image of the target video.
Step 303, inputting the feature vector of each frame of image of the target video into the recurrent neural network to obtain the feature vector of the target video.
In this embodiment, based on the feature vector of each frame image of the target video obtained in step 302, the electronic device may input the feature vector of each frame image of the target video to the recurrent neural network, so as to obtain the feature vector of the target video. The feature vectors of the video can be used for representing the association relationship among the feature vectors of the frame images of the video.
In this embodiment, the recurrent neural network is an artificial neural network with nodes directionally connected into a ring. The essential feature of such a network is that there is both an internal feedback and a feed-forward connection between the processing units, the internal state of which may exhibit dynamic timing behavior.
In this embodiment, the recurrent neural network may be used to represent a correspondence between feature vectors of each frame of image of the video and feature vectors of the video, and the electronic device may train the recurrent neural network that can represent a correspondence between feature vectors of each frame of image of the video and feature vectors of the video in various ways.
As an example, the electronic device may generate a correspondence table storing correspondence between feature vectors of frame images of a plurality of videos and feature vectors of videos based on statistics of the feature vectors of the frame images of a large number of videos and the feature vectors of the videos, and use the correspondence table as a recurrent neural network. In this way, the electronic device may calculate euclidean distances between the feature vector of each frame image of the target video and the feature vectors of each frame image of the plurality of videos in the correspondence table, and if the euclidean distance between the feature vector of each frame image of one video and the feature vector of each frame image of the target video in the correspondence table is greater than a preset distance threshold, use the feature vector of the video in the correspondence table as the feature vector of the target video.
As another example, the electronic device may first obtain a feature vector of each frame image of the sample video and a feature vector of the sample video; and then, taking the feature vector of each frame image of the sample video as input, taking the feature vector of the sample video as output, and training to obtain the recurrent neural network capable of representing the corresponding relation between the feature vector of each frame image of the video and the feature vector of the video. In this way, the electronic device can input the feature vectors of each frame of image of the target video from the input side of the recurrent neural network, sequentially process the parameters of each layer in the recurrent neural network, and output the feature vectors from the output side of the recurrent neural network, wherein the information output from the output side is the feature vectors of the target video.
Step 304, inputting the characteristic vector of the target video into a full-link layer to obtain the predicted fatigue driving occurrence time period of the target driver,
in this embodiment, based on the feature vector of the target video obtained in step 303, the electronic device may input the feature vector of the target video to the full-link layer, so as to obtain the predicted fatigue driving occurrence time period of the target driver. The fatigue driving phenomenon can occur after a period of time when the driver has the phenomena of long-time eye closure, frequent yawning, frequent nodding, long-time sight line deviation and the like in the driving process. Different phenomena may correspond to different periods of fatigue driving occurrence. When the fatigue driving phenomenon does not occur in the driving process of the driver, the fatigue driving occurrence time period of the driver can be determined according to the current continuous driving time period of the driver.
In this embodiment, each node of the fully-connected layer is connected to all nodes of the output layer of the recurrent neural network, and is used to integrate the feature vectors of the video output by the output layer of the recurrent neural network. The parameters of a fully connected layer are also typically the most due to its fully connected nature. Meanwhile, after the characteristic vector of the video is subjected to linear transformation by using the parameters of the full connection layer, a nonlinear excitation function can be added to convert the linear transformation result, so that a nonlinear factor is introduced to enhance the expression capability of the deep learning model. The excitation function may be a softmax function, which is a common excitation function in an artificial neural network and is not described in detail herein.
In this embodiment, the fully connected layer may be used to represent a correspondence between the feature vector of the video and the predicted fatigue driving occurrence time period of the driver, and the electronic device may train the fully connected layer that may represent a correspondence between the feature vector of the video and the predicted fatigue driving occurrence time period of the driver in various ways.
As an example, the electronic device may generate a correspondence table storing correspondence between feature vectors of a plurality of videos and a fatigue driving occurrence period of the driver in the video based on counting the feature vectors of a large number of videos and the fatigue driving occurrence period of the driver in the video, and regard the correspondence table as a full connection layer. In this way, the electronic device may calculate euclidean distances between the feature vector of the target video and the feature vectors of the plurality of videos in the correspondence table, and if the euclidean distance between the feature vector of one video and the feature vector of the target video in the correspondence table is greater than a preset distance threshold, take the fatigue driving occurrence time period of the driver in the video in the correspondence table as the predicted fatigue driving occurrence time period of the target driver.
As another example, the electronic device may first obtain a feature vector of the sample video and a fatigue driving occurrence period of the driver in the sample video; and then, taking the feature vector of the sample video as input, taking the fatigue driving occurrence time period of the driver in the sample video as output, and training to obtain a full connection layer capable of representing the corresponding relation between the feature vector of the video and the predicted fatigue driving occurrence time period of the driver. In this way, the electronic device can input the feature vector of the target video from the input side of the full connection layer, process the parameters and the excitation function of the full connection layer, and output the feature vector from the output side of the full connection layer, wherein the information output by the output side is the predicted fatigue driving occurrence time period of the target driver.
It should be noted that the convolutional neural network, the cyclic neural network, and the full connection layer in the deep learning model may be trained separately, or may be trained simultaneously as a whole, which is not limited in this embodiment.
In step 305, it is determined whether the predicted fatigue driving occurrence period of the target driver is within a preset period range.
In this embodiment, based on the predicted fatigue driving occurrence time period of the target driver obtained in step 304, the electronic device may compare the minimum value of the predicted fatigue driving occurrence time period of the target driver with the minimum value of the preset time period, compare the maximum value of the predicted fatigue driving occurrence time period of the target driver with the maximum value of the preset time period, and determine that the predicted fatigue driving occurrence time period of the target driver is within the preset time period range if the minimum value of the predicted fatigue driving occurrence time period of the target driver is not less than the minimum value of the preset time period and the maximum value of the predicted fatigue driving occurrence time period of the target driver is not greater than the maximum value of the preset time period; otherwise, determining that the predicted fatigue driving occurrence time period of the target driver is not within the preset time period range.
Step 306, in response to determining that the predicted fatigue driving occurrence time period of the target driver is within the preset time period range, outputting the predicted fatigue driving occurrence time period of the target driver.
In the present embodiment, the electronic device may output the predicted fatigue driving occurrence period of the target driver in a case where it is determined that the predicted fatigue driving occurrence period of the target driver is within the preset period range. For example, the electronic device may send the predicted time period of occurrence of fatigue driving of the target driver to an audio playing device installed on the automobile, the audio playing device may play the predicted time period of occurrence of fatigue driving of the target driver, and the target driver may take corresponding measures within the predicted time period of occurrence of fatigue driving, thereby reducing traffic accidents caused by fatigue driving.
As can be seen from fig. 3, compared with the embodiment corresponding to fig. 2, the flow 300 of the information output method in the present embodiment highlights the structure of the deep learning model and the working principle of each structure. Thus, the scheme described in the embodiment can improve the accuracy of predicting the fatigue driving occurrence time period of the driver, thereby contributing to further reducing traffic accidents caused by fatigue driving.
With further reference to FIG. 4, a flow 400 of one embodiment of a deep learning model training method according to the present application is shown. The process 400 of the deep learning model training method includes the following steps:
step 401, obtaining a plurality of sample videos recorded with driving processes of sample drivers and fatigue driving occurrence time periods of the sample drivers corresponding to each sample video in the plurality of sample videos.
In this embodiment, the electronic device (for example, the service 103 shown in fig. 1) on which the deep learning model training method is executed may obtain a plurality of sample videos in which driving processes of sample drivers are recorded and a fatigue driving occurrence period of the sample driver corresponding to each of the plurality of sample videos.
In this embodiment, the electronic device may obtain a plurality of sample videos in which the driving process of the sample driver is recorded, and play the sample videos for a person skilled in the art, and the person skilled in the art may label, according to experience, a fatigue driving occurrence time period of the sample driver for each sample video in the plurality of sample videos.
Step 402, sequentially inputting each sample video of the plurality of sample videos to the initialized deep learning model, and obtaining a predicted fatigue driving occurrence time period of the sample driver corresponding to each sample video of the plurality of sample videos.
In this embodiment, based on the plurality of sample videos recorded with the driving process of the sample driver obtained in step 401, the electronic device may sequentially input each of the plurality of sample videos to the initialized deep learning model, so as to obtain the predicted fatigue driving occurrence time period of the sample driver corresponding to each of the plurality of sample videos. Here, the electronic device may input each sample video from an input side of the initialized deep learning model, sequentially perform processing of parameters of each layer in the initialized deep learning model, and output the sample video from an output side of the initialized deep learning model, where information output from the output side is a predicted fatigue driving occurrence time period of a sample driver corresponding to the sample video. The initial deep learning model can be an untrained deep learning model or an untrained deep learning model, and each layer of the initial deep learning model is provided with initialization parameters which can be continuously adjusted in the training process of the deep learning model.
Step 403, comparing the predicted fatigue driving time period of the sample driver corresponding to each sample video in the multiple sample videos with the fatigue driving occurrence time period of the sample driver corresponding to the sample video, so as to obtain the prediction accuracy of the initialized deep learning model.
In this embodiment, based on the predicted time period of occurrence of fatigue driving of the sample driver corresponding to each of the plurality of sample videos obtained in step 402, the electronic device may compare the predicted time period of occurrence of fatigue driving of the sample driver corresponding to each of the plurality of sample videos with the time period of occurrence of fatigue driving of the sample driver corresponding to the sample video, so as to obtain the prediction accuracy of the initialized deep learning model. Specifically, if the predicted time period of occurrence of fatigue driving of the sample driver corresponding to one sample video is the same as or similar to the time period of occurrence of fatigue driving of the sample driver corresponding to the sample video, the initialized deep learning model is predicted correctly; if the predicted time period of occurrence of fatigue driving of the sample driver corresponding to one sample video is different from or not similar to the time period of occurrence of fatigue driving of the sample driver corresponding to the sample video, the deep learning model is initialized to predict wrongly. Here, the electronic device may calculate a ratio of the number of prediction corrections to the total number of samples as a prediction accuracy of the initialized deep learning model.
At step 404, it is determined whether the prediction accuracy is greater than a preset accuracy threshold.
In this embodiment, based on the prediction accuracy of the initialized deep learning model obtained in step 403, the electronic device may compare the prediction accuracy of the initialized deep learning model with a preset accuracy threshold, and if the prediction accuracy is greater than the preset accuracy threshold, execute step 405; if not, step 406 is performed.
And step 405, taking the initialized deep learning model as a deep learning model after training.
In this embodiment, when the prediction accuracy of the initialized deep learning model is greater than the preset accuracy threshold, it indicates that the training of the deep learning model is completed, and at this time, the electronic device may use the initialized deep learning model as the trained deep learning model.
In step 406, parameters of the initialized deep learning model are adjusted.
In this embodiment, in the case that the prediction accuracy of the initialized deep learning model is not greater than the preset accuracy threshold, the electronic device may adjust the parameters of the initialized deep learning model, and return to the step 402 until a deep learning model that can represent and record the correspondence between the video of the current driving process of the driver and the predicted fatigue driving occurrence time period of the driver is trained.
With further reference to fig. 5, as an implementation of the methods shown in the above-mentioned figures, the present application provides an embodiment of an information output apparatus, which corresponds to the embodiment of the method shown in fig. 2, and which is particularly applicable to various electronic devices.
As shown in fig. 5, the information output apparatus 500 of the present embodiment may include: a target video acquisition unit 501, a fatigue driving occurrence period prediction unit 502, a fatigue driving occurrence period determination unit 503, and a fatigue driving occurrence period output unit 504. The target video acquiring unit 501 is configured to acquire a target video in which a current driving process of a target driver is recorded; the fatigue driving occurrence time period prediction unit 502 is configured to input a target video into a pre-trained deep learning model to obtain a predicted fatigue driving occurrence time period of a target driver, wherein the deep learning model is used for representing a corresponding relation between a video recorded with a current driving process of the driver and the predicted fatigue driving occurrence time period of the driver; a fatigue driving occurrence period determination unit 503 configured to determine whether the predicted fatigue driving occurrence period of the target driver is within a preset period range; a fatigue driving occurrence period output unit 504 configured to output the predicted fatigue driving occurrence period of the target driver in response to the determination that the preset period range is included.
In the present embodiment, in the information output apparatus 500: the specific processing of the target video obtaining unit 501, the fatigue driving occurring time period predicting unit 502, the fatigue driving occurring time period determining unit 503 and the fatigue driving occurring time period outputting unit 504 and the technical effects brought by the processing can refer to the relevant descriptions of step 201, step 202, step 203 and step 204 in the corresponding embodiment of fig. 2, which are not repeated herein.
In some optional implementations of this embodiment, the information output apparatus 500 may further include: and a prompt information acquisition unit (not shown in the figure) configured to acquire and output prompt information including a manner of alleviating fatigue driving.
In some optional implementations of the present embodiment, the deep learning model may include a convolutional neural network, a cyclic neural network, and a fully-connected layer.
In some optional implementations of the present embodiment, the fatigue driving occurrence period prediction unit 502 may include: an image feature vector obtaining subunit (not shown in the figure), configured to input the target video to a convolutional neural network, so as to obtain feature vectors of each frame of image of the target video, where the convolutional neural network is used to represent a corresponding relationship between the video and the feature vectors of each frame of image of the video; a video feature vector obtaining subunit (not shown in the figure), configured to input feature vectors of each frame image of the target video to a recurrent neural network, so as to obtain feature vectors of the target video, where the recurrent neural network is used to represent a corresponding relationship between the feature vectors of each frame image of the video and the feature vectors of the video, and the feature vectors of the video are used to represent an association relationship between the feature vectors of each frame image of the video; and a fatigue driving occurrence time period predicting subunit (not shown in the figure) configured to input the feature vector of the target video to a fully connected layer, so as to obtain the predicted fatigue driving occurrence time period of the target driver, wherein the fully connected layer is used for representing the corresponding relationship between the feature vector of the video and the predicted fatigue driving occurrence time period of the driver.
In some optional implementations of this embodiment, the information output apparatus 500 may further include a deep learning model training unit (not shown in the figure), and the deep learning model training unit may include: a sample acquiring subunit (not shown in the figure) configured to acquire a plurality of sample videos in which driving processes of sample drivers are recorded and a fatigue driving occurrence time period of the sample driver corresponding to each of the plurality of sample videos; and a deep learning model training subunit (not shown in the figure) configured to train to obtain the deep learning model by taking each of the plurality of sample videos as an input and taking a fatigue driving occurrence time period of a sample driver corresponding to each of the plurality of sample videos as an output.
In some optional implementations of the present embodiment, the deep learning model training subunit may include: a deep learning model training module (not shown in the figure) configured to perform the following training steps: sequentially inputting each sample video in the plurality of sample videos to the initialized deep learning model to obtain a predicted fatigue driving occurrence time period of a sample driver corresponding to each sample video in the plurality of sample videos, comparing the predicted fatigue driving occurrence time period of the sample driver corresponding to each sample video in the plurality of sample videos with the predicted fatigue driving occurrence time period of the sample driver corresponding to the sample video to obtain a predicted accuracy of the initialized deep learning model, determining whether the predicted accuracy is greater than a preset accuracy threshold, and if the predicted accuracy is greater than the preset accuracy threshold, taking the initialized deep learning model as the trained deep learning model.
In some optional implementations of this embodiment, the deep learning model training subunit may further include: and a parameter adjusting module (not shown in the figure) configured to adjust the parameters for initializing the deep learning model in response to the accuracy not greater than the preset accuracy threshold, and continue to execute the training step.
Referring now to FIG. 6, shown is a block diagram of a computer system 600 suitable for use in implementing the electronic device of an embodiment of the present application. The electronic device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program performs the above-described functions defined in the method of the present application when executed by a Central Processing Unit (CPU) 601. It should be noted that the computer readable medium described herein can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present application may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a target video acquisition unit, a fatigue driving occurrence period prediction unit, a fatigue driving occurrence period determination unit, and a fatigue driving occurrence period output unit. Here, the names of these units do not constitute a limitation of the unit itself in some cases, and for example, the target video acquiring unit may also be described as a "unit that acquires a target video in which the current driving course of the target driver is recorded".
As another aspect, the present application also provides a computer-readable medium, which may be contained in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a target video recorded with a current driving process of a target driver; inputting the target video into a pre-trained deep learning model to obtain the predicted fatigue driving occurrence time period of the target driver, wherein the deep learning model is used for representing and recording the corresponding relation between the video of the current driving process of the driver and the predicted fatigue driving occurrence time period of the driver; determining whether the predicted fatigue driving occurrence time period of the target driver is within a preset time period range; in response to determining that the predicted fatigue driving occurrence period of the target driver is within the preset period range, a predicted fatigue driving occurrence period of the target driver is output.
The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.

Claims (16)

1. An information output method comprising:
acquiring a target video recorded with a current driving process of a target driver;
inputting the target video into a pre-trained deep learning model to obtain the predicted fatigue driving occurrence time period of the target driver, wherein the deep learning model is used for representing the corresponding relation between the video recorded with the current driving process of the driver and the predicted fatigue driving occurrence time period of the driver;
determining whether the predicted fatigue driving occurrence time period of the target driver is within a preset time period range;
outputting the predicted fatigue driving occurrence period of the target driver in response to the determination that the predetermined period is within the preset period range.
2. The method of claim 1, wherein the method further comprises:
and acquiring and outputting prompt information including a mode of relieving fatigue driving.
3. The method of claim 1, wherein the deep learning model comprises a convolutional neural network, a cyclic neural network, and a fully-connected layer.
4. The method of claim 3, wherein the inputting the target video into a pre-trained deep learning model to obtain the predicted fatigue driving occurrence time period of the target driver comprises:
inputting the target video into the convolutional neural network to obtain the feature vectors of the frames of images of the target video, wherein the convolutional neural network is used for representing the corresponding relationship between the video and the feature vectors of the frames of images of the video;
inputting the feature vectors of the frames of images of the target video into the recurrent neural network to obtain the feature vectors of the target video, wherein the recurrent neural network is used for representing the corresponding relation between the feature vectors of the frames of images of the video and the feature vectors of the video, and the feature vectors of the video are used for representing the incidence relation between the feature vectors of the frames of images of the video;
and inputting the characteristic vector of the target video into the full-connection layer to obtain the predicted fatigue driving occurrence time period of the target driver, wherein the full-connection layer is used for representing the corresponding relation between the characteristic vector of the video and the predicted fatigue driving occurrence time period of the driver.
5. The method of claim 1, wherein the deep learning model is trained by:
the method comprises the steps of obtaining a plurality of sample videos recorded with driving processes of sample drivers and fatigue driving occurrence time periods of the sample drivers corresponding to each sample video in the plurality of sample videos;
and taking each sample video in the plurality of sample videos as input, taking the fatigue driving occurrence time period of the sample driver corresponding to each sample video in the plurality of sample videos as output, and training to obtain the deep learning model.
6. The method according to claim 5, wherein the training of the deep learning model by taking each of the plurality of sample videos as an input and taking a fatigue driving occurrence time period of the sample driver corresponding to each of the plurality of sample videos as an output comprises:
the following training steps are performed: sequentially inputting each sample video of the plurality of sample videos to an initialized deep learning model to obtain a predicted fatigue driving occurrence time period of the sample driver corresponding to each sample video of the plurality of sample videos, comparing the predicted fatigue driving time period of the sample driver corresponding to each sample video of the plurality of sample videos with the predicted fatigue driving occurrence time period of the sample driver corresponding to the sample video to obtain a predicted accuracy of the initialized deep learning model, determining whether the predicted accuracy is greater than a preset accuracy threshold, and if so, taking the initialized deep learning model as a trained deep learning model.
7. The method according to claim 6, wherein the training of the deep learning model by taking each of the plurality of sample videos as an input and taking a fatigue driving occurrence time period of the sample driver corresponding to each of the plurality of sample videos as an output further comprises:
and responding to the condition that the accuracy is not larger than the preset accuracy threshold, adjusting the parameters of the initialized deep learning model, and continuing to execute the training step.
8. An information output apparatus comprising:
the target video acquisition unit is configured for acquiring a target video recorded with a current driving process of a target driver;
the fatigue driving occurrence time period prediction unit is configured to input the target video into a pre-trained deep learning model to obtain a predicted fatigue driving occurrence time period of the target driver, wherein the deep learning model is used for representing a corresponding relation between a video recorded with a current driving process of the driver and the predicted fatigue driving occurrence time period of the driver;
a fatigue driving occurrence period determination unit configured to determine whether a predicted fatigue driving occurrence period of the target driver is within a preset period range;
a fatigue driving occurrence period output unit configured to output the predicted fatigue driving occurrence period of the target driver in response to a determination that the predicted fatigue driving occurrence period is within the preset period range.
9. The apparatus of claim 8, wherein the apparatus further comprises:
and the prompt information acquisition unit is configured to acquire and output prompt information including a mode of relieving fatigue driving.
10. The apparatus of claim 8, wherein the deep learning model comprises a convolutional neural network, a cyclic neural network, and a fully-connected layer.
11. The apparatus according to claim 10, wherein the fatigue driving occurrence period prediction unit includes:
the image feature vector acquisition subunit is configured to input the target video to the convolutional neural network to obtain feature vectors of each frame of image of the target video, wherein the convolutional neural network is used for representing a corresponding relationship between the video and the feature vectors of each frame of image of the video;
the video feature vector acquisition subunit is configured to input feature vectors of each frame of image of the target video to the recurrent neural network to obtain feature vectors of the target video, wherein the recurrent neural network is used for representing a corresponding relationship between the feature vectors of each frame of image of the video and the feature vectors of the video, and the feature vectors of the video are used for representing an association relationship between the feature vectors of each frame of image of the video;
and the fatigue driving occurrence time period predicting subunit is configured to input the feature vector of the target video to the fully-connected layer to obtain the predicted fatigue driving occurrence time period of the target driver, wherein the fully-connected layer is used for representing the corresponding relationship between the feature vector of the video and the predicted fatigue driving occurrence time period of the driver.
12. The apparatus of claim 8, wherein the apparatus further comprises a deep learning model training unit comprising:
the system comprises a sample acquisition subunit, a display unit and a control unit, wherein the sample acquisition subunit is configured to acquire a plurality of sample videos in which driving processes of sample drivers are recorded and fatigue driving occurrence time periods of the sample drivers corresponding to each of the plurality of sample videos;
and the deep learning model training subunit is configured to take each sample video of the plurality of sample videos as input, take a fatigue driving occurrence time period of the sample driver corresponding to each sample video of the plurality of sample videos as output, and train to obtain the deep learning model.
13. The apparatus of claim 12, wherein the deep learning model training subunit comprises:
a deep learning model training module configured to perform the following training steps: sequentially inputting each sample video of the plurality of sample videos to an initialized deep learning model to obtain a predicted fatigue driving occurrence time period of the sample driver corresponding to each sample video of the plurality of sample videos, comparing the predicted fatigue driving time period of the sample driver corresponding to each sample video of the plurality of sample videos with the predicted fatigue driving occurrence time period of the sample driver corresponding to the sample video to obtain a predicted accuracy of the initialized deep learning model, determining whether the predicted accuracy is greater than a preset accuracy threshold, and if so, taking the initialized deep learning model as a trained deep learning model.
14. The apparatus of claim 13, wherein the deep learning model training subunit further comprises:
and the parameter adjusting module is configured to respond to the condition that the accuracy is not greater than the preset accuracy threshold value, adjust the parameters of the initialized deep learning model and continue to execute the training step.
15. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
16. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when being executed by a processor, carries out the method according to any one of claims 1-7.
CN201711136211.8A 2017-11-16 2017-11-16 Information output method and device Active CN107909037B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711136211.8A CN107909037B (en) 2017-11-16 2017-11-16 Information output method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711136211.8A CN107909037B (en) 2017-11-16 2017-11-16 Information output method and device

Publications (2)

Publication Number Publication Date
CN107909037A CN107909037A (en) 2018-04-13
CN107909037B true CN107909037B (en) 2021-06-29

Family

ID=61845742

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711136211.8A Active CN107909037B (en) 2017-11-16 2017-11-16 Information output method and device

Country Status (1)

Country Link
CN (1) CN107909037B (en)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110473042B (en) * 2018-05-11 2022-02-01 北京京东尚科信息技术有限公司 Method and device for acquiring information
CN109145784B (en) * 2018-08-03 2022-06-03 百度在线网络技术（北京）有限公司 Method and apparatus for processing video
CN109493566B (en) * 2018-12-18 2020-09-08 长安大学 Fatigue driving prediction method based on GPS data
CN109770925B (en) * 2019-02-03 2020-04-24 闽江学院 Fatigue detection method based on deep space-time network
CN109927729B (en) * 2019-03-07 2020-06-23 南京微达电子科技有限公司 Method and device for estimating safe distance of continuous high-speed driving and evaluating and controlling stability
CN113071497B (en) * 2021-04-28 2022-05-24 中国第一汽车股份有限公司 Driving scene judging method, device, equipment and storage medium

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102717765A (en) * 2012-07-09 2012-10-10 湖南赛格导航技术研究有限公司 Fatigue driving detection method and anti-fatigue driving auxiliary device
CN102997900A (en) * 2011-09-15 2013-03-27 歌乐株式会社 Vehicle systems, devices, and methods for recognizing external worlds
CN103092160A (en) * 2012-12-27 2013-05-08 深圳市元征软件开发有限公司 Vehicle-mounted monitoring system with eye identification, vehicle-mounted monitoring method and vehicle-mounted terminal
CN103090825A (en) * 2012-12-10 2013-05-08 陕西西北铁道电子有限公司 Compatible railroad bridge condition monitoring system based on Beidou satellite
CN104183091A (en) * 2014-08-14 2014-12-03 苏州清研微视电子科技有限公司 System for adjusting sensitivity of fatigue driving early warning system in self-adaptive mode
CN104346926A (en) * 2013-07-31 2015-02-11 国际商业机器公司 Travelling time prediction method and apparatus, and correlative terminal device
CN104392610A (en) * 2014-12-19 2015-03-04 山东大学 Distributed video based highway traffic incident influence range dynamic monitoring method
CN104952209A (en) * 2015-04-30 2015-09-30 广州视声光电有限公司 Driving prewarning method and device
CN106407922A (en) * 2016-09-08 2017-02-15 哈尔滨工程大学 Online dictionary learning deformation model-based fatigue state recognition method
CN106940929A (en) * 2016-01-04 2017-07-11 杭州海康威视数字技术股份有限公司 Traffic data Forecasting Methodology and device
CN106941602A (en) * 2017-03-07 2017-07-11 中国铁道科学研究院 Trainman's Activity recognition method, apparatus and system

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8489284B2 (en) * 2008-08-21 2013-07-16 International Business Machines Corporation Automated dynamic vehicle blind spot determination
US9592828B2 (en) * 2015-04-13 2017-03-14 Nec Corporation Long term driving danger prediction system

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102997900A (en) * 2011-09-15 2013-03-27 歌乐株式会社 Vehicle systems, devices, and methods for recognizing external worlds
CN102717765A (en) * 2012-07-09 2012-10-10 湖南赛格导航技术研究有限公司 Fatigue driving detection method and anti-fatigue driving auxiliary device
CN103090825A (en) * 2012-12-10 2013-05-08 陕西西北铁道电子有限公司 Compatible railroad bridge condition monitoring system based on Beidou satellite
CN103092160A (en) * 2012-12-27 2013-05-08 深圳市元征软件开发有限公司 Vehicle-mounted monitoring system with eye identification, vehicle-mounted monitoring method and vehicle-mounted terminal
CN104346926A (en) * 2013-07-31 2015-02-11 国际商业机器公司 Travelling time prediction method and apparatus, and correlative terminal device
CN104183091A (en) * 2014-08-14 2014-12-03 苏州清研微视电子科技有限公司 System for adjusting sensitivity of fatigue driving early warning system in self-adaptive mode
CN104392610A (en) * 2014-12-19 2015-03-04 山东大学 Distributed video based highway traffic incident influence range dynamic monitoring method
CN104952209A (en) * 2015-04-30 2015-09-30 广州视声光电有限公司 Driving prewarning method and device
CN106940929A (en) * 2016-01-04 2017-07-11 杭州海康威视数字技术股份有限公司 Traffic data Forecasting Methodology and device
CN106407922A (en) * 2016-09-08 2017-02-15 哈尔滨工程大学 Online dictionary learning deformation model-based fatigue state recognition method
CN106941602A (en) * 2017-03-07 2017-07-11 中国铁道科学研究院 Trainman's Activity recognition method, apparatus and system

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
Dynamic driver fatigue detection using hidden Markov model in real driving condition;Rongrong Fu等;《Expert Systems With Applications》;20161231;第63卷;第397-411页 *
Evaluating Driving Fatigue Detection Algorithms Using Eye Tracking Glasses;Xiang-Yu Gao等;《7th Annual International IEEE EMBS Conference on Neural Engineering》;20150424;第767-770页 *
公交车辆检测及其行程时间预测;刘昱岗等;《中国公路学报》;20161031;第29卷(第10期);第95-104、125页 *
疲劳驾驶检测方法研究综述;薄纯娟等;《大连民族学院学报》;20130531;第15卷(第3期);第266-271页 *

Also Published As

Publication number Publication date
CN107909037A (en) 2018-04-13

Similar Documents

Publication Publication Date Title
CN107909037B (en) Information output method and device
Sharma et al. Student engagement detection using emotion analysis, eye tracking and head movement with machine learning
CN108520220B (en) Model generation method and device
Dewan et al. A deep learning approach to detecting engagement of online learners
US9889856B1 (en) Method and apparatus for controlling unmanned vehicle
CN107491166B (en) Method for adjusting parameters of virtual reality equipment and virtual reality equipment
DE102018102194A1 (en) Electronic equipment, information processing and program
CN108491808B (en) Method and device for acquiring information
Matran-Fernandez et al. Towards the automated localisation of targets in rapid image-sifting by collaborative brain-computer interfaces
CN108776786A (en) Method and apparatus for generating user's truth identification model
KR20190111278A (en) Electronic device and Method for controlling the electronic device thereof
CN109784277B (en) Emotion recognition method based on intelligent glasses
US11393252B2 (en) Emotion sensing artificial intelligence
US20200152328A1 (en) Cognitive analysis for identification of sensory issues
CN112307947A (en) Method and apparatus for generating information
CN113282840B (en) Comprehensive training acquisition management platform
WO2019235458A1 (en) Recalled image estimation device, recalled image estimation method, control program, and recording medium
Hibbard Virtual reality for vision science
CN117635383A (en) Virtual teacher and multi-person cooperative talent training system, method and equipment
CN110403604B (en) Method for constructing environment space and training attention based on attention concentration degree
CN107844777B (en) Method and apparatus for generating information
CN115937946A (en) Online learning state detection method based on multi-mode data fusion
CN116403135A (en) Video significance prediction method and system based on audio and video features
WO2023152584A1 (en) Meeting session control based on attention determination
EP4166079A1 (en) Conversation-based mental disorder screening method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant