WO2021095153A1 - Driver anomaly response system, driver anomaly response method, and program - Google Patents

Abstract

The present invention comprises: a first acquisition means (111) which acquires a normal situation image in which a driver is imaged in a normal situation; a first detection means (112) which analyzes the acquired normal situation image and detects a normal situation action; a learning means (113) which learns the detected normal situation action; a second acquisition means (114) which acquires an unknown image in which a driver is imaged in a situation in which it is unknown whether the situation is normal or anomalous; a second detection means (115) which analyzes the acquired unknown image and detects an unknown action; a determination means (116) which determines an anomaly level in accordance with the extent to which the detected unknown action differs from the learned result; a selection means (117) which selects a countermeasure in accordance with the determined anomaly level; and an execution means (118) which executes the selected countermeasure. Thus, it is possible to provide a driver anomaly response system, driver anomaly response method, and program which can determine an anomaly level of a driver while driving and carry out a countermeasure according to the anomaly level.

Description

Driver error response system, driver error response method, and program

The present invention is a driver abnormality response system and a driver abnormality response method capable of determining an abnormality level of a driver of a vehicle or the like, selecting a countermeasure according to the determined abnormality level, and executing the selected countermeasure. , And about the program.

In recent years, attention has been focused on technology for resolving traffic accidents and scandals caused by driver abnormalities while driving. For example, when the driver defines at least one predictive region in the range where the head is assumed to be located in the process of transitioning from the normal posture to the abnormal posture, and the head stays in the predictive region, the driver drives. A technique for determining that a person is in an abnormal posture has been proposed (Patent Document 1).

Japanese Unexamined Patent Publication No. 2019-105872

In order to resolve traffic accidents and scandals caused by driver abnormalities while driving, it is important to implement countermeasures according to the degree of driver abnormality (abnormality level). However, although Patent Document 1 describes a deceleration stop type abnormality response system having a lane departure prevention function and an off-road deviation prevention function, it determines the abnormality level of the driver during driving and sets the abnormality level. There is a problem that it is not possible to implement the corresponding countermeasures.

For this problem, the present inventor can determine the abnormality level according to the degree of the difference when the driver's driving method is different from the learning result if the driver's driving method in the normal state is learned. Since it is possible to do so, we focused on the fact that it is possible to implement countermeasures according to the determined abnormality level.

The present invention determines an abnormality level when the driver's driving method differs from the learning result by learning the normal driving method of a driver such as a vehicle, and takes countermeasures according to the determined abnormality level. It is an object of the present invention to provide a driver abnormality response system, a driver abnormality response method, and a program capable of selecting and executing the selected countermeasure.

The present invention provides the following solutions.

The invention according to the first feature is
It is a driver abnormality response system that determines the abnormality level of the driver while driving and executes the selected countermeasure.
The first acquisition means for acquiring a normal image captured by a normal driver, and
The first detection means for analyzing the acquired normal image and detecting the normal operation,
The learning means for learning the detected normal operation and
A second acquisition means for acquiring an unknown image captured by an unknown driver, whether it is normal or abnormal, and
A second detection means that analyzes the acquired unknown image and detects an unknown operation,
A determination means for determining an abnormality level according to the degree to which the detected unknown motion differs from the learned result.
A selection means for selecting a countermeasure according to the determined abnormality level, and
An execution means for executing the selected countermeasure and
Provide a driver abnormality response system equipped with.

According to the invention according to the first feature, in the driver abnormality response system that determines the abnormality level of the driver during driving and executes the selected countermeasure, the normal driver is captured and the normal image is acquired. The first acquisition means for detecting the normal operation by analyzing the acquired normal image, the learning means for learning the detected normal operation, and whether it is normal. A second acquisition means for acquiring an unknown image captured by an abnormal or unknown driver, a second detection means for analyzing the acquired unknown image to detect an unknown operation, and the detected unknown. A determination means for determining an abnormality level according to the degree to which the operation differs from the learned result, a selection means for selecting a countermeasure according to the determined abnormality level, and the selected countermeasure. It is provided with an execution means to be executed.

The invention according to the first feature is in the category of driver abnormality response system, but the same action and effect can be obtained even with a driver abnormality response method and a program.

The invention according to the second feature is the driver abnormality response system which is the invention according to the first feature.
In the first acquisition means, the biological information of the driver at the normal time is further acquired, and the biological information is acquired.
In the first detection means, further normal biometric information is detected.
In the learning means, further learning the biological information at the time of normal,
In the second acquisition means, the biological information of the driver, which is unknown whether it is normal or abnormal, is further acquired.
In the second detection means, it is further detected whether the driver's biological information is normal or abnormal, and the driver's biological information is detected.
The determination means provides a driver abnormality response system that determines an abnormality level in consideration of the degree of whether the driver's biological information is normal or abnormal.

According to the invention according to the second feature, in the driver abnormality response system according to the first feature, the first acquisition means further acquires the biological information of the driver at the normal time, and the first detection means. In, the normal biometric information is further detected, the normal biometric information is further learned by the learning means, and the biometric information of the driver whose normal state or abnormal state is unknown is further learned by the second acquisition means. The second detection means further detects whether the driver's biological information is normal or abnormal, and the determination means determines that the abnormal level of the driver's biological information is normal. The judgment is made in consideration of the degree of whether it is abnormal or abnormal.

The invention according to the third feature is a driver abnormality response system which is an invention according to the first feature or the second feature.
In the first acquisition means, the normal time image captured by the normal time driver is acquired in chronological order, and the normal time image is acquired.
In the first detection means, the acquired time-series normal time image is analyzed to detect the normal time operation, and the normal time operation is detected.
In the learning means, the normal operation of the detected time series is learned, and
The determination means provides a driver abnormality response system that determines an abnormality level according to the degree to which the detected unknown operation differs from the learned time-series result.

According to the invention according to the third feature, in the driver abnormality response system which is the invention according to the first feature or the second feature, the normal driver is imaged in chronological order by the first acquisition means. The time image is acquired, the first detection means analyzes the acquired normal time image of the time series to detect the normal operation, and the learning means detects the normal operation of the detected time series. In the determination means, when determining the abnormality level, the determination is performed according to the degree to which the detected unknown operation differs from the result of the learned time series.

The invention according to the fourth feature is a driver abnormality response system which is an invention according to any one of the first to third features.
In the first acquisition means, the location information during normal operation is further acquired, and the location information is acquired.
In the learning means, further learning including location information during normal driving is performed.
In the second acquisition means, the location information during operation, which is unknown whether it is normal or abnormal, is further acquired.
The determination means provides a driver abnormality response system that makes a determination by adding the location information to the unknown operation when determining an abnormality level.

According to the invention relating to the fourth feature, in the driver abnormality response system which is the invention according to any one of the first feature to the third feature, the first acquisition means further provides location information during normal operation. Is acquired, and the learning means further learns including the location information during normal driving, and the second acquisition means further obtains location information during driving that is unknown whether it is normal or abnormal. When the abnormality level is determined by the determination means, the determination is performed by adding the location information to the unknown operation.

The invention according to the fifth feature is a driver abnormality response system which is an invention according to any one of the first to fourth features.
The determination means provides a driver abnormality response system that determines an abnormality level according to the number of times or time that the detected unknown operation differs from the learned result.

According to the invention according to the fifth feature, in the driver abnormality response system which is the invention according to any one of the first to fourth features, the detected unknown operation is learned by the determination means. The abnormal level is determined according to the number of times or time different from the result.

The invention according to the sixth feature is a driver abnormality response system which is an invention according to any one of the first to fifth features.
In the first detection means, the normal image is analyzed to identify the driver, and the normal operation of the driver is detected.
In the learning means, the detected normal operation is learned for each of the specified drivers.
In the second detection means, the acquired unknown image is analyzed to identify the driver, and the unknown operation is detected.
The determination means provides a driver abnormality response system that determines an abnormality level according to the degree to which the detected unknown operation of the identified driver differs from the learned result of the identified driver.

According to the invention relating to the sixth feature, in the driver abnormality response system according to any one of the first to fifth features, the first detection means analyzes a normal image to obtain a driver. The normal operation of the driver is specified, the normal operation of the driver is detected, the detected normal operation is learned for each of the specified drivers in the learning means, and the acquired unknown image is analyzed by the second detection means. The driver is identified, the unknown motion is detected, and in the determination means, the abnormal level is increased according to the degree to which the detected unknown motion of the identified driver is different from the learned result of the identified driver. To judge.

The invention according to the seventh feature is a driver abnormality response system which is an invention according to any one of the first to fifth features.
In the first acquisition means, the normal driver images captured by the normal driver are acquired for a plurality of people, and the images are acquired for a plurality of people.
The determination means provides a driver abnormality response system that determines an abnormality level according to the degree to which the detected unknown operation differs from the average value or the median value of the learned results.

According to the invention relating to the seventh feature, in the driver abnormality response system according to any one of the first to fifth features, the normal driver in which the normal driver is imaged is captured by the first acquisition means. Time images are acquired for a plurality of people, and the determination means determines the abnormality level according to the degree to which the detected unknown motion differs from the average value or the median value of the learned results.

The invention according to the eighth feature is
To the driver abnormality response system that determines the abnormality level of the driver while driving and executes the selected countermeasure.
The step of acquiring the normal image captured by the normal driver, and
A step of analyzing the acquired normal image to detect normal operation, and
The step of learning the detected normal operation and
The step of acquiring an unknown image captured by an unknown driver, whether it is normal or abnormal, and
A step of analyzing the acquired unknown image to detect an unknown operation,
A step of determining an abnormality level according to the degree to which the detected unknown motion differs from the learned result.
A step of selecting a countermeasure according to the determined abnormality level, and
The steps to implement the selected workaround and
Provide a method for dealing with driver abnormalities.

The invention according to the ninth feature is
To the driver abnormality response system that determines the abnormality level of the driver while driving and executes the selected countermeasure.
Steps to acquire a normal image captured by a normal driver,
A step of analyzing the acquired normal image to detect normal operation,
The step of learning the detected normal operation,
Steps to acquire an unknown image taken by an unknown driver, whether it is normal or abnormal,
The step of analyzing the acquired unknown image to detect an unknown operation,
A step of determining an abnormality level according to the degree to which the detected unknown motion differs from the learned result.
A step of selecting a countermeasure according to the determined abnormality level,
Steps to implement the selected workaround,
Provide a program to execute.

According to the present invention, by learning how to drive a driver of a vehicle or the like in a normal state, an abnormality level is determined when the driving method of the driver is different from the learning result, and a response is made according to the determined abnormality level. It is possible to provide a driver abnormality response system, a driver abnormality response method, and a program capable of selecting a countermeasure and executing the selected countermeasure.

FIG. 1 is a schematic view of a preferred embodiment of the present invention. FIG. 2 is a diagram showing the relationship between the functional blocks of the computer 100 and the imaging device 200 and each function. FIG. 3 is a flowchart of the driver abnormality handling process performed by the driver abnormality handling system. FIG. 4 is a diagram showing the relationship between the functional blocks of the computer 100, the imaging device 200, and the biological information acquisition device 300 and each function when the biological information is also acquired. FIG. 5 is a flowchart of a driver abnormality handling process when determining an abnormality level with the same driver. FIG. 6 is an example of a normal image. FIG. 7 is an example of an unknown image. FIG. 8 is an example showing an abnormal operation corresponding to an abnormal level and a countermeasure. FIG. 9 is an example showing the number of times and the time for determining the abnormality level.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings. It should be noted that this is only an example, and the technical scope of the present invention is not limited to this.

[Overview of driver error response system]
FIG. 1 is a schematic view of a preferred embodiment of the present invention. The outline of the present invention will be described with reference to FIG. The driver abnormality response system includes a computer 100, an image pickup device 200, and a communication network 400.

Note that, in FIG. 1, the number of image pickup devices 200 is not limited to one, and may be plural. Further, the computer 100 is not limited to an existing device, but may be a virtual device, a cloud service, or the like.

As shown in FIG. 2, the computer 100 includes a control unit 110, a communication unit 120, a storage unit 130, an input unit 140, and an output unit 150. The control unit 110 includes a first acquisition means 111, a first detection means 112, a learning means 113, a second acquisition means 114, a second detection means 115, a determination means 116, a selection means 117, and an execution means 118. The control unit 110 cooperates with the communication unit 120, the storage unit 130, the input unit 140, and the output unit 150 to realize the functions of the above means, if necessary. Further, as shown in FIG. 2, the image pickup apparatus 200 includes an image pickup unit 20, a control unit 210, a communication unit 220, a storage unit 230, an input unit 240, and an output unit 250. The communication network 400 is a network that enables communication between the computer 100 and the image pickup apparatus 200.

The computer 100 is a calculation device capable of data communication with the image pickup device 200. Although the illustration is based on the premise of an in-vehicle device, it may be a mobile terminal such as a smartphone, a notebook computer, a wearable device, or the like, and may be a virtual device as well as an existing device. It may also be a service such as a cloud.

The image pickup device 200 is an image pickup device provided with an image pickup device such as an image pickup device and a lens capable of data communication with the computer 100, and can take an image of a driver in operation. Although the illustration is based on the assumption of an in-vehicle camera, it may be an image pickup device having necessary functions such as a digital camera, a digital video, a camera of a mobile terminal, a laptop computer, or a wearable device. Further, the captured image may be stored in the storage unit 230. The image to be captured shall have the resolution required for analysis, and is basically a moving image, but may be a still image.

In the driver abnormality response system of FIG. 1, first, the first acquisition means 111 acquires a normal image captured by the normal driver (step S101). The acquisition here may be performed directly from the image pickup device 200, may be performed from a database that collects normal images, an external server, or the like, or may be performed from a digital camera, a video camera, or the like other than the image pickup device 200. Alternatively, it may be performed from a storage medium such as a CD-ROM, a DVD, or a USB memory. If it is a moving image, audio data may be acquired at the same time.

FIG. 6 is an example of a normal image. The normal image shall have a resolution and an imaging range in which the driver's normal movement of the face and hands, gestures, posture, line of sight, facial expression, etc. can be confirmed. The normal image is basically a moving image, but may be a still image.

Returning to FIG. 1, the first detection means 112 of the computer 100 analyzes the normal image acquired in step S101 and detects the normal operation (step S102). When there are a plurality of acquired images, normal operation is detected for each of the images. The normal operation here is defined as the normal movement of the driver's face and hands, gestures, posture, line of sight, facial expression, and the like.

Next, the learning means 113 learns the normal operation detected in step step S102 (step S103). Since the movements, gestures, postures, eyes, facial expressions, etc. of the face and hands detected in step S102 are normal, what range each of them falls within, how often and how many times are performed. Learn what kind of time it will last, and so on. If the normal image is insufficient for learning the normal operation, step S101 and step S102 are repeated. On the contrary, if the learning is sufficiently performed, steps S101 to S103 may be omitted.

Next, the second acquisition means 114 acquires an unknown image captured by an unknown driver, whether it is normal or abnormal (step S104). As the timing for starting the acquisition of step S104, the timing when the driver starts driving can be considered. In order to know when the driver starts driving, the computer 100 may be notified when the imaging device 200 starts imaging.

FIG. 7 is an example of an unknown image. The unknown image shall have a resolution and an imaging range in which the movement, gesture, posture, line of sight, facial expression, etc. of the driver's face and hands can be confirmed. The unknown image is basically a moving image, but may be a still image.

Returning to FIG. 1, the second detection means 115 of the computer 100 analyzes the acquired unknown image and detects an unknown operation (step S105). The unknown motion here is assumed to be the movement of the face or hand, the gesture, the posture, the line of sight, the facial expression, etc. of the driver of the acquired unknown image.

Next, the determination means 116 of the computer 100 determines the abnormality level according to the degree to which the unknown operation detected in step S105 is different from the result learned in step S103 (step S106). Here, we compare the normal movements in which the detected unknown movements are learned and how much they differ from each index such as the driver's face and hand movements, gestures, postures, eyes, and facial expressions. Determine the abnormal level according to the degree. Judgment of how much is different can be made, for example, according to the number of times and time different from the result of the learned normal movement for each index (face and hand movement, gesture, posture, line of sight, facial expression, etc.). It is possible. In addition, the number of levels of abnormality can be set according to the system. In FIG. 1, it is assumed that the abnormality level is determined to be level 2 inattentive driving.

FIG. 8 is an example showing an abnormal operation corresponding to an abnormal level and a countermeasure. In the example of FIG. 8, the abnormality level is set to 5 levels from level 1 to level 5. The abnormal behavior of level 1 is "more times of rubbing the nose than usual (cold / allergy, etc.)". In this case, the determination can be made from "the movement of the driver's hand is more likely to come near the nose than in the normal state", "there are more gestures to rub the nose than in the normal state", and the like. The abnormal operation of level 2 is "inattentive driving". In this case, the judgment is made from "the driver's posture is tilted compared to normal", "the driver's line of sight is not facing forward compared to normal", "the driver's facial expression cannot be confirmed compared to normal", etc. be able to. The abnormal behavior of level 3 is "screaming". In this case, "the driver's face is moving his mouth more violently than in normal times", "the driver is waving his hands more than in normal times", "the facial expression is full of anger compared to normal times", etc. Judgment can be made. The abnormal behavior of level 4 is "consciousness is jumping (sudden illness / lack of sleep, etc.)". In this case, "the driver's hand is not driving compared to normal", "the driver's posture is tilted compared to normal", and "the driver's head and body are regularly tilted compared to normal". Judgment can be made from "Yes", "The driver's line of sight cannot be confirmed compared to normal (eyes closed / frame out)", "The driver's facial expression remains closed compared to normal", etc. .. Abnormal behavior at level 5 is "unconscious". In this case, the determination can be made from the fact that the level 4 state continues for a long time. The abnormal leveling stages and examples of each abnormal operation described here are merely examples, and there may be more or less examples of abnormal leveling stages and abnormal operations. Countermeasures corresponding to the abnormality level in FIG. 8 will be described later.

FIG. 9 is an example showing the number of times and the time for determining the abnormality level. In the example of FIG. 9, the abnormality level is set to 5 levels from level 1 to level 5 as in the example of FIG. Level 1 abnormal behavior In order to judge "more times of rubbing the nose than usual (cold / allergy, etc.)", "the movement of the driver's hand is more likely to come near the nose than in normal times", " For indicators such as "more gestures of rubbing the nose than in normal times", if the number of unknown movements is "5 times or more" and the time is "total 3 minutes or more", it is judged to be abnormal level 1. .. In order to judge the abnormal operation of level 2 "inattentive driving", "the driver's posture is tilted compared to normal", "the driver's line of sight is not facing forward compared to normal", "compared to normal" For an index such as "the driver's facial expression cannot be confirmed", if the number of unknown actions is "3 times or more" and the time is "cumulative 3 minutes or more", it is determined to be abnormal level 2. In order to judge the abnormal behavior of level 3 "screaming", "the driver's face is moving his mouth more violently than in normal times", "the driver is waving his hands more than in normal times", " For indicators such as "the facial expression is full of anger compared to normal times", if the number of unknown actions is "2 times or more" and the time is "1 minute or more in total", it is judged to be abnormal level 3. .. Level 4 abnormal behavior In order to judge "consciousness is jumping (sudden illness / lack of sleep, etc.)", "the driver's hand is not driving compared to normal" and "driver's posture compared to normal" Is tilted "," The driver's head and body are tilted regularly compared to normal times "," The driver's line of sight cannot be confirmed compared to normal times (eyes closed / frame out) "," Normal For indicators such as "the driver's facial expression remains closed compared to the time", if the number of unknown actions is "1 time or more" and the time is "1 minute or more in total", it is judged as abnormal level 4. And. In order to judge the abnormal operation "unconsciousness" of level 5, the state of level 4 has continued for a long time, that is, the state is further 2 minutes or more, that is, "total 3 minutes" as the time from the state judged to be level 4. In the case of "above", it is determined that the abnormality level is 5. Here, when determining the above-mentioned abnormal level, it is possible to set depending on the system whether to determine by "when the condition of the number of times and time is satisfied" or "when the condition of the number of times or time is satisfied". Suppose there is.

Returning to FIG. 1, the selection means 117 of the computer 100 selects a countermeasure according to the abnormality level determined in step S106 (step S107). As the countermeasures here, for example, a notification system for notifying and a vehicle control system for controlling the driver's vehicle can be considered, and not one countermeasure but a plurality of countermeasures may be taken at the same time. To do. In FIG. 1, it is assumed that "notify the driver" and "notify the administrator" are selected as the notification system and "speed down" is selected as the vehicle control system as countermeasures against "abnormal level 2: inattentive driving". An example of countermeasures corresponding to the abnormality level will be described with reference to FIG. In the example of FIG. 8, as a countermeasure in the case of level 1 "more times of rubbing the nose than usual (cold / allergy, etc.)", "notify the driver (warning / follow-up)" is used as the notification system. Shown. For example, it is possible to call attention by notifying the driver by means of voice, sound, light, image or character display, vibration, or the like from the vehicle. After that, as follow-up observation, the level 1 may be determined again with a smaller number of times and time than the normal index shown in FIG. 9, or the abnormal level may be raised if the same state continues. .. As a countermeasure in the case of level 2 "inattentive driving", "notify the driver" and "notify the administrator" are shown as the notification system, and "speed down" is shown as the vehicle control system. The notification system "notify the driver" is as described above, and the "notify the administrator" notifies the administrator registered in advance by e-mail, telephone, SNS (social networking service), or the like. As a result, for example, in the case of a company car, the convenience can be improved such that the manager of the company can grasp the situation, and the safety awareness of the driver can be improved. Further, by directly controlling the vehicle as a vehicle control system to perform "speed down", it is possible to reduce the probability of an accident and reduce the damage in the event of an accident. As a countermeasure in the case of level 3 "screaming", "notify the administrator" is shown as a notification system, and "speed down" and "speed limit" are shown as a vehicle control system. The correspondence between the notification system "notify the administrator" and the vehicle control system "speed down" is as described above. The "speed limit" of the vehicle control system is, for example, the possibility of driving by the driver or an accident by limiting the upper limit speed at which the vehicle can be directly controlled for a certain period of time after the level 2 judgment. I think it can be lowered. As a countermeasure in the case of level 4 "consciousness is jumping (sudden illness / lack of sleep, etc.)", "notification to surrounding people" is shown as a notification system, and "sudden stop" is shown as a vehicle control system. .. The notification system "notification to surrounding people" is, for example, from the vehicle to the inside and outside of the vehicle by using means such as voice, sound, light, image and character display to the passenger and people around the vehicle. By giving the notification, the effect of having the driver of the vehicle work can be expected. Further, the "sudden stop" of the vehicle control system is to prevent the occurrence of an accident by promptly stopping at a safe place, not by suddenly braking. At that time, the hazard lamp or the like may be turned on at the same time. As a countermeasure in the case of "unconsciousness" of level 5, in addition to the countermeasure of level 4, "report to an emergency agency" is shown as a notification system. As the "report to the emergency agency", for example, a public institution such as a fire department may be notified by calling 119, etc., to the effect that the vehicle position information and the driver are unconscious, or depending on the type of vehicle. A similar report may be made to the corresponding emergency contact. In FIG. 8, since it is an example of one abnormal operation for each abnormal level, countermeasures are shown for each level, but when there are multiple abnormal operations at the same level, countermeasures are set according to the abnormal operation. You can do it.

Returning to FIG. 1, finally, the execution means 118 of the computer 100 executes the countermeasure selected in step S107 (step S108). In FIG. 1, it is assumed that "notify the driver" and "notify the administrator" are executed as the notification system, and "speed down" is executed as the vehicle control system.

As described above, according to the present invention, by learning how to drive a driver such as a vehicle in a normal state, an abnormality level is determined when the driving method of the driver is different from the learning result, and the determined abnormality level is determined. It is possible to provide a driver abnormality response system, a driver abnormality response method, and a program capable of selecting a countermeasure according to the situation and executing the selected countermeasure.

[Explanation of each function]
FIG. 2 is a diagram showing the relationship between the functional blocks of the computer 100 and the imaging device 200 and each function. The computer 100 includes a control unit 110, a communication unit 120, a storage unit 130, an input unit 140, and an output unit 150. The control unit 110 includes a first acquisition means 111, a first detection means 112, a learning means 113, a second acquisition means 114, a second detection means 115, a determination means 116, a selection means 117, and an execution means 118. The control unit 110 cooperates with the communication unit 120, the storage unit 130, the input unit 140, and the output unit 150 to realize the functions of the above means, if necessary. The image pickup device 200 includes an image pickup unit 20, a control unit 210, a communication unit 220, a storage unit 230, an input unit 240, and an output unit 250. The communication network 400 is a network that enables communication between the computer 100 and the image pickup apparatus 200.

The computer 100 is a calculation device capable of data communication with the image pickup device 200. Although the illustration is based on the premise of an in-vehicle device, it may be a mobile terminal such as a smartphone, a notebook computer, a wearable device, or the like, and may be a virtual device as well as an existing device. It may also be a service such as a cloud.

The control unit 110 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The control unit 110 cooperates with the communication unit 120, the storage unit 130, the input unit 140, and the output unit 150 to realize the functions of the above means, if necessary.

The communication unit 120 is provided with a device for enabling communication with other devices. The communication may be a wired connection or a wireless connection. Further, it is assumed that communication with an external server or database is performed via the communication unit 120 in order to acquire a normal image as needed.

The storage unit 130 includes a data storage unit using a hard disk or a semiconductor memory, and stores data necessary for processing such as a normal image, a normal operation, a learning result, an unknown image, an abnormality level, a judgment result, and a countermeasure. .. The storage unit 130 may be an external server, a database, a cloud service, or the like.

The input unit 140 shall be provided with the functions necessary for managing or operating the driver abnormality response system. As an example for realizing input, it is possible to provide a keyboard, a mouse, a pen tablet, a touch panel, a microphone for voice recognition, and the like. It may be possible to input via a network. The function of the present invention is not particularly limited depending on the input method.

The output unit 150 shall be provided with the functions necessary for managing or operating the driver abnormality response system. As an example for realizing the output, a display, a display such as a projection on a projector, and a form such as an audio output can be considered. It may be possible to output via a network. The function of the present invention is not particularly limited depending on the output method.

The image pickup device 200 is an image pickup device provided with an image pickup device such as an image pickup device and a lens capable of data communication with the computer 100, and captures an image of a driver during operation. It is possible to image the driver while driving. Although the illustration is based on the assumption of an in-vehicle camera, it may be an image pickup device having necessary functions such as a digital camera, a digital video, a camera of a mobile terminal, a laptop computer, or a wearable device. Further, the captured image may be stored in the storage unit 230. The image to be captured shall have the resolution required for analysis, and is basically a moving image, but may be a still image.

The image pickup device 200 includes an image pickup device such as a lens, an image pickup element, and the like as an image pickup unit 20, and captures an image as an image pickup image such as a moving image or a still image. Further, the image obtained by imaging is a precise image having a resolution necessary for image analysis.

The control unit 210 includes a CPU, RAM, ROM, and the like.

The communication unit 220 is provided with a device for enabling communication with other devices. The communication may be a wired connection or a wireless connection.

The storage unit 230 is provided with a data storage unit using a hard disk or a semiconductor memory, and stores necessary data such as captured images. Information on the date and time when the image was taken and the location may also be stored. The storage unit 230 may be an external server, a database, a cloud service, or the like.

The input unit 240 is provided with functions necessary for operating the image pickup apparatus 200. As an example for realizing input, it is possible to provide a hardware button, a touch panel, a microphone, and the like. It may be possible to input via a network.

The output unit 250 shall have functions necessary for operating the image pickup apparatus 200. As an example for realizing the output, a form such as a display on a display or an audio output can be considered. It may be possible to output via a network.

[Driver error handling process]
FIG. 3 is a flowchart of the driver abnormality handling process performed by the driver abnormality handling system. The processing executed by each of the above-mentioned means will be described with reference to this flowchart.

First, the first acquisition means 111 of the computer 100 acquires a normal image captured by the normal driver (step S301). The acquisition here may be performed directly from the image pickup device 200, may be performed from a database that collects normal images, an external server, or the like, or may be performed from a digital camera, a video camera, or the like other than the image pickup device 200. Alternatively, it may be performed from a storage medium such as a CD-ROM, a DVD, or a USB memory. If it is a moving image, audio data may be acquired at the same time.

FIG. 6 is an example of a normal image. The normal image shall have a resolution and an imaging range in which the driver's normal movement of the face and hands, gestures, posture, line of sight, facial expression, etc. can be confirmed. The normal image is basically a moving image, but may be a still image.

Returning to FIG. 3, the first detection means 112 of the computer 100 analyzes the normal image acquired in step S301 and detects the normal operation (step S302). When there are a plurality of acquired images, normal operation is detected for each of the images. The normal operation here is defined as the normal movement of the driver's face and hands, gestures, posture, line of sight, facial expression, and the like.

Next, the learning means 113 learns the normal operation detected in step step S302 (step S303). Since the movements, gestures, postures, eyes, facial expressions, etc. of the face and hands detected in step S302 are normal, what range each of them falls within, how often and how many times are performed. Learn what kind of time it will last, and so on. If the normal image is insufficient for learning the normal operation, step S301 and step S302 are repeated. On the contrary, if the learning is sufficiently performed, steps S301 to S303 may be omitted.

The imaging device 200 starts imaging at the timing when the driver starts driving (step S304). The imaging device 200 notifies the computer 100 of the start of imaging at the same time as the start of imaging.

Upon receiving the notification from the imaging device 200 of the start of imaging, the second acquisition means 114 of the computer 100 acquires an unknown image captured by an unknown driver, whether it is normal or abnormal, from the imaging device 200 ( Step S305).

FIG. 7 is an example of an unknown image. The unknown image shall have a resolution and an imaging range in which the movement, gesture, posture, line of sight, facial expression, etc. of the driver's face and hands can be confirmed. The unknown image is basically a moving image, but may be a still image.

Returning to FIG. 3, the second detection means 115 of the computer 100 analyzes the acquired unknown image to detect the unknown operation (step S306). The unknown motion here is assumed to be the movement of the face or hand, the gesture, the posture, the line of sight, the facial expression, etc. of the driver of the acquired unknown image.

Next, the determination means 116 of the computer 100 determines the abnormality level according to the degree to which the unknown operation detected in step S306 is different from the result learned in step S303 (step S307). Here, we compare the normal movements in which the detected unknown movements are learned and how much they differ from each index such as the driver's face and hand movements, gestures, postures, eyes, and facial expressions. Determine the abnormal level according to the degree. Judgment of how much is different can be made, for example, according to the number of times and time different from the result of the learned normal movement for each index (face and hand movement, gesture, posture, line of sight, facial expression, etc.). It is possible. In addition, the number of levels of abnormality can be set according to the system. In the flowchart of FIG. 3, it is assumed that the abnormality level is determined to be level 2 inattentive operation.

FIG. 8 is an example showing an abnormal operation corresponding to an abnormal level and a countermeasure. In the example of FIG. 8, the abnormality level is set to 5 levels from level 1 to level 5. The abnormal behavior of level 1 is "more times of rubbing the nose than usual (cold / allergy, etc.)". In this case, the determination can be made from "the movement of the driver's hand is more likely to come near the nose than in the normal state", "there are more gestures to rub the nose than in the normal state", and the like. The abnormal operation of level 2 is "inattentive driving". In this case, the judgment is made from "the driver's posture is tilted compared to normal", "the driver's line of sight is not facing forward compared to normal", "the driver's facial expression cannot be confirmed compared to normal", etc. be able to. The abnormal behavior of level 3 is "screaming". In this case, "the driver's face is moving his mouth more violently than in normal times", "the driver is waving his hands more than in normal times", "the facial expression is full of anger compared to normal times", etc. Judgment can be made. The abnormal behavior of level 4 is "consciousness is jumping (sudden illness / lack of sleep, etc.)". In this case, "the driver's hand is not driving compared to normal", "the driver's posture is tilted compared to normal", and "the driver's head and body are regularly tilted compared to normal". Judgment can be made from "Yes", "The driver's line of sight cannot be confirmed compared to normal (eyes closed / frame out)", "The driver's facial expression remains closed compared to normal", etc. .. Abnormal behavior at level 5 is "unconscious". In this case, the determination can be made from the fact that the level 4 state continues for a long time. The abnormal leveling stages and examples of each abnormal operation described here are merely examples, and there may be more or less examples of abnormal leveling stages and abnormal operations.

FIG. 9 is an example showing the number of times and the time for determining the abnormality level. In the example of FIG. 9, the abnormality level is set to 5 levels from level 1 to level 5 as in the example of FIG. Level 1 abnormal behavior In order to judge "more times of rubbing the nose than usual (cold / allergy, etc.)", "the movement of the driver's hand is more likely to come near the nose than in normal times", " For indicators such as "more gestures of rubbing the nose than in normal times", if the number of unknown movements is "5 times or more" and the time is "total 3 minutes or more", it is judged to be abnormal level 1. .. In order to judge the abnormal operation of level 2 "inattentive driving", "the driver's posture is tilted compared to normal", "the driver's line of sight is not facing forward compared to normal", "compared to normal" For an index such as "the driver's facial expression cannot be confirmed", if the number of unknown actions is "3 times or more" and the time is "cumulative 3 minutes or more", it is determined to be abnormal level 2. In order to judge the abnormal behavior of level 3 "screaming", "the driver's face is moving his mouth more violently than in normal times", "the driver is waving his hands more than in normal times", " For indicators such as "the facial expression is full of anger compared to normal times", if the number of unknown actions is "2 times or more" and the time is "1 minute or more in total", it is judged to be abnormal level 3. .. Level 4 abnormal behavior In order to judge "consciousness is jumping (sudden illness / lack of sleep, etc.)", "the driver's hand is not driving compared to normal" and "driver's posture compared to normal" Is tilted "," The driver's head and body are tilted regularly compared to normal times "," The driver's line of sight cannot be confirmed compared to normal times (eyes closed / frame out) "," Normal For indicators such as "the driver's facial expression remains closed compared to the time", if the number of unknown actions is "1 time or more" and the time is "1 minute or more in total", it is judged as abnormal level 4. And. In order to judge the abnormal operation "unconsciousness" of level 5, the state of level 4 has continued for a long time, that is, the state is further 2 minutes or more, that is, "total 3 minutes" as the time from the state judged to be level 4. In the case of "above", it is determined that the abnormality level is 5. Here, when determining the above-mentioned abnormal level, it is possible to set depending on the system whether to determine by "when the condition of the number of times and time is satisfied" or "when the condition of the number of times or time is satisfied". Suppose there is.

Returning to FIG. 3, the selection means 117 of the computer 100 selects a countermeasure according to the abnormality level determined in step S307 (step S308). As the countermeasures here, for example, a notification system for notifying and a vehicle control system for controlling the driver's vehicle can be considered, and not one countermeasure but a plurality of countermeasures may be taken at the same time. To do. In the flowchart of FIG. 3, it is assumed that "notify the driver" and "notify the administrator" are selected as the notification system and "speed down" is selected as the vehicle control system as countermeasures against "abnormal level 2: inattentive driving". .. An example of countermeasures corresponding to the abnormality level will be described with reference to FIG. In the example of FIG. 8, as a countermeasure in the case of level 1 "more times of rubbing the nose than usual (cold / allergy, etc.)", "notify the driver (warning / follow-up)" is used as the notification system. Shown. For example, it is possible to call attention by notifying the driver by means of voice, sound, light, image or character display, vibration, or the like from the vehicle. After that, as follow-up observation, the level 1 may be determined again with a smaller number of times and time than the normal index shown in FIG. 9, or the abnormal level may be raised if the same state continues. .. As a countermeasure in the case of level 2 "inattentive driving", "notify the driver" and "notify the administrator" are shown as the notification system, and "speed down" is shown as the vehicle control system. The notification system "notify the driver" is as described above, and the "notify the administrator" notifies the administrator registered in advance by e-mail, telephone, SNS, etc., for example. As a result, for example, in the case of a company car, the convenience can be improved such that the manager of the company can grasp the situation, and the safety awareness of the driver can be improved. Further, by directly controlling the vehicle as a vehicle control system to perform "speed down", it is possible to reduce the probability of an accident and reduce the damage in the event of an accident. As a countermeasure in the case of level 3 "screaming", "notify the administrator" is shown as a notification system, and "speed down" and "speed limit" are shown as a vehicle control system. The correspondence between the notification system "notify the administrator" and the vehicle control system "speed down" is as described above. The "speed limit" of the vehicle control system is, for example, the possibility of driving by the driver or an accident by limiting the upper limit speed at which the vehicle can be directly controlled for a certain period of time after the level 2 judgment. I think it can be lowered. As a countermeasure in the case of level 4 "consciousness is jumping (sudden illness / lack of sleep, etc.)", "notification to surrounding people" is shown as a notification system, and "sudden stop" is shown as a vehicle control system. .. The notification system "notification to surrounding people" is, for example, from the vehicle to the inside and outside of the vehicle by using means such as voice, sound, light, image and character display to the passenger and people around the vehicle. By giving the notification, the effect of having the driver of the vehicle work can be expected. Further, the "sudden stop" of the vehicle control system is to prevent the occurrence of an accident by promptly stopping at a safe place, not by suddenly braking. At that time, the hazard lamp or the like may be turned on at the same time. As a countermeasure in the case of "unconsciousness" of level 5, in addition to the countermeasure of level 4, "report to an emergency agency" is shown as a notification system. As the "report to the emergency agency", for example, a public institution such as a fire department may be notified by calling 119, etc., to the effect that the vehicle position information and the driver are unconscious, or depending on the type of vehicle. A similar report may be made to the corresponding emergency contact. In FIG. 8, since it is an example of one abnormal operation for each abnormal level, countermeasures are shown for each level, but when there are multiple abnormal operations at the same level, countermeasures are set according to the abnormal operation. You can do it.

Returning to FIG. 3, the execution means 118 of the computer 100 executes the countermeasure selected in step S308 (step S309). In FIG. 3, it is assumed that "notify the driver" and "notify the administrator" are executed as the notification system, and "speed down" is executed as the vehicle control system.

Finally, the control unit 110 of the computer 100 confirms whether or not to end the driver abnormality response process (step S310). If the driver abnormality handling process is not completed, the unknown image acquisition process in step S305 is performed, and if the driver abnormality handling process is terminated, the processing of the computer 100 is terminated. As the timing for ending the driver abnormality handling process, it is conceivable that a notification of the end of imaging is received from the imaging device 200.

In the flowchart of FIG. 3, the case where the abnormality level is determined in step S307 to be level 2 has been described as an example, but here, when it does not correspond to any abnormality level from level 1 to level 5, that is, when it is normal, the step. It is assumed that step S309 is skipped from S308 and the process proceeds to step S310.

Further, an unknown image in the case where the abnormality level is determined in step S307 and does not correspond to any abnormality level may be acquired as a normal image and used for learning the next normal operation.

As described above, according to the present invention, by learning how to drive a driver such as a vehicle in a normal state, an abnormality level is determined when the driving method of the driver is different from the learning result, and the determined abnormality level is determined. It is possible to provide a driver abnormality response system, a driver abnormality response method, and a program capable of selecting a countermeasure according to the situation and executing the selected countermeasure.

[Processing for handling driver abnormalities using biological information]
FIG. 4 is a diagram showing the relationship between the functional blocks of the computer 100, the imaging device 200, and the biological information acquisition device 300 and each function when the biological information is also acquired. In addition to the computer 100 and the imaging device 200 having the configuration shown in FIG. 2, a biological information acquisition device 300 is provided. Although only the functional blocks of the biological information acquisition device 300 are shown in FIG. 4, it is assumed that the functional blocks of the computer 100 and the imaging device 200 are the same as those in FIG. The communication network 400 enables communication between the computer 100 and the imaging device 200, and between the computer 100 and the biological information acquisition device 300, respectively.

The biological information acquisition device 300 is composed of a biological information acquisition unit 30, a control unit 310, a communication unit 320, a storage unit 330, an input unit 340, and an output unit 350. The biological information acquisition device 300 is a device including a camera device or a sensor device capable of acquiring biological information such as heartbeat, body temperature, electrocardiogram, blood pressure, respiration, sympathetic nerve, and parasympathetic nerve, which can perform data communication with the computer 100, and is operated. Acquire the biometric information of the driver inside. Although the illustration is based on the premise of a wearable device, it may be a device capable of acquiring biological information such as a camera device, a medical terminal, or an in-vehicle device. Further, the biological information acquired in the storage unit 330 may be stored.

The biological information acquisition device 300 includes a device such as a camera device and a sensor device capable of acquiring biological information as the biological information acquisition unit 30. The biological information here is information capable of determining the physical or mental state of the driver such as heartbeat, body temperature, electrocardiogram, blood pressure, respiration, sympathetic nerve, and parasympathetic nerve. When the biological information acquisition unit 30 is a camera device, it is possible to analyze the image acquired by the control unit 310 to obtain biological information such as heartbeat and respiration. When the biological information acquisition unit is a sensor device, it is possible to obtain heartbeat, body temperature, electrocardiogram, blood pressure, respiration, etc. from each sensor. Further, by analyzing the heartbeat, body temperature, electrocardiogram, blood pressure, respiration, etc. by the control unit 310, it is possible to obtain biological information such as sympathetic nerves and parasympathetic nerves.

The control unit 310 includes a CPU, RAM, ROM, and the like.

The communication unit 320 is provided with a device for enabling communication with other devices.

The storage unit 330 is provided with a data storage unit using a hard disk or a semiconductor memory, and stores necessary data such as biometric information. Information on the date and time and place where the biological information was acquired may also be stored. The storage unit 330 may be an external server, a database, a cloud service, or the like.

The input unit 340 shall have a function necessary for operating the biological information acquisition device 300. As an example for realizing input, it is possible to provide a hardware button, a touch panel, a microphone, and the like.

The output unit 350 is provided with functions necessary for operating the biological information acquisition device 300. As an example for realizing the output, a form such as a display on a display or an audio output can be considered.

Here, the driver abnormality response process using the biological information executed by each of the above-mentioned means will be described with reference to the flowchart of FIG. Since the processing related to the normal image and the unknown image is the processing described above as the explanation of FIG. 3, here, it will be described mainly on the biological information to be additionally processed.

First, the first acquisition means 111 of the computer 100 further acquires the biological information of the driver in the normal state together with the image in the normal state captured by the driver in the normal state (step S301). The acquisition of the biometric information of the driver in the normal state here may be performed directly from the biometric information acquisition device 300, may be performed from a database collecting the biometric information at the time of living body, an external server, or the like, or the biometric information. It may be performed from a device other than the acquisition device 300, or may be performed from a storage medium such as a CD-ROM, a DVD, or a USB memory. The acquired biological information shall be information that can determine the physical or mental state of the driver such as heartbeat, body temperature, electrocardiogram, blood pressure, respiration, sympathetic nerve, and parasympathetic nerve.

Next, the first detection means 112 of the computer 100 further detects the normal biometric information in addition to the detection of the normal operation (step S302). When there are a plurality of acquired normal biometric information, the normal biometric information is detected for each of the acquired biometric information. The normal biological information here is assumed to be the driver's normal heartbeat, body temperature, electrocardiogram, blood pressure, respiration, sympathetic nerve, parasympathetic nerve, and the like.

Next, the learning means 113 further learns the normal biometric information in addition to the learning of the normal movement (step S303). Since the biological information such as heartbeat, body temperature, electrocardiogram, blood pressure, respiration, sympathetic nerve, and parasympathetic nerve detected in step S302 is normal, it is learned what range each of them numerically falls within. To do. If the normal biometric information is insufficient for learning the normal biometric information, step S301 and step S302 are repeated. On the contrary, if the learning is sufficiently performed, steps S301 to S303 may be omitted.

When it is desired to use the biometric information for the driver abnormality response process, the biometric information acquisition device 300 starts the biometric information acquisition at a timing before the driver starts driving (step S304). Further, the biometric information acquisition device 300 notifies the computer 100 whether or not the biometric information is acquired at the timing when the driver starts driving.

When the biometric information acquisition device 200 is acquiring biometric information, the second acquisition means 114 of the computer 100 is in the normal state or the abnormal state together with the acquisition of the unknown image from the image pickup device 200. The biometric information of the unknown driver is acquired from the biometric information acquisition device 300 (step S305). The unknown driver biometric information here is data such as heartbeat, body temperature, electrocardiogram, blood pressure, respiration, sympathetic nerve, and parasympathetic nerve when the driver is normal or abnormal or unknown.

The second detection means 115 of the computer 100 detects whether the driver's biological information is normal or abnormal, in addition to detecting an unknown operation (step S306). Whether the driver's biological information is normal or abnormal here is whether the data such as heart rate, body temperature, electrocardiogram, blood pressure, respiration, sympathetic nerve, and parasympathetic nerve are numerically within the normal range. Detected by.

Next, the determination means 116 of the computer 100 determines the abnormality level in consideration of the degree of whether the driver's biological information is normal or abnormal when determining the abnormality level of the unknown operation (step S307). ). For example, if any one or more of the data such as heartbeat, body temperature, electrocardiogram, blood pressure, respiration, sympathetic nerve, and parasympathetic nerve are numerically out of the normal range, that is, abnormal, the abnormality is found. The anomaly level is determined by taking into consideration whether is numerically large or small, how many anomalous data are available, and the like. If the abnormality is numerically large or there is a lot of abnormality data, the abnormality level is raised to make a judgment. How much biological information is added can be set according to the system.

As described above, according to the present invention, by learning how to drive a driver such as a vehicle in a normal state, when determining an abnormality level when the driving method of the driver is different from the learning result, only an image is used. Driver abnormality response system, driver abnormality that can determine the abnormality level more appropriately, select the countermeasure according to the determined abnormality level, and execute the selected countermeasure by using the biometric information. It becomes possible to provide a response method and a program.

[Driver error handling processing considering time series]
By considering the time series of the acquired images in the driver abnormality handling process, it is possible to perform appropriate driver abnormality handling processing even when the acquired image is not a moving image but a still image, for example. Here, the driver abnormality handling process in consideration of the time series will be described with reference to the flowchart of FIG. Here, it is assumed that the processing that considers the time series is mainly described.

First, the first acquisition means 111 of the computer 100 acquires a normal image captured by the normal driver in time series (step S301).

Next, the first detection means 112 of the computer 100 analyzes the time-series normal-time image acquired in step S301 to detect the normal-time operation (step S302).

Next, the learning means 113 learns the normal operation of the time series detected in step S302 (step S303).

Then, the determination means 116 of the computer 100 determines the abnormal level of the unknown operation according to the degree to which the unknown operation detected in step S306 differs from the learned time-series result (step). S307). For example, when the unknown image 01, the unknown image, 02, the unknown image 03, the unknown image 04, and the unknown image 05 are acquired in chronological order, the abnormality level of each of the five unknown images is not determined to be level 1 or higher. Also, considering the order of the time series, there is a possibility that the abnormality level is level 1 or higher when compared with the normal operation of the time series.

As described above, according to the present invention, by learning the normal driving method of a driver such as a vehicle in consideration of the time series, the abnormality level is determined when the driving method of the driver is different from the learning result. By considering the time series at the time, even if a still image is used, the abnormality level is judged more appropriately, the countermeasure is selected according to the determined abnormality level, and the selected countermeasure is executed. It is possible to provide a driver abnormality response system, a driver abnormality response method, and a program that can be used.

[Driver error handling processing that takes into account location information]
In the driver abnormality handling process, it is possible to perform more appropriate driver abnormality handling processing by acquiring location information such as GPS information and map information together with the image and adding it. Here, the driver abnormality handling process in consideration of the location information will be described with reference to the flowchart of FIG. Here, it is assumed that the processing related to the location information is mainly described.

First, the first acquisition means 111 of the computer 100 acquires the location information during normal operation together with the normal image captured by the driver during normal operation (step S301). The acquisition of the location information during normal driving here is based on the assumption that GPS information and map information are added to the image information in advance, but it may be acquired by other means. The location information to be acquired shall be information such as GPS information and map information that can determine the location at the time of driving.

The learning means 113 learns the normal operation including the location information during the normal operation (step S303). This is because, for example, a road with a good view and a road with a poor view have different movements of the driver such as face and hand movements, gestures, postures, eyes, and facial expressions even under normal conditions. In addition, it is considered that the driver's behavior differs between general roads and highways, and between streets and mountain roads.

The second acquisition means 114 further acquires the location information together with the acquisition of the unknown image from the imaging device 200 (step S305).

Then, the determination means 116 makes a determination by adding the location information to the unknown operation when determining the abnormal level of the unknown operation (step S307). As an example of a method for making a judgment that takes into account location conditions, depending on whether the location information of unknown operation is a road with a good view or a road with a poor view, a general road or a highway, a city or a mountain road, etc. , It is conceivable to make a judgment by comparing with the normal operation learned under the same conditions.

As described above, according to the present invention, by learning how to drive a driver such as a vehicle in a normal state by adding location information, an abnormality level is determined when the driving method of the driver is different from the learning result. A driver anomaly response system that can more appropriately determine the anomaly level by taking into account location information, select countermeasures according to the determined anomaly level, and execute the selected countermeasures. It is possible to provide a driver abnormality response method and a program.

[Driver error handling process when determining the error level with the same driver]
FIG. 5 is a flowchart of a driver abnormality handling process when determining an abnormality level with the same driver, and can be realized with the same configuration as in FIG. The processing executed by each of the above-mentioned means will be described with reference to this flowchart. Here, the difference from the flow of FIG. 3 will be mainly described.

First, the first acquisition means 111 of the computer 100 acquires a normal image captured by the normal driver (step S501).

Next, the first detection means 112 analyzes the normal image acquired in step S501 to identify the driver, and detects the normal operation of the driver (step S502). In order to identify the driver, not only the image data of the person shown in the normal image but also the image data of the interior of the vehicle may be used, which is useful for identifying the driver in addition to the image data. Data may be added in advance.

Next, the learning means 113 learns the normal operation detected in step S502 for each specified driver (step S503). Regarding the face and hand movements, gestures, postures, eyes, facial expressions, etc. of each driver detected in step S502, what range does each fall within, how often and how many times are performed? Learn whether the time will continue.

The imaging device 200 starts imaging at the timing when the driver starts driving (step S504). The imaging device 200 notifies the computer 100 of the start of imaging at the same time as the start of imaging.

Upon receiving the notification from the imaging device 200 of the start of imaging, the second acquisition means 114 of the computer 100 acquires an unknown image captured by an unknown driver, whether it is normal or abnormal, from the imaging device 200 ( Step S505).

Next, the second detection means 115 analyzes the acquired unknown image, identifies the driver, and detects the unknown operation (step S506).

Next, the determination means 116 determines the abnormality level according to the degree to which the unknown operation detected in step S506 of the specified driver is different from the result learned in step S503 of the specified driver (step S507). Here, we compare the normal movements in which the detected unknown movements are learned and how much they differ from each index such as the driver's face and hand movements, gestures, postures, eyes, and facial expressions. Determine the abnormal level according to the degree.

The selection means 117 selects a countermeasure according to the abnormality level determined in step S507 (step S508).

Execution means 118 executes the countermeasure selected in step S508 (step S509).

Finally, the control unit 110 of the computer 100 confirms whether or not to end the driver abnormality handling process (step S510). If the driver abnormality handling process is not completed, the unknown image acquisition process in step S505 is performed, and if the driver abnormality handling process is ended, the processing of the computer 100 is terminated. As the timing for ending the driver abnormality handling process, it is conceivable that a notification of the end of imaging is received from the imaging device 200.

As described above, according to the present invention, by identifying the driver and learning the normal driving method of the driver such as a vehicle for each driver, the abnormal level when the driving method of the driver is different from the learning result. It is possible to provide a driver abnormality response system, a driver abnormality response method, and a program capable of determining with high accuracy, selecting a countermeasure according to the determined abnormality level, and executing the selected countermeasure. It will be possible.

In the flowchart of FIG. 5, the high-precision driver abnormality handling process when determining the abnormality level with the same driver has been described, but conversely, it is also possible to compare a large number of normal drivers with a certain driver. is there. In that case, there is an advantage that the driver abnormality handling process can be performed even for a new driver. When comparing a large number of normal drivers with a certain driver, when the normal image is acquired in step S301 of FIG. 3, the normal images captured by the normal drivers are acquired for a plurality of people, and the normal image is acquired in step S307. At the time of determining the abnormality level, it is possible to determine the abnormality level according to the degree to which the unknown operation detected in step S306 differs from the average value or the median value of the results of the normal operation learned in step S303.

The above-mentioned means and functions are realized by a computer (including a CPU, an information processing device, various terminals, a virtual device, and a cloud service) reading and executing a predetermined program. The program may be provided, for example, from a computer via a network, or may be provided in a form recorded on a readable recording medium.

Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments described above. In addition, the effects described in the embodiments of the present invention merely list the most preferable effects arising from the present invention, and the effects according to the present invention are limited to those described in the embodiments of the present invention. is not it.

100 computers, 200 imaging devices, 300 biometric information acquisition devices, 400 communication networks

Claims (9)

1. It is a driver abnormality response system that determines the abnormality level of the driver while driving and executes the selected countermeasure.
   The first acquisition means for acquiring a normal image captured by a normal driver, and
   The first detection means for analyzing the acquired normal image and detecting the normal operation,
   The learning means for learning the detected normal operation and
   A second acquisition means for acquiring an unknown image captured by an unknown driver, whether it is normal or abnormal, and
   A second detection means that analyzes the acquired unknown image and detects an unknown operation,
   A determination means for determining an abnormality level according to the degree to which the detected unknown motion differs from the learned result.
   A selection means for selecting a countermeasure according to the determined abnormality level, and
   An execution means for executing the selected countermeasure and
   Driver abnormality response system equipped with.
2. (Acquisition of biological information)
   In the first acquisition means, the biological information of the driver at the normal time is further acquired, and the biological information is acquired.
   In the first detection means, further normal biometric information is detected.
   In the learning means, further learning the biological information at the time of normal,
   In the second acquisition means, the biological information of the driver, which is unknown whether it is normal or abnormal, is further acquired.
   In the second detection means, it is further detected whether the driver's biological information is normal or abnormal, and the driver's biological information is detected.
   The driver abnormality response system according to claim 1, wherein in the determination means, when determining an abnormality level, the determination is made in consideration of the degree of whether the biological information of the driver is normal or abnormal.
3. (Time series)
   In the first acquisition means, the normal time image captured by the normal time driver is acquired in chronological order, and the normal time image is acquired.
   In the first detection means, the acquired time-series normal time image is analyzed to detect the normal time operation, and the normal time operation is detected.
   In the learning means, the normal operation of the detected time series is learned, and
   The driver abnormality according to claim 1 or 2, wherein in the determination means, when determining the abnormality level, the detected unknown operation is determined according to the degree of difference from the learned time series result. Corresponding system.
4. (Adding location)
   In the first acquisition means, the location information during normal operation is further acquired, and the location information is acquired.
   In the learning means, further learning including location information during normal driving is performed.
   In the second acquisition means, the location information during operation, which is unknown whether it is normal or abnormal, is further acquired.
   The driver abnormality response system according to any one of claims 1 to 3, wherein in the determination means, when determining an abnormality level, the determination is made by adding the location information to the unknown operation.
5. (Adding the number of times and time)
   The driver abnormality response according to any one of claims 1 to 4, wherein in the determination means, the detected unknown operation determines the abnormality level according to the number of times or time different from the learned result. system.
6. (Absolute evaluation with the same driver)
   In the first detection means, the normal image is analyzed to identify the driver, and the normal operation of the driver is detected.
   In the learning means, the detected normal operation is learned for each of the specified drivers.
   In the second detection means, the acquired unknown image is analyzed to identify the driver, and the unknown operation is detected.
   Any of claims 1 to 5, wherein in the determination means, the abnormality level is determined according to the degree to which the detected unknown operation of the specified driver differs from the learned result of the specified driver. The driver abnormality response system described in item 1.
7. (Relative evaluation with average value)
   In the first acquisition means, the normal driver images captured by the normal driver are acquired for a plurality of people, and the images are acquired for a plurality of people.
   The invention according to any one of claims 1 to 5, wherein in the determination means, the abnormality level is determined according to the degree to which the detected unknown operation differs from the average value or the median value of the learned results. Driver abnormality response system.
8. (Method claim)
   It is a driver abnormality response method that determines the abnormality level of the driver while driving and executes the selected countermeasure.
   The step of acquiring the normal image captured by the normal driver, and
   A step of analyzing the acquired normal image to detect normal operation, and
   The step of learning the detected normal operation and
   The step of acquiring an unknown image captured by an unknown driver, whether it is normal or abnormal, and
   A step of analyzing the acquired unknown image to detect an unknown operation,
   A step of determining an abnormality level according to the degree to which the detected unknown motion differs from the learned result.
   A step of selecting a countermeasure according to the determined abnormality level, and
   The steps to implement the selected workaround and
   How to deal with driver abnormalities.
9. (Program claim)
   In a driver anomaly response system that determines the driver's anomaly level while driving and implements the selected countermeasure.
   Steps to acquire a normal image captured by a normal driver,
   A step of analyzing the acquired normal image to detect normal operation,
   The step of learning the detected normal operation,
   Steps to acquire an unknown image taken by an unknown driver, whether it is normal or abnormal,
   The step of analyzing the acquired unknown image to detect an unknown operation,
   A step of determining an abnormality level according to the degree to which the detected unknown motion differs from the learned result.
   A step of selecting a countermeasure according to the determined abnormality level,
   Steps to implement the selected workaround,
   A program to run.

Images