CN112441009B

CN112441009B - State estimation device, state estimation method, and storage medium

Info

Publication number: CN112441009B
Application number: CN202010876616.0A
Authority: CN
Inventors: 中村智宽
Original assignee: Clarion Co Ltd
Current assignee: Faurecia Clarion Electronics Co Ltd
Priority date: 2019-08-27
Filing date: 2020-08-27
Publication date: 2024-03-12
Anticipated expiration: 2040-08-27
Also published as: US20210059615A1; US11627918B2; EP3785608A1; JP2021033748A; JP7328828B2; CN112441009A

Abstract

The present invention aims to provide a state estimation device, a state estimation method and a storage medium, which can more appropriately estimate a precursor of a state which is likely to influence the operation of a user. The state estimation device of the present invention includes: a biological information acquisition unit for acquiring biological information of a user; and a precursor detection unit that uses a learning completion model in which the biological information related to the predetermined physical condition abnormality is learned as teaching data, and determines whether or not the biological information of the user matches the precursor of the predetermined physical condition abnormality, thereby detecting the precursor.

Description

State estimation device, state estimation method, and storage medium

Technical Field

The present invention relates to a state estimation device, a state estimation method, and a storage medium storing a state estimation program.

Background

As a background art, japanese patent application laid-open No. 2018-127112 (hereinafter referred to as patent document 1) has been proposed. Patent document 1 discloses "[ technical problem to be solved ] to provide a wakefulness estimation device capable of estimating the wakefulness of a driver of a vehicle having an automatic driving mode and a manual driving mode with high accuracy. Means for solving the problems the vehicle is switchable between an automatic driving mode and a manual driving mode, and comprises: a vehicle behavior detection unit that detects driving information of a vehicle; a first awareness identifying unit that identifies a first awareness of the driver from the detected driving information; a biological information detection unit for detecting 1 or more biological information of the driver; a second awareness identifying unit that identifies a second awareness of the driver from the detected 1 or more pieces of biometric information; and a wakefulness estimation unit that estimates a third wakefulness of the driver from at least one of the identified first wakefulness and second wakefulness. The wakefulness estimation unit estimates a third wakefulness from the first wakefulness and the second wakefulness in the manual driving mode, and estimates the third wakefulness from the second wakefulness in the automatic driving mode. "

Prior art literature

Patent literature

Patent document 1: japanese patent application laid-open No. 2018-127112

Disclosure of Invention

Technical problem to be solved by the invention

In patent document 1, for example, the wakefulness can be estimated only after the user has felt drowsiness and abnormality occurs in the behavior or driving operation of the vehicle. Accordingly, an object of the present invention is to provide a technique capable of estimating a precursor of a state that may affect a user operation more appropriately.

Means for solving the technical problems

The present invention includes various aspects for solving at least some of the above-described technical problems, for example, as follows. In order to solve the above-described problems, a state estimating device according to the present invention includes: a biological information acquisition unit for acquiring biological information of a user; and a precursor detection unit that uses a learning completion model in which the biological information related to a predetermined physical condition abnormality is learned as teaching data, and determines whether or not the biological information of the user matches the predetermined physical condition abnormality, thereby detecting a precursor.

Effects of the invention

The present invention can more appropriately estimate a precursor of a state that may affect the motion of a user. Other technical problems, aspects and effects than those described above will become apparent from the following description of the embodiments.

Drawings

Fig. 1 is a diagram showing an example of a configuration of a state estimating device to which an embodiment of the present invention is applied.

Fig. 2 is a diagram showing an example of a data structure of the precursor model storage unit.

Fig. 3 is a diagram showing an example of a data structure of the biological information storage unit.

Fig. 4 is a diagram showing an example of a data structure of the teaching data storage section.

Fig. 5 is a diagram showing an example of a data structure of the teaching data learning proportion storage unit.

Fig. 6 is a diagram showing an exemplary functional configuration of the controller.

Fig. 7 is a diagram showing an example of the flow of the precursor detection process.

Fig. 8 is a diagram showing an example of the awake abnormality notification screen.

Fig. 9 is a diagram showing an example of a flow of the physical condition abnormality detection process.

Fig. 10 is a diagram showing an example of a flow of the teaching data generation process.

Fig. 11 is a diagram showing an example of a flow of training processing.

Fig. 12 is a diagram showing an example of a flow of the model update processing.

Fig. 13 is a diagram showing an example of a flow of correction processing.

Fig. 14 is a diagram showing an example of a configuration to which the state estimating device according to the second embodiment is applied.

Fig. 15 is a diagram showing an example of teaching data setting information in the third embodiment.

Description of the reference numerals

1 … … controller, 2 … … display, 3 … … memory device, 4 … … sound input/output device, 5 … … input device, 10 … … biological information acquisition device, 11 … … network communication device, 12 … … state detection device, 21 … … processor, 22 … … memory, 23 … … I/F,24 … … bus, 41 … … microphone, 42 … … speaker, 51 … … touch panel, 52 … … dial switch, 100 … … state estimation device, 200 … … precursor model memory portion, 300 … … biological information memory portion, 400 … … teaching data memory portion, 500 … … teaching data learning ratio memory portion.

Detailed Description

Next, an example in which the state estimating device 100 of the present invention is applied to an in-vehicle device will be described with reference to the drawings. However, the present invention is not limited to the in-vehicle apparatus, and may be applied to an apparatus used when an object requiring care is operated. Examples of the device include a device used for driving, operating, and boarding a mobile body such as an airplane, a train, or a ship, and a heavy machine such as a forklift, a crane, or an excavator.

Fig. 1 to 14 do not show all the components of the state estimating device 100, but some of the components are omitted as appropriate for easy understanding. In all the drawings for describing the embodiments, the same reference numerals are given to the same components in principle, and a repetitive description thereof may be omitted. In the following embodiments, it is needless to say that the constituent elements (including the element steps) are not necessarily essential, except for the cases specifically shown and the cases which are considered to be essential in principle.

It should be noted that when "consisting of a", "having a", "including a", and the like are described, it is needless to say that elements other than the above elements are not excluded, except that the elements are specifically and clearly indicated. In the same manner, in the following embodiments, when shapes, positional relationships, and the like of constituent elements and the like are mentioned, examples of the shapes and the like substantially similar to or similar to those of the constituent elements are included, except for cases where they are specifically explicitly shown and cases where they are not considered to be obvious in principle.

Further, as an expression of the present invention, "state information" basically means information obtained by visually observing a human body of a user from the outside, and "abnormal physical condition" means a state deviating from an expected normal state among states comprehensively judged by visually observing the obtained states from the outside. For example, the "status information" includes all information that can be quantified by objectively observing a living body such as "blink frequency", "yawning", "viewpoint movement amount", "body shake, inclination", "snoring", "stiffness (no posture change)", "cramp", and the like.

Examples of the "abnormal physical condition" include all kinds of comprehensive abnormal physical conditions which may adversely affect the operation or exercise actions, such as "low wakefulness (including drowsiness)", "high wakefulness (including abnormal mental high and the like)", "unconsciousness (including involuntary movements such as epilepsy and syncope)", "heart attack", "autonomic nerve disorder (including metabolic disorder and the like". Such various biological states and information thereof are not limited to the case where only the element is specifically and clearly shown, and other elements are not excluded.

Further, "biological information" means information obtained by measuring the human body of the user from the outside. For example, as the "biological information", brain waves, pulse waves, blood pressure, body temperature, and the like are included. Such various biological information is not limited to the specific elements other than those specifically shown, and it is needless to say that the other elements are not excluded.

Furthermore, it is generally apparent that when an abnormal onset of a certain physical condition occurs, before that, a characteristic change in biological information occurs. For example, with respect to "low wakefulness (drowsiness)", it is known that an increase in blink frequency or an increase in yawning is a visually observable symptom, and the heart rate is reduced as a precursor in advance (in the order of several tens of minutes).

In addition, regarding the onset of "epilepsy", it is known that a physical condition which is difficult to visually observe, such as an increase in brain wave disorder, is abnormal in heartbeat (HRV: heart Rate Variability (heart rate variability, variation in milliseconds of RR interval)) as a precursor in advance (approximately 8 minutes).

That is, if the precursor before the abnormal occurrence of such a physical condition can be estimated from the biological information, a warning can be given or various countermeasures can be taken before the occurrence.

Fig. 1 is a diagram showing an example of a configuration of a state estimating device to which an embodiment of the present invention is applied. The state estimating device 100 detachably mounted on the mobile body is an information processing device capable of acquiring biological information and state information. However, the state estimating device 100 to which the present invention is applied is not limited to the state estimating device 100 shown in fig. 1. For example, various control devices incorporated in a moving body or heavy machinery may be used.

The state estimation device 100 includes a controller 1, a display 2, a storage device 3, a sound input/output device 4 (including a microphone 41 as a sound input device and a speaker 42 as a sound output device), an input device 5, a biological information acquisition device 10, a network communication device 11, and a state detection device 12.

The controller 1 is a central unit that performs various processes. For example, the current position is calculated based on information output from a vehicle speed sensor, an acceleration sensor, and a GPS (Global Positioning System: global positioning system) receiving device, which are not shown. Map data and the like necessary for display are read from the storage device 3 based on the obtained information of the current position.

Further, the controller 1 graphically expands the read map data, superimposes thereon a mark representing the current position, and displays it on the display 2. The recommended route, that is, the optimal route connecting the current position or the departure point instructed by the user and the destination (or the route through the ground or the road) is searched for using the map data or the like stored in the storage device 3. The speaker 42, the display 2, and the like are used to guide the user.

The controller 1 of the state estimating apparatus 100 is configured to connect the devices to each other by the bus 24. The controller 1 has: a processor 21 for performing numerical operations and controlling various processes of the respective devices; a memory 22 for storing map data, operation data, and the like read from the storage device 3; an I/F (interface) 23 for connecting various hardware with the controller 1.

The display 2 is a unit that displays graphic information generated by the controller 1 or the like. The display 2 is constituted by a liquid crystal display, an organic EL (Electro Luminescence: electroluminescence) display, or the like. The display 2 includes a head-up display, a dashboard, a center console, and the like. The display 2 may display information on a communication terminal such as a smart phone (smart phone) via communication.

The storage device 3 is configured by at least a readable and writable storage medium such as an HDD (Hard Disk Drive), an SSD (Solid State Drive) and a nonvolatile memory card.

The storage medium stores therein: map data (including link data of links constituting a road on a map and link cost as a reference) as a reference required for a general route search device; a precursor model storage unit 200; a biological information storage unit 300; a teaching data storage unit 400; and a teaching data learning scale storage unit 500.

Fig. 2 is a diagram showing an example of the data structure of the precursor model storage unit 200. The precursor model storage unit 200 stores the physical condition abnormality name 201 and the precursor model parameter 202 in association with each other. The physical condition abnormality name 201 is information for specifying physical condition abnormalities that are detection targets of the user, and includes, for example, "low wakefulness (including drowsiness)", "high wakefulness (including abnormally high mental strength and the like)", "unconsciousness (including involuntary movements such as epilepsy and syncope)", "heart attack", "autonomic nerve disorder (including metabolic disorder and the like)", and the like, which are all comprehensive "physical condition abnormalities" that may adversely affect the operation or driving.

The precursor model parameters 202 are parameters obtained by modeling the precursor part for each abnormal body condition by using the characteristics of the change in the biological information of the type acquired by the biological information acquisition device 10. Here, the precursor model is a learning completion model called a neural network configured using a machine learning method called deep learning, but is not limited thereto. For example, a bayesian classifier, various AI (artificial intelligence) supporting a vector machine, or the like may be used to detect precursors.

Fig. 3 is a diagram showing an example of a data structure of the biological information storage unit 300. The biological information storage unit 300 stores a user ID 301, a time (start and end) 302, a data length (recording time and sampling period) 303, a detected physical condition abnormality 304, and biological information 305 in association with each other.

The user ID 301 is information for specifying a user. The time (start, end) 302 is information for determining the recording start time and end time of the biological information. The data length (recording time, sampling period) 303 is information for determining the time and sampling period from the start to the end of recording of the biological information. The detected physical condition abnormality 304 is information for determining a physical condition abnormality associated with biological information. The biometric information 305 is information obtained by associating a record of predetermined biometric information with a biometric information ID for specifying the information.

Fig. 4 is a diagram showing an example of a data structure of the teaching data storage section 400. The teaching data storage unit 400 stores the biometric information ID 401 and a label 402 in association with each other. The biological information ID 401 is information for determining biological information. If the biological information specified by the biological information ID 401 is biological information indicating a precursor of a prescribed physical condition abnormality, the flag 402 is information specifying the physical condition abnormality. If the biometric information is not the biometric information indicating a precursor of a predetermined physical condition abnormality, that is, if the biometric information is healthy biometric information without a physical condition abnormality, the flag 402 is null.

Fig. 5 is a diagram showing an example of a data structure of the teaching data learning ratio storage unit 500. The teaching data learning proportion storage unit 500 stores the physical condition abnormality name 501 and the marked data learning proportion (mark/total number) 502 in association with each other. The physical condition abnormality name 501 is information for determining a physical condition abnormality. The marked data learning ratio (mark/total number) 502 is information indicating a subdivision ratio of teaching data for machine learning of the precursor model for each physical condition abnormality determined by the physical condition abnormality name 501.

The data to which the mark is given is biometric information data for a predetermined period before occurrence of the physical condition abnormality when the physical condition abnormality occurs, and is data associated with the occurrence of the physical condition abnormality. The data to which the mark is not given is biometric information data for a predetermined period of time when no physical condition abnormality has occurred thereafter. For example, in the case where the marked data learning ratio (mark/total number) 502 of the teaching data for constructing the precursor model of the "low wakefulness" physical condition abnormality is "1:2", marked data representing "low wakefulness" occupies approximately half of the total number of teaching data for learning.

The description returns to fig. 1. The sound input-output device 4 includes a microphone 41 as a sound input device and a speaker 42 as a sound output device. The microphone 41 is used to acquire sounds, such as sounds made by a user or other passengers, external to the state estimating apparatus 100.

The speaker 42 outputs a message generated by the controller 1 to the user as sound. The microphone 41 and the speaker 42 are provided at predetermined portions of the moving body, respectively. However, the housing may be housed in an integrated case. The state estimation device 100 may include a plurality of microphones 41 and a plurality of speakers 42, respectively.

The state estimating device 100 may output sound from a microphone and a speaker of another device connected thereto (for example, a smart phone connected by a USB (Universal Serial Bus: universal serial bus) cable or the like, a smart phone connected wirelessly by Wifi, bluetooth (registered trademark) or the like), or the like) instead of the microphone 41 and the speaker 42.

The input device 5 is a device that receives an instruction from a user by a user operation. The input device 5 is constituted by a touch panel 51, a dial switch 52, scroll keys as other hard switches (not shown), zoom keys, a gesture sensor that detects a gesture of a user, and the like. The input device 5 includes a remote controller capable of remotely instructing the state estimation device 100 to operate. The remote controller includes dial switches, scroll keys, scaling keys, and the like, and can transmit information that each key or switch is operated to the state estimating device 100.

The touch panel 51 is attached to the display surface side of the display 2, and can see through the display screen. The touch panel 51 can specify a touch position corresponding to XY coordinates of an image displayed on the display 2, and convert the touch position into coordinates to output. The touch panel 51 is constituted by a pressure-sensitive or electrostatic input detection element or the like. The touch panel 51 may be configured to be capable of performing multi-touch in which a plurality of touch positions can be detected simultaneously.

The dial switch 52 is configured to be rotatable clockwise and counterclockwise, and generates a pulse signal every time it rotates by a prescribed angle and outputs it to the controller 1. The controller 1 obtains the rotation angle from the number of pulse signals.

The biological information acquiring apparatus 10 is an apparatus for acquiring biological information such as pulse waves of a user. For example, a photoelectric pulse method such as a reflection pulse measurement may be used for the acquisition of pulse wave information, but the present invention is not limited thereto, and various measurement methods such as a transmission pulse measurement, a phonocardiography, and an electrocardiography may be used.

As the biological information acquiring apparatus 10, for example, a doppler sensor, a pad (piezoelectric) sensor, or the like, which is appropriate for the use environment, can be used. The hardware of the biological information acquisition apparatus 10 may be such that a sensor is attached to a smart watch, a seat, a steering wheel, a column, or the like, and the sensor is transmitted to the state estimation apparatus 100 via a wireless or wired communication path such as Bluetooth (Bluetooth) or a USB cable.

The network communication device 11 is a device for connecting the state estimation device 100 to a network corresponding to a control network standard (not shown) such as CAN (Controller Area Network: controller area network) in a mobile unit, and for communicating with an ECU (Electronic Control Unit: electronic control unit) which is another control device in the mobile unit connected to the network by exchanging CAN messages. The network communication device 11 is also capable of connecting the state estimation device 100 to a mobile phone network, not shown, and communicating with other devices connected to the mobile phone network.

The state detection device 12 is a device for acquiring information of the onset state of a predetermined physical condition abnormality from a face image or the like of a user. The state detection device 12 acquires, for example, image information of a portion of the face or the like that can be visually observed from the outside by photographing or the like, and analyzes the image to acquire information on the status of the onset of the predetermined physical condition abnormality. For example, the state detection device 12 detects blink frequency, yawning, eye closure time ratio (PERCLOSE) per unit time (eye opening time ratio), microswitches (involuntary movement of the eyeball), shaking head, and the like, determines whether or not the detected information corresponds to a symptom of "low wakefulness", and detects a state of "low wakefulness". As a specific mechanism and algorithm for acquiring the information of the seizure status by the status detecting device 12, a conventional technique can be adopted.

Fig. 6 is a diagram showing a functional configuration of the controller 1. The controller 1 includes a basic control unit 111, an input receiving unit 112, an output processing unit 113, a biological information acquiring unit 114, a precursor detecting unit 115, a notifying unit 116, a state detecting unit 117, a teaching data generating unit 118, and a precursor model generating unit 119.

The basic control unit 111 is a functional unit that is a center for executing various processes, and controls operations of other functional units (the input receiving unit 112, the output processing unit 113, the biological information acquiring unit 114, the precursor detecting unit 115, the notifying unit 116, the state detecting unit 117, the teaching data generating unit 118, and the precursor model generating unit 119) according to the content of the processes. The basic control unit 111 also performs processing such as map matching processing to determine the current position by acquiring information of various sensors, GPS receiving devices, and the like.

The input receiving unit 112 receives an input instruction from the user input via the input device 5 or the microphone 41, and transmits the input instruction to the basic control unit 111 together with the touch coordinate position, the sound information, and the like, which are information related to the input instruction, so as to execute processing corresponding to the requested content. For example, when the user requests execution of a certain process, the input receiving section 112 requests the basic control section 111 for the requested instruction. That is, the input receiving unit 112 can be said to be an instruction receiving unit that receives an instruction by a user's operation.

The output processing unit 113 receives information constituting a screen to be displayed, such as polygon information, converts the information into a signal for drawing on the display 2, and instructs the display 2 to draw.

The biological information acquiring unit 114 establishes communication with the biological information acquiring apparatus 10, and performs continuous communication or intermittent communication to acquire information such as a pulse wave which is biological information acquired by the biological information acquiring apparatus 10, and stores it in the RAM 22 or the storage apparatus 3 together with history information for a certain period.

When the acquired biological information deviates from the predetermined range, the biological information acquisition unit 114 causes the notification unit 116 to issue a notification and deletes the biological information. Thus, the severity of the illness of the user due to the abnormality can be found early, and the deterioration of the detection accuracy of the precursor due to the mixing of the abnormal data into the teaching data can be avoided.

The precursor detection unit 115 uses a learning completion model (precursor model) in which biological information related to a predetermined physical condition abnormality is learned as teaching data, and determines whether or not the biological information acquired by the biological information acquisition unit 114 corresponds to a precursor of the predetermined physical condition abnormality, thereby detecting a precursor.

More specifically, when the biological information is acquired, the precursor detection unit 115 performs removal of outliers, interpolation of data, and the like on the biological information to shape the data, and calculates a predetermined index value (for example, frequency decomposition of an interval, HRV, and the like). Then, the precursor detection unit 115 causes the precursor model, which is a learning completion model, to detect a precursor of the physical condition abnormality, and when the precursor is detected, causes the notification unit 116 to notify that the precursor is detected (detected).

The notification unit 116 receives instructions from the functional unit such as the precursor detection unit 115, outputs or displays a predetermined message sentence or the like in a sound, outputs an alarm sound, transmits an email, and notifies a predetermined object by a predetermined method set by transmitting various messages including SNS (Social Network Service: social networking service) or the like. Alternatively, the notification unit 116 may vibrate the surface of the seat, the backrest, the armrest, the steering wheel, or the like used by the user for short notice.

The state detection unit 117 detects the onset state of the predetermined physical condition abnormality of the user. The state detection unit 117 acquires information of the onset state of a predetermined physical condition abnormality acquired from the facial image or the like of the user by the state detection device 12. For example, the state detection unit 117 detects all general physical condition abnormalities that may adversely affect operation or driving, such as "low wakefulness (including drowsiness)", "high wakefulness (including abnormally high mental strength, etc)", "unconsciousness (including involuntary movements such as epilepsy and syncope)", "heart attack", "autonomic nerve disorder (including metabolic disorder, etc.

When the state detection unit 117 detects the seizure state, the teaching data generation unit 118 generates teaching data using first information obtained by adding a mark to the biological information of the user in a predetermined period immediately before the seizure state is reached, and second information obtained by adding the biological information of the user in a predetermined period in which the seizure state is not detected. The teaching data generation unit 118 generates teaching data so that the first information and the second information are in a predetermined ratio (desirably 1:1).

The teaching data generation unit 118 determines the data length (recording period, sampling period) of each of the first information and the second information in response to the abnormal physical condition.

When the teaching data generating unit 118 receives an input indicating an objection to the precursor detected by the precursor detecting unit 115, for example, when the "no trouble" input from the user is received after the precursor of "low wakefulness" is notified, if the state detecting unit 117 does not detect the attack state within a predetermined period thereafter, the teaching data generating unit generates teaching data in which the biological information when the precursor is detected is regarded as second information, that is, biological information when the second information is not in an abnormal state, in order to improve the accuracy of detecting the precursor.

The precursor model generating unit 119 uses the teaching data generated by the teaching data generating unit 118 for machine learning, and generates a learning completion model as a precursor model. The precursor model generation process itself is basically the same process as the existing machine learning.

The functional units of the controller 1 described above, that is, the basic control unit 111, the input receiving unit 112, the output processing unit 113, the biological information acquiring unit 114, the precursor detecting unit 115, the notifying unit 116, the state detecting unit 117, the teaching data generating unit 118, and the precursor model generating unit 119, can be constructed by reading and executing a predetermined program by the processor 21. For this purpose, the memory 22 stores a program for realizing the processing of each functional unit.

The above-described components are classified according to the main processing contents so that the configuration of the state estimating device 100 can be easily understood. Therefore, the invention of the present application is not limited by the classification method of the constituent elements and the names thereof. The configuration of the state estimating device 100 may be classified into a larger number of components according to the processing content. Further, the classification may be performed such that 1 component performs more processes.

The processing of each functional unit may be performed by one hardware or a plurality of hardware, or the processing may be transferred to an external cloud server or the like via the network communication device 11 by each functional unit and the storage device 3, or the data may be stored in the external cloud server or the like.

Next, an operation related to the precursor detection process will be described. Fig. 7 is a diagram showing an example of the flow of the precursor detection process. When the state estimating device 100 is started, the precursor detection process is started at predetermined intervals (for example, at predetermined intervals of once every 1 second, once every 5 seconds, once every 1 minute, or the like).

First, the biological information acquisition unit 114 acquires biological information (step S001). Specifically, the biological information acquiring section 114 establishes communication with the biological information acquiring apparatus 10, and performs continuous communication or intermittent communication to acquire information that is a pulse wave of biological information acquired by the biological information acquiring apparatus 10, and stores it in the RAM 22 or the storage apparatus 3 together with history information for a certain period.

Then, the precursor detector 115 calculates a predetermined index value of the biological information (step S002). Specifically, the precursor detection unit 115 acquires biological information from the biological information acquisition unit 114, removes outliers from the biological information, interpolates data, and the like to shape the data, and calculates a predetermined index value (for example, frequency decomposition of an interval, HRV, and the like).

Then, the precursor detection unit 115 detects a precursor of the physical condition abnormality using the precursor model (step S003). Specifically, the precursor detection unit 115 detects a precursor by determining whether or not the biological information acquired by the biological information acquisition unit 114 corresponds to a precursor of a predetermined physical condition abnormality using a learning completion model (precursor model) obtained by learning, as teaching data, precursor data of the corresponding biological information for each physical condition abnormality name 201 of the precursor model storage unit 200.

Then, the precursor detection unit 115 determines whether or not a precursor is detected (step S004). Specifically, the precursor detection unit 115 determines that a precursor is detected when a precursor of each physical condition abnormality is detected from the precursor model of each physical condition abnormality. However, the present invention is not limited to this, and the detection of the precursor may be determined based on whether or not a combination of detection of some physical condition abnormalities satisfies a predetermined condition. If no precursor is detected (no in step S004), the precursor detection unit 115 ends the precursor detection process.

When the precursor is detected (yes in step S004), the precursor detection unit 115 notifies the precursor by notification, display, alarm sound, mail, message, or the like (step S005). Specifically, the precursor detection unit 115 causes the notification unit 116 to notify that a precursor is detected.

The above is an example of the flow of the precursor detection process. By adopting the precursor detection process, it is possible to more appropriately estimate a precursor of a physical condition abnormality which is a state that may affect the user operation.

In the precursor detection process described above, the binary judgment of whether the precursor is detected or not is performed, but the present invention is not limited to this, and the judgment may be performed in a plurality of stages using the likelihood of the precursor model. For example, the wakefulness may be determined in a plurality of stages.

Fig. 8 is a diagram showing an example of the awake abnormality notification screen. The awake anomaly notification screen 600 is an example of the screen output in step S005 of the precursor detection process.

The awake anomaly notification screen 600 includes a so-called navigation screen including a surrounding map centered on a marker 601 indicating the position of the host vehicle and a recommended route display 602 indicating a predetermined travel road.

The navigation screen is a screen of application software that is started up when the state estimation device 100 is started up. Therefore, the navigation screen is not limited to the navigation screen, and may be a screen of other application software. For example, the screen may be a moving image display screen, an operation screen for playing music, a menu screen or a setting screen of the state estimating device 100 indicated by the basic control unit 111, or the like.

In the awake anomaly notification screen 600, a message box 603 for displaying a message is displayed, and a sound 604 of the message of the same content is output. The message is intended to be, for example, "predict you to get trapped somewhat. Is rest? In that way, the "precursor of the detected physical condition abnormality", the physical condition abnormality in which the precursor is detected, and the action of coping with the onset of the physical condition abnormality "are included.

In addition, the message box 603 includes a button for receiving an objection to the notified content input. For example, in the case of the wakefulness abnormality notification screen 600 in which a precursor of drowsiness is detected, an area for receiving an input of an objection of "no-drowsiness" is included.

When a touch input is performed to a region where an objection input is received, if no physical condition abnormality related to the precursor occurs within a predetermined period thereafter, the precursor is considered to be a false detection, and a correction process to be described later for changing teaching data can be performed.

Fig. 9 is a diagram showing an example of a flow of the physical condition abnormality detection process. When the state estimating device 100 is started, the physical condition abnormality detection process is started at a predetermined interval (for example, once every 1 second, once every 5 seconds, or once every 1 minute).

First, the state detection section 117 acquires state (face image) information (step S101). Specifically, the state detecting section 117 establishes communication with the state detecting device 12, and performs continuous communication or intermittent communication to acquire information of a face image as state information acquired by the state detecting device 12, and stores it in the RAM 22 or the storage device 3 together with history information for a certain period.

Then, the state detection unit 117 shapes the state (face image) information (step S102). Specifically, the state detection unit 117 performs noise removal, data interpolation, and the like on the state information to shape the image.

Then, the state detection unit 117 determines whether or not a certain physical condition abnormality is satisfied (step S103). Specifically, the state detection unit 117 acquires, from the state detection device 12, the blink frequency, the yawning, the eye closure time ratio per unit time (eye opening time ratio), the micro saccade (involuntary movement of the eyeball), the head shake, and the like, and the result of the judgment as to whether or not the symptoms corresponding to the physical condition abnormality are present, and judges whether or not a certain physical condition abnormality is satisfied. If the body condition is not abnormal (no in step S103), the state detection unit 117 ends the body condition abnormality detection process.

If the physical condition is abnormal (yes in step S103), the state detection unit 117 outputs a predetermined signal corresponding to the physical condition abnormality (step S104). Specifically, the state detection unit 117 outputs a signal corresponding to the content of the physical condition abnormality, that is, the physical condition abnormality that has occurred, as a detection signal of the physical condition abnormality to the teaching data generation unit 118 or the like.

The above is the flow of the physical condition abnormality detection process. With the physical condition abnormality detection processing, the teaching data generation unit 118 can be notified of the physical condition abnormality using the state information obtained from the state detection device 12.

Fig. 10 is a diagram showing an example of a flow of the teaching data generation process. When the state estimating device 100 is started, the teaching data generating process is started at predetermined intervals (for example, at predetermined intervals such as once every 1 second, once every 5 seconds, or once every 1 minute).

First, the biological information acquiring unit 114 acquires biological information (step S201). Specifically, the biological information acquiring unit 114 establishes communication with the biological information acquiring apparatus 10, and performs continuous communication or intermittent communication to acquire information that is a pulse wave of biological information acquired by the biological information acquiring apparatus 10, and stores it in the RAM 22 or the storage apparatus 3 together with history information for a predetermined period (for example, a period from 40 minutes before the processing time to the processing time).

Then, the precursor detector 115 calculates a predetermined index value of the biological information (step S202). Specifically, the precursor detection unit 115 acquires biological information from the biological information acquisition unit 114, performs outlier removal, data interpolation, and the like on the biological information, performs data shaping, and calculates a predetermined index value (for example, frequency decomposition of an interval, heart rate, HRV, and the like).

Then, the teaching data generation unit 118 determines whether or not the calculated index value deviates from the standard range (step S203). For example, the teaching data generation unit 118 determines whether or not the calculated index value deviates from the normal health state.

When the calculated index value deviates from the standard range (yes in step S203), the biological information acquiring unit 114 discards the biological information and gives an instruction to the notifying unit 116 to notify that the index value deviates from the standard range by notification, display, alarm sound, mail, message, or the like (step S204). Accordingly, the storage capacity of the storage device 3 can be prevented from being compressed, and the estimation accuracy of the precursor can be prevented from being lowered.

When the calculated index value does not deviate from the standard range (no in step S203), the teaching data generating unit 118 determines whether or not the amount of biological information (teaching data length) necessary for composing the teaching data is insufficient (step S205). Specifically, the teaching data generating unit 118 determines whether or not the data length of the history information of the pulse wave stored by the biological information acquiring unit 114 in step S201 is shorter than the data length (recording time, sampling period) 303 of the teaching data corresponding to the abnormal physical condition. When the amount of biological information (teaching data length) required to construct the teaching data is insufficient (yes in step S205), the teaching data generation unit 118 stores the acquired biological information (step S2051), and returns the control to step S201. The storage location of the biometric information is, for example, a temporary memory (not shown).

When the amount of biological information (teaching data length) necessary for composing the teaching data is not insufficient (no in step S205), the teaching data generating unit 118 determines whether or not a signal indicating that the physical condition is abnormal is detected (step S206). Specifically, the teaching data generation unit 118 determines whether or not a detection signal of a physical condition abnormality is output from the state detection unit 117. If not detected (no in step S206), the teaching data generation unit 118 advances control to step S208.

When the signal of the physical condition abnormality is detected (yes in step S206), the teaching data generating unit 118 gives a flag corresponding to the physical condition abnormality to the history of the biological information (step S207). Specifically, the teaching data generation unit 118 reads the biometric information from a temporary memory, not shown, and stores the tag in association with the biometric information ID. The biometric information ID is, for example, a hash value of the biometric information. The teaching data generation unit 118 associates biological information to give a mark corresponding to a signal of abnormal physical condition to the teaching data.

Then, the teaching data generation unit 118 determines whether or not the learning ratio of the marked data is smaller than a predetermined value (step S208). Specifically, the teaching data generating unit 118 refers to the learning proportion (mark/total number) 502 of the marked data of the abnormal physical condition according to the state, and determines whether or not the learning proportion is smaller than 45% (percentage), for example. When the learning ratio of the marked data is smaller than the predetermined value (yes in step S208), the teaching data generating unit 118 advances control to step S210.

When the learning rate of the marked data is not less than the predetermined value (no In step S208), the teaching data generating unit 118 deletes the two earliest pieces of data (FIFO: first In, first Out) having the same mark from the existing teaching data (step S209).

For example, in the case where the total number of pieces of teaching data is 100, where the number of pieces of marked data is 50 (50%), and the threshold value of the marked data learning ratio is 45%, the teaching data generating section 118 deletes the two earliest pieces of data from the marked data, and the total number of pieces of teaching data is 98, and the number of pieces of marked data is 48.

The teaching data generation unit 118 then stores the history of the biological information for the predetermined period together with the mark as teaching data (step S210). Specifically, the teaching data generation unit 118 cuts out the biological information of the data length required for the teaching data in the biological information acquired in step S201, obtains the hash value of the biological information as the biological information ID, and stores 1 data in the teaching data storage unit 400 with the marking information if the marking information is added to the biological information.

Then, the teaching data generation unit 118 recalculates the marked data learning scale. For example, when the teaching data to which the mark is given is additionally stored as 1 data, the total number of teaching data is 99, and the number of marked data is 49, and the marked data learning ratio is 49/99=about 49.4%. The teaching data generation unit 118 stores the teaching data in the teaching data learning ratio storage unit 500.

The above is an example of the flow of the teaching data generation processing. By adopting the teaching data generation processing, when new biological information is acquired, teaching data including biological information with a mark (to which a mark is given), that is, biological information that is a precursor when a physical condition abnormality occurs and biological information that is not when a physical condition abnormality occurs, at a predetermined ratio (approximately 1:1) can be generated.

Further, by adopting the physical condition abnormality detection processing and the teaching data generation processing, the occurrence of the physical condition abnormality can be detected using the state detection device as a sensor different from the biological information acquisition device, teaching data in which biological information is marked can be generated, and a precursor that is difficult to judge from only biological information can be detected using the learning completion model in which machine learning is performed.

The teaching data generation process may be executed in real time, but is not limited to this, and for example, batch processing may be performed intensively at the time of ending the process of the state estimating device 100.

Fig. 11 is a diagram showing an example of a flow of training processing. The training process starts when the state estimation device 100 is started. Alternatively, the training process may be performed at an irregular period, such as when the processing load of the state estimation device 100 is equal to or less than a predetermined value.

First, the precursor model generating unit 119 calculates index values of biological information for the generated teaching data, respectively (step S301). More specifically, the precursor model generating unit 119 extracts biological information from the teaching data by a predetermined method, removes outliers from the biological information, interpolates the data, and the like to shape the data, and calculates a predetermined index value (for example, frequency decomposition of an interval, HRV, and the like).

Then, the precursor model generation unit 119 performs training of the precursor model using the index value of the biological information. Specifically, the precursor model generation unit 119 uses a conventional precursor model as a training target, and causes the precursor detection unit 115 to estimate a solution of the biological information extracted in step S301 (for example, a physical condition abnormality having a precursor), thereby obtaining an estimated solution. For example, the precursor model generation unit 119 inputs the index value of the biological information extracted in step S301 into the precursor model (step S302). The input biological information is propagated by internal parameters of the precursor model, and the precursor model outputs an estimation result. For example, when the precursor model outputs "0.8" as the estimated result and the flag of the biological information to which the teaching data is given is "1", the difference between the estimated result of the precursor model and the actual flag is "1-0.8", that is, "0.2". When the difference between the estimated result of the precursor model and the actual flag is smaller than the predetermined value (no in step S303), the precursor model generating unit 119 ends the training process without changing the internal parameters of the precursor model.

When the difference between the estimated result of the precursor model and the actual flag is equal to or greater than the predetermined value (yes in step S303), the precursor model generating unit 119 changes the internal parameters of the precursor model (step S304). For example, the precursor model generation section 119 can change the internal parameters of the precursor model using an error back propagation method. If necessary, the precursor model generation unit 119 may select the same type of biological information as that used to generate the precursor model, but not used to generate the model, input the biological information to the precursor model, and change the internal parameters of the precursor model based on the difference between the estimated result output from the precursor model and the mark actually added. After a certain biological information is thus input to change the internal parameters of the precursor model, a process of inputting other biological information other than the biological information to change the internal parameters of the precursor model is called a cycle. The number of cycles may be appropriately changed according to the size of the learning data and the composition of the model.

Next, the model update process is explained. Fig. 12 is a diagram showing an example of a flow of the model update processing. The purpose of the model update process is to select an appropriate model from among a plurality of models generated by performing a plurality of training processes. The model update process starts when the state estimation device 100 is started. Alternatively, the training process may be started after the training process is performed.

First, the precursor model generation unit 119 determines whether or not a plurality of models are stored based on the data stored in the precursor model storage unit 200 (step S311). When the plurality of precursor models are not stored in the precursor model storage unit 200 (no in step S311), the precursor model generation unit 119 ends the model update process.

When a plurality of precursor models are stored in the precursor model storage unit 200 (yes in step S311), the precursor model generation unit 119 inputs index values based on the biological information into the plurality of precursor models, respectively (step S312). In this process, the index values input to the precursor models are the same data. The biological information input into the precursor models is teaching data that has been stored, and is data that is not used for training of each precursor model.

Then, the precursor model generation unit 119 calculates the accuracy of each of the plurality of precursor models to which the same data is input, and compares them (step S313). As a calculation method of the accuracy comparison, for example, a method using a correct answer rate and a method using a reproduction rate may be employed.

Then, the precursor detection unit 115 is set so that the precursor detection unit 115 performs precursor detection processing using the precursor model determined to be more accurate in step S313 (step S314).

The above is an example of the flow of the training process and the model update process. With the training process and the model updating process, when new biological information is acquired, a precursor model that can estimate a precursor with higher accuracy can be generated using teaching data including the biological information to which a mark is given. The training process may be performed in real time, or may be performed in a batch process in a concentrated manner at the end of the process of the state estimating device 100, in addition to the above-described time.

Fig. 13 is a diagram showing an example of a flow of correction processing. When it is determined to be yes in step S004, the correction processing is started.

First, the teaching data generation unit 118 receives an objection input for precursor detection (step S401). Specifically, when the awake anomaly notification screen 600 is output in step S005 of the precursor detection process, the teaching data generation unit 118 receives a touch input to the area where the anomaly "no-trouble" input is received.

Then, the teaching data generation unit 118 determines whether or not a physical condition abnormality has occurred within a predetermined period (step S402). Specifically, after receiving the objection input in step S401, the teaching data generating unit 118 starts counting the predetermined period. During the time counting period, it is monitored whether or not a predetermined signal corresponding to the physical condition abnormality is output in the physical condition abnormality detection process.

When the physical condition abnormality occurs within the predetermined period (yes in step S402), the teaching data generating unit 118 ends the correction process. This is because the objection is incorrect, is a condition in which symptoms of abnormal physical condition should be detected, and does not require a change in the precursor model.

When no physical condition abnormality occurs within a predetermined period (no in step S402), the teaching data generating unit 118 generates teaching data by changing the biological information of the detected precursor to teaching data to which no mark is given (step S403). This is because the objection is correct, and not a condition in which symptoms of abnormal physical condition should be detected, it is necessary to adjust the precursor model according to individuality (specific case).

The above is an example of the flow of the correction processing. With this correction processing, the model can be corrected to be suitable for the user, and the precursor model for coping with the biological information based on the individual differences such as physique among the users can be personalized from the common model, and the detection accuracy of the precursor can be improved.

The above is the state estimating device 100 to which the embodiment of the present invention is applied. The state estimating device 100 can more appropriately estimate a precursor of a state that may affect the operation of the user.

However, the present invention is not limited to the above embodiment. The above-described embodiments may be variously modified within the scope of the technical idea of the present invention. For example, the above-described embodiment performs the precursor detection process, the physical condition abnormality detection process, the teaching data generation process, the training process, the model update process, and the correction process, but is not limited thereto. Any of the processes may be implemented alone or may be implemented in combination.

Fig. 14 is a diagram showing an example of a configuration to which the state estimating device according to the second embodiment is applied. The state estimating device 100' to which the second embodiment is applied is basically the same as the state estimating device 100 to which the above embodiment is applied, but some differences exist. The explanation will be centered on this difference.

The state estimating device 100' to which the second embodiment is applied does not include the state detecting device 12. In addition, the storage device 3 does not include the teaching data storage unit 400 and the teaching data learning ratio storage unit 500. The controller 1 performs precursor detection processing, but does not perform physical condition abnormality detection processing, teaching data generation processing, training processing, model update processing, and correction processing.

That is, instead of relearning based on actual data of the precursor model, the learned precursor model is used to detect precursors from the acquired biological information. Accordingly, various costs such as hardware costs and software costs can be suppressed, and a precursor of a state that may affect the operation of the user can be estimated appropriately.

Fig. 15 is a diagram showing an example of teaching data setting information in the third embodiment. The state estimating device according to the third embodiment is applied to generate teaching data and a precursor model with higher accuracy in order to detect a state with higher accuracy in order to detect precursor of a plurality of abnormal physical conditions at the same time,

in order to achieve the above object, the teaching data setting information 900 sets the state detection device 902, the biological information acquisition device 903, and the teaching data use site 904 to be used for each physical condition abnormality name 901. Specifically, the teaching data use portion 904 is a period determined based on the physical condition abnormality detection time and a time from which a predetermined time is traced back. For example, the term "low wakefulness (drowsiness)" means "a period of time other than 5 minutes before detection within 30 minutes before detection", and the term "epilepsy" means "a period of time of 4 minutes before and after 8 minutes before detection". The teaching data setting information 900 is stored in the storage device 3, and the state detection unit 117 of the controller 1 refers to the state detection device 902 to determine the state detection device to be used for each physical condition abnormality, and the biological information acquisition unit 114 refers to the biological information acquisition device 903 to determine the biological information acquisition device to be used for each physical condition abnormality.

Then, the precursor detection unit 115 runs 1 or more precursor models in parallel in accordance with the acquired biological information, and detects the precursor using a plurality of learning completion models in which a plurality of physical condition abnormalities are learned, respectively. Thus, the optimum precursor detection can be performed in response to each abnormal physical condition, and the precursor detection can be performed with higher accuracy.

The teaching data generation unit 118 refers to the teaching data use portion 904 and specifies the portion of the biological information to be used for each abnormal physical condition. Thus, information more suitable for the physical condition abnormality can be used simultaneously for a plurality of physical condition abnormalities, and precursor detection can be performed with higher accuracy.

In the above embodiments, when the signal of the physical condition abnormality is not detected, the signal is processed as a case where the mark is not given to the teaching data, but the present invention is not limited to this, and a mark other than the physical condition abnormality may be given. For example, in the case where the biological information is not a precursor indicating a predetermined physical condition abnormality, that is, it is considered that the biological information is healthy without the physical condition abnormality, the virtual variable may be set to the marker 402. Specifically, the virtual variable is represented by "0" and "1" in two states, and the virtual variable "1" is used when representing a precursor of a physical condition abnormality, and the virtual variable "0" is used when not representing a precursor of a physical condition abnormality.

When the virtual variable is used for the marker 402 in this way, for example, when yes in step S402 of the correction process (when a physical condition abnormality occurs within a predetermined period), the teaching data generating unit 118 assigns a virtual variable "1" to the marker 402, and ends the correction process.

In addition, when "no" is performed in step S402 of the correction process (when no physical condition abnormality occurs within a predetermined period), the teaching data generating unit 118 generates teaching data by changing the biological information in which the precursor is detected to the teaching data to which the virtual variable "0" is given to the flag in step S403.

As described above, by adding the mark data indicating this meaning to the teaching data even when the physical condition is not abnormal, unexpected learning results due to the lack of the mark data can be prevented.

Claims

1. A state estimating device, comprising:

a biological information acquisition unit for acquiring biological information of a user;

a precursor detection unit that uses a learning completion model in which the biological information related to a predetermined physical condition abnormality is learned as teaching data, and determines whether or not the biological information of the user matches the predetermined physical condition abnormality, thereby detecting a precursor;

A teaching data generation unit that generates information including biological information of the user as the teaching data;

a precursor model generating unit that causes the learning completion model to learn the teaching data generated by the teaching data generating unit; and

a state detection unit configured to detect an onset state of the predetermined physical condition abnormality of the user,

when the state detection unit detects the seizure state, the teaching data generation unit generates the teaching data using first information and second information, wherein the first information is information obtained by adding a mark to the biological information of the user in a predetermined period before the seizure state, and the second information is the biological information of the user in the predetermined period when the seizure state is not detected.

2. The state estimating device according to claim 1, wherein:

the predetermined physical condition abnormality is a physical condition abnormality that affects the exercise operation of the user.

3. The state estimating device according to claim 2, wherein:

the abnormal physical condition that affects the exercise action of the user is a state in which the user feels drowsy or a state in which seizures occur.

4. The state estimation device according to any one of claims 1 to 3, characterized in that:

the biological information acquisition unit acquires information of a pulse wave as the biological information.

5. The state estimation device according to any one of claims 1 to 3, characterized in that:

the teaching data generation unit generates the teaching data such that the first information and the second information are in a predetermined ratio.

6. The state estimation device according to any one of claims 1 to 3, characterized in that:

the teaching data generating unit determines the predetermined period in response to the abnormal physical condition.

7. The state estimation device according to any one of claims 1 to 3, characterized in that:

the teaching data generating unit generates teaching data in which biological information in which a precursor is detected is used as second information when the state detecting unit does not detect the attack state for a predetermined period after receiving an input indicating an objection to the precursor detected by the precursor detecting unit.

8. The state estimation device according to any one of claims 1 to 3, characterized in that:

also includes a notification unit for notifying abnormality by a predetermined method,

The biological information acquisition unit causes the notification unit to notify and delete the biological information when the acquired biological information deviates from a predetermined range.

9. The state estimating device according to claim 1, wherein:

the precursor detection unit detects precursors of the predetermined physical condition abnormalities using a plurality of learning completion models each of which learns the predetermined physical condition abnormalities.

10. A computer-readable storage medium storing a program for causing a computer to execute a state estimation process, the storage medium characterized by:

when the program is executed, the computer is caused to function as a control device,

causing the control device to execute the steps of:

a biological information acquisition step of acquiring biological information of a user;

a precursor detection step of detecting a precursor by determining whether or not the biological information of the user matches the precursor of the predetermined physical condition abnormality, using a learning completion model in which the biological information related to the predetermined physical condition abnormality is learned as teaching data;

a teaching data generation step of generating information including biological information of the user as the teaching data;

A precursor model generating step of causing the learning completion model to learn the teaching data generated in the teaching data generating step; and

a state detection step of detecting an onset state of the predetermined physical condition abnormality of the user,

when the seizure state is detected in the state detection step, the teaching data is generated in the teaching data generation step using first information and second information, wherein the first information is information in which a mark is given to the biological information of the user in a predetermined period before the seizure state, and the second information is the biological information of the user in the predetermined period in the case where the seizure state is not detected.

11. A state estimation method for causing a computer to execute a state estimation flow, the computer including a control device, the state estimation method characterized by causing the control device to execute: