JP2007122362A - State estimation method using neural network and state estimation apparatus using neural network - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a state estimation method using a neural network and its state estimation apparatus capable of highly precise estimation. <P>SOLUTION: The state estimation method using the neural network includes: an initial learning step for learning the neural network on the basis of input data and teacher data corresponding to a class label of the input data; a virtual teacher data assignment step for assigning virtual teacher data corresponding to the virtual class label to evaluation data whose class label is unknown; an evaluation step for learning the neural network on the basis of the evaluation data and the virtual teacher data of the evaluation data to evaluate convergence property of a learning curve of the learning; and a class label estimation step for estimating as a class label of the evaluation data a virtual class label corresponding to the virtual teacher label of the learning curve which has the highest convergence property out of a plurality of learning curves every virtual teacher data of various kinds assigned to the evaluation data. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a state estimation method using a neural network and a state estimation apparatus using a neural network for estimating a driver state and the like with high accuracy.

Various methods for estimating a driver's psychological state, physical state, and driving skill have been proposed in order to provide driving assistance according to the driver state. As this estimation method, it is a method using a neural network (see Patent Document 1). In this method, teacher data corresponding to a class label indicating a driver state such as impatience and fatigue is set for input data composed of a combination of various information such as vehicle information, and learning data composed of the input data and the teacher data is used. Learning is performed and a neural network is constructed in advance so that the error between the teacher data and the output data is sufficiently converged. Then, evaluation data composed of a combination of vehicle information obtained while the vehicle is traveling is input to the neural network, and a driver state for the evaluation data is estimated by feedforward calculation using the neural network.
Japanese Patent Laid-Open No. 10-324175

  The driver state varies greatly depending on personal characteristics and driving conditions. Therefore, complete learning that covers all the patterns is impossible, and it is impossible to construct in advance a neural network corresponding to various conditions of various drivers. Therefore, when the learning data used when learning is different from the actually obtained evaluation data (that is, when evaluation data with insufficient learning is input), the estimation accuracy of the driver state decreases. In order to improve the estimation accuracy, when the neural network is tuned according to the personal characteristics of the driver, re-learning must be performed using correct learning data. Therefore, every time evaluation data is input, the driver needs to set the current driver state as teacher data, which is very time-consuming and burdens the driver.

  Therefore, an object of the present invention is to provide a state estimation method and a state estimation apparatus using a neural network capable of highly accurate estimation.

  The state estimation method using a neural network according to the present invention includes an initial learning step for learning a neural network based on input data and teacher data corresponding to the class label of the input data, and evaluation data with an unknown class label. Based on the virtual teacher data assignment step for assigning virtual teacher data corresponding to the virtual class label and the evaluation data and the virtual teacher data of the evaluation data, the neural network is learned, and the convergence characteristics of the learning curve of the learning are evaluated. And the virtual class label corresponding to the virtual teacher data of the learning curve having a high convergence characteristic among the plurality of learning curves for each of the plurality of different virtual teacher data assigned to the evaluation data is estimated as the class label of the evaluation data And a class label estimating step.

  In this state estimation method using a neural network, in an initial learning step, learning is performed based on learning data composed of input data and teacher data corresponding to the class label of the input data, and a neural network based on initial learning is constructed. When evaluation data with an unknown class label is input after constructing the initial learning neural network, in the state estimation method, virtual teacher data corresponding to the virtual class label is assigned to the evaluation data in the virtual teacher data allocation step. assign. At this time, different virtual teacher data (virtual class labels) are assigned to the evaluation data. In the state estimation method, in the evaluation step, for each combination of the evaluation data and the virtual teacher data of the evaluation data, learning is performed using the evaluation data and the virtual teacher data as learning data, and the convergence characteristics of the learning are evaluated. A learning curve (for example, a learning error for which the square error between the virtual teacher data and the output data in learning at that learning number is matched) is generated, and the convergence characteristic of the learning curve is evaluated . Therefore, a plurality of learning curves corresponding to the number of virtual teacher data are generated for the evaluation data. The more the virtual teacher data assigned to the evaluation data is valid teacher data (the virtual class label is the valid class label for the evaluation data), the smoother the learning curve converges. . Further, in the state estimation method, in the class label estimation step, a learning curve having a high convergence characteristic is extracted from a plurality of learning curves generated for the evaluation data, and the virtual curve corresponding to the virtual teacher data of the extracted learning curve is extracted. Is the class label for the evaluation data. That is, the class label having the highest validity for the evaluation data is estimated. As described above, in this state estimation method, a plurality of virtual teacher data (virtual class labels) are set for the evaluation data instead of one evaluation for the evaluation data by the feedforward calculation of the learned neural network. Since the evaluation is performed a plurality of times based on the temporal process in learning (the convergence characteristic of the learning curve), the estimation accuracy of the class label for the evaluation data is high. Furthermore, since the convergence of global learning in each learning process for multiple virtual class labels is used as a judgment index, the estimation accuracy is high and robustness can be obtained even with respect to noise and some data loss in the evaluation data. high. Also, in this state estimation method, evaluation is performed by automatically assigning a plurality of virtual teacher data to the evaluation data, so that the user does not need to set the teacher data every time the evaluation data is input. Do not put a burden.

  The state estimation method using the neural network of the present invention may include a class label output step for outputting the class label of the evaluation data estimated in the class label estimation step.

  In this state estimation method using a neural network, a class label estimated with respect to evaluation data is output as an estimation result in a class label output step. As a result, it is possible to utilize the estimation result such as informing the user of the output class label or performing relearning using the evaluation data and the class label as learning data.

  The state estimation method using the neural network of the present invention includes an additional learning step of learning the neural network using virtual teacher data corresponding to the class label of the evaluation data estimated in the class label estimation step as teacher data of the evaluation data. It is good also as a structure.

  In this state estimation method using a neural network, in the additional learning step, virtual teacher data corresponding to the estimated class label is used as teacher data of the evaluation data, and additional learning is performed based on the learning data including the evaluation data and the teacher data. And update the neural network. In this way, in this state estimation method, additional learning using evaluation data that does not have teacher data that is input after the construction of a neural network by initial learning is possible, so that a neural network with high estimation accuracy corresponding to the user's personal characteristics, etc. Can be tuned to the network.

  In the state estimation method using the neural network of the present invention, the evaluation step may be configured to learn the neural network based on the evaluation data, the virtual teacher data of the evaluation data, and the input data and the teacher data of the input data.

  In the evaluation step of the state estimation method using this neural network, when performing learning for generating a learning curve, in addition to virtual learning data of the evaluation data and its virtual teacher data, input data and its teacher data Learning is performed based on the learning data. As described above, in this state estimation method, learning is also performed using correctly labeled learning data and the convergence characteristics of the learning curve are evaluated, so that the accuracy of class label estimation for the evaluation data is further improved.

  In the state estimation method using the neural network of the present invention, the input data and the evaluation data may be values detected by the driving state detection means, and the class label may be a driver state.

  In this state estimation method using a neural network, input data and evaluation data consist of various information obtained while driving the vehicle detected by the driving state detection means, and the class label indicates various states of the driver while driving the vehicle. Yes, the driver state can be estimated with high accuracy from the evaluation data obtained during driving. The driving state detection means includes, for example, driver information detection means such as the driver's face orientation, line of sight, motion, physiological information (for example, heart rate), vehicle surroundings such as inter-vehicle distance, lane, pedestrian, and traffic jam information. Information detection means, vehicle speed, rudder angle, accelerator opening, degree of brake depression, front / rear G, lateral G, and other vehicle information detection means. The driver state includes, for example, the driver's psychological state such as impatience and rush, the driver's physical state such as fatigue and arousal level, and the driver's driving skill such as beginner's driving and skilled driving.

  In the state estimation method using the neural network of the present invention, the neural network is a three-layer hierarchical neural network, the correspondence between the input data and the teacher data of the input data, the evaluation data and the virtual teacher data of the evaluation data, Or a correspondence relationship between the evaluation data and the teacher data estimated with respect to the evaluation data may be learned by the error back propagation method, and a learning curve may be generated based on the error by the error back propagation method.

  The state estimation device using a neural network according to the present invention includes initial learning means for learning a neural network based on input data and teacher data corresponding to the class label of the input data, and evaluation data having an unknown class label. Based on the virtual teacher data allocating means for allocating virtual teacher data corresponding to the virtual class label and the evaluation data and the virtual teacher data of the evaluation data, the neural network is learned, and the convergence characteristics of the learning curve of the learning are evaluated. And a virtual class label corresponding to the virtual teacher data of the learning curve having a high convergence characteristic among a plurality of learning curves for each of a plurality of different virtual teacher data assigned to the evaluation data is estimated as a class label of the evaluation data Class label estimating means.

  The state estimation apparatus using the neural network of the present invention may include a class label output unit that outputs a class label of evaluation data estimated by the class label estimation unit.

  The state estimation apparatus using the neural network of the present invention includes additional learning means for learning the neural network using the virtual teacher data corresponding to the class label of the evaluation data estimated by the class label estimation means as the teacher data of the evaluation data. It is good also as a structure.

  In the state estimation apparatus using the neural network of the present invention, the evaluation means may be configured to learn the neural network based on the evaluation data, the virtual teacher data of the evaluation data, and the input data and the teacher data of the input data.

  In the state estimation device using the neural network according to the present invention, the input data and the evaluation data may be values detected by the driving state detecting means, and the class label may be a driver state.

  In the state estimation device using the neural network according to the present invention, the neural network is a three-layer hierarchical neural network, the correspondence between the input data and the teacher data of the input data, the evaluation data and the virtual teacher data of the evaluation data, Or a correspondence relationship between the evaluation data and the teacher data estimated with respect to the evaluation data may be learned by the error back propagation method, and a learning curve may be generated based on the error by the error back propagation method.

  The state estimation device using each neural network described above has the same operational effects as the state estimation method using each neural network described above.

  According to the present invention, it is possible to perform highly accurate estimation in class label estimation for evaluation data using a neural network.

  Embodiments of a state estimation method using a neural network and a state estimation apparatus using a neural network will be described below with reference to the drawings.

  In the present embodiment, the present invention is applied to a driver state identification device that is mounted on a vehicle and identifies a driver state. In the driver state identification device according to the present embodiment, various information necessary for identifying the driver state is obtained by a sensor or the like, and the driver state (class label) for the evaluation data including the information obtained by the neural network classifier Identify In the driver state identification device according to the present embodiment, driving safety according to the driver state is determined, and various driving assistances for improving driving safety are performed.

  The neural network classifier used in the present embodiment is a three-layer hierarchical neural network, and is constructed by learning using an error back propagation method. Information necessary for identifying the driver state includes various information obtained from the driver, various information obtained from the environment around the vehicle, and various information obtained from the vehicle. A combination of these various information is input to the neural network classifier as evaluation data. Examples of the driver state to be identified include psychological states such as impatience and rush, physical state such as arousal level and fatigue, driving skills such as elementary driving and skilled driving.

  The driver state identification device 1 will be described with reference to FIGS. FIG. 1 is a configuration diagram of a driver state identification device according to the present embodiment. FIG. 2 is a conceptual diagram of an identification method in the driver state identification device of FIG. FIG. 3 is an example of evaluation data. FIG. 4 is an example of temporary class data and temporary teacher data corresponding to the temporary class label. FIG. 5 is an example when temporary teacher data (temporary class label) is assigned to evaluation data.

  The driver state identification device 1 identifies a driver state with a neural network classifier and performs various driving assistances according to the driver state. In particular, the driver state identification device 1 sets a plurality of provisional class labels for evaluation data obtained during driving of the vehicle and performs provisional learning for each provisional class label in order to improve the identification accuracy of the driver state. Based on all the learning curves, the temporary class label that is most appropriate for the evaluation data is identified as the class label of the evaluation data. Further, the driver state identification device 1 performs re-learning using learning data including the class label identified as the evaluation data, and tunes the neural network classifier according to the characteristics of the driver. For this purpose, the driver state identification device 1 includes a driver state recognition unit, an environment information recognition unit, a vehicle information recognition unit, a display 40, a speaker 41, a driving support system 42, a neural network identification ECU [Electronic Control Unit] 50 (hereinafter referred to as the following). And a driver state adaptive driving support ECU 60 (hereinafter referred to as “driving support ECU 60”).

  In the present embodiment, each process performed by the identification ECU 50 corresponds to the initial learning means, virtual teacher data allocation means, evaluation means, class label estimation means, class label output means, and additional learning means described in the claims. The driver state recognition unit, the environment information recognition unit, and the vehicle information recognition unit correspond to the driving state detection unit described in the claims.

  As the driver state recognition means, there are a face orientation / line-of-sight recognition sensor 10, a motion / posture recognition sensor 11, a face / eyeball recognition sensor 12, a foot recognition sensor 13, a heart rate sensor 14, and the like. The face direction / line-of-sight recognition sensor 10 is a sensor that recognizes the driver's face direction and line of sight, and transmits the recognition information to the identification ECU 50. The motion / posture recognition sensor 11 is a sensor that recognizes the driver's motion and posture, and transmits the recognition information to the identification ECU 50. The face / eyeball recognition sensor 12 is a sensor that recognizes the driver's face and eyeball, and transmits the recognition information to the identification ECU 50. The foot recognition sensor 13 is a sensor that recognizes the motion and posture of the driver's foot, and transmits the recognition information to the identification ECU 50. The heart rate sensor 14 detects the heart rate as the physiological information of the driver, and transmits the detected information to the identification ECU 50.

  Note that the face orientation / line-of-sight recognition sensor 10, the motion / posture recognition sensor 11, the face / eyeball recognition sensor 12, and the foot recognition sensor 13 may be configured by each sensor alone, or the face of the driver, the entire body, You may comprise each camera which images a step, ECU etc. which perform image processing. Here, only the heart rate is illustrated as the physiological information of the driver. However, the brain wave, the respiratory rate, the body temperature, the number of blinks, and the like may be acquired.

  The environmental information recognition means includes an inter-vehicle distance sensor 20, a lane recognition sensor 21, a traffic signal recognition sensor 22, a sign recognition sensor 23, a temporary stop line recognition sensor 24, a pedestrian recognition sensor 25, a traffic environment information acquisition communication device 26, a car. There is a navigation system 27 and the like. The inter-vehicle distance sensor 20 is a sensor that detects the inter-vehicle distance from the preceding vehicle, and transmits the detection information to the identification ECU 50. The lane recognition sensor 21 is a sensor for recognizing a traveling lane, and transmits the recognition information to the identification ECU 50. The traffic light recognition sensor 22 is a sensor for recognizing a traffic light existing in front of the vehicle, and transmits the recognition information to the identification ECU 50. The sign recognition sensor 23 is a sensor for recognizing a sign existing in front of the vehicle, and transmits the recognition information to the identification ECU 50. The temporary stop line recognition sensor 24 is a sensor that recognizes a temporary stop line existing in front of the vehicle, and transmits the recognition information to the identification ECU 50. The pedestrian recognition sensor 25 is a sensor that recognizes a pedestrian existing in front of the vehicle, and transmits the recognition information to the identification ECU 50. The communication device 26 for acquiring traffic environment information is a communication device for communicating with roadside stations and communication devices mounted on other vehicles and acquiring traffic jam information, information on other vehicles, etc., and identifying the acquired traffic environment information. It transmits to ECU50. The car navigation system 27 is a system that detects the current position and traveling direction of the vehicle and provides route guidance to the destination, and transmits these pieces of information to the identification ECU 50.

  The inter-vehicle distance sensor 20, the lane recognition sensor 21, the traffic signal recognition sensor 22, the sign recognition sensor 23, the temporary stop line recognition sensor 24, and the pedestrian recognition sensor 25 may be configured as a single sensor or a vehicle. You may comprise by the camera etc. which perform the image processing etc., the camera and radar which image the front.

  As vehicle information recognition means, there are a vehicle speed sensor 30, a steering angle sensor 31, an accelerator opening sensor 32, a brake depression degree sensor 33, a front / rear G sensor 34, a lateral G sensor 35, a vertical G sensor 36, a blinker operation sensor 37, and the like. . The vehicle speed sensor 30 is a sensor that detects the speed of the vehicle, and transmits the detected value to the identification ECU 50. The steering angle sensor 31 is a sensor that detects the steering angle of the steering wheel, and transmits the detected value to the identification ECU 50. The accelerator opening sensor 32 is a sensor that detects the opening of the accelerator pedal, and transmits the detected value to the identification ECU 50. The brake depression degree sensor 33 is a sensor that detects the degree of depression of the brake pedal, and transmits the detected value to the identification ECU 50. The front / rear G sensor 34 is a sensor that detects the front / rear G acting on the vehicle, and transmits the detected value to the identification ECU 50. The lateral G sensor 35 is a sensor that detects the lateral G acting on the vehicle, and transmits the detected value to the identification ECU 50. The vertical G sensor 36 is a sensor that detects the vertical G acting on the vehicle, and transmits the detected value to the identification ECU 50. The turn signal operation sensor 37 is a detection sensor that detects the operation state of the turn signal, and transmits the detected value to the identification ECU 50.

  The display 40 and the speaker 41 are used in common with each system in the vehicle, and the driver state identification device 1 is used when alerting the driver or presenting advice. Examples of the driving support system 42 include a pre-crash safety system, an adaptive cruise control system, and a lane keeping system.

  The identification ECU 50 includes a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], and the like, and constitutes a neural network classifier. The identification ECU 50 does not have a pre-learning process for constructing a neural network classifier based on general learning data before the start of use by the user (before vehicle sales), and no class label after the start of use by the user (after vehicle sales). An identification learning process for identifying an appropriate class label with respect to the evaluation data, a re-learning process for tuning the neural network classifier with new learning data after the user starts use, and the like are performed.

  The neural network classifier has a large number of neurons, and the neurons are connected with weights to form a network structure (neural network). Each neuron in the input layer of the neural network classifier receives data of each information included in the evaluation data, and outputs data indicating a class label from each neuron in the output layer. Neural network classifiers include arousal level models, rush models, and elementary driving models according to class labels, and class labels that can be identified are determined by combinations of input information. For example, when the detected values of vehicle speed, lateral G, heart rate, and inter-vehicle distance are input, five states of polite, normal, impatient, fatigue, and abnormality can be identified as class labels.

  The preliminary learning process will be described. Before the preliminary learning, for example, a large number of drivers (subjects) drive the vehicle and travel on the model course, and the driver information recognition unit, the environment information recognition unit, and the vehicle information recognition unit each time a fixed time passes. Each piece of information described above is acquired, and input data consisting of a combination of some information is collected. Each driver sets the psychological state, physical information, driving skill, and the like when the information is acquired as class labels. At this time, the data is acquired by changing the travel time zone (and hence the traffic volume), the sleeping time of the driver, the work amount before driving (the amount of exercise), and the like. Note that the input data and class labels used for the preliminary learning may be acquired by the experiment running and the setting by the driver as described above, or may be acquired by other methods.

  When each input data and its corresponding class label are input, the identification ECU 50 sets predetermined teacher data for the class label, and generates learning data composed of the input data and the teacher data. The identification ECU 50 inputs the input data to the neural network classifier and performs learning by the error back propagation method, and the error between the output data from each neuron in the output layer of the neural network classifier and the teacher data is sufficiently small. The weight of each neuron is updated so that it becomes (convergence state). In this way, the identification ECU 50 performs learning for all learning data that are correctly labeled, and constructs a basic neural network classifier. As a learning end condition, the learning is performed a predetermined number of times or the square error is a predetermined value or less. Further, as the learning update formula, for example, formula (1) of the improved error back propagation method is used. This improved error backpropagation method adds the inertia term and vibration term to the delta term of the normal error backpropagation method, and the inertial term and the vibration term fall into the local minimum with the function of accelerating learning, respectively. In some cases, it vibrates and probabilistically escapes from it.

  In Equation (1), Δω () is a weight correction amount, c is the number of learnings, ε is a positive learning constant, d is a generalization error, o is output data, and α is It is the proportionality constant of the inertia term, and β is the proportionality constant of the vibration term.

  The identification learning process will be described. With reference to FIG. 2, an outline of the identification learning process and the relearning process will be described. When information is collected by a sensor or the like while driving the vehicle and new evaluation data for identifying the state of the driver is obtained, a class label that may be identified for the evaluation data is assigned a temporary class label (temporary Assigned as teacher data). Accordingly, a plurality of provisional class labels are assigned to certain evaluation data, and a plurality of provisional learning data composed of evaluation data and teacher data is configured. For each provisional class label, learning by the error back propagation method is performed, and a learning curve indicating a change in the square error according to the number of learning is generated. In performing this learning, correctly labeled learning data used in preliminary learning or the like is also used in order to improve identification accuracy. The learning process with the highest convergence characteristics is extracted from the learning curve for each temporary class label, using the temporal process of learning (the convergence of the learning curve) as a decision index, and the temporary class label corresponding to the learning curve is evaluated. Identifies as a data class label. This evaluation data and the identified class label are determined to be valid as learning data. Therefore, using the new learning data composed of the evaluation data and the teacher data of the identified class label, learning is performed by the error back propagation method, and the weight of each neuron of the neural network classifier is updated.

In the identification ECU 50, information is taken from the sensors of the respective recognition means at regular intervals, and the information is combined into evaluation data. Since the evaluation data is data for identifying the driver's psychological state, physical information and driving skill, a large number of samples collected in a predetermined period are required to determine each state of the driver. The predetermined period and sampling period to be collected are determined according to the driver state to be identified, and it is necessary to identify the psychological state and physical state with a short span, so the predetermined period and sampling period are set to a short time, Since the driving skill needs to be identified with a long span, a long time is set for the predetermined period and the sampling cycle. FIG. 3 shows an example of evaluation data, which consists of a vehicle speed, a lateral G, a heart rate, and an inter-vehicle distance as a combination of information, and a vector format of detection values X ₁ , X ₂ , X ₃ , X ₄ of each sensor. Consists of. Evaluation data is collected for each sampling period in a predetermined period, and evaluation data of m samples up to evaluation data 1,..., Evaluation data m is acquired.

  Upon obtaining the evaluation data, the identification ECU 50 assigns a class label that can be identified from information included in the evaluation data to the evaluation data as a temporary class label, and associates the temporary teacher data with the temporary class label. The class label and the teacher data have a one-to-one relationship, and the teacher data is configured in a vector format in which each value is 0/1. FIG. 4 shows an example of a temporary class label and an example of temporary teacher data corresponding to the temporary class label. As the temporary class label, polite, normal, impatience, fatigue, and abnormality are assigned. The temporary teacher data is a case where the number of neurons in the output layer of the neural network classifier is 5, and is represented in a vector format of five 0/1 values. FIG. 5 shows an example in which temporary teacher data (that is, temporary class label) is assigned to the evaluation data shown in FIG. As can be seen from FIG. 5, five temporary teacher data are assigned to evaluation data 1, evaluation data 2,. In this example, there are m × 5 combinations.

  In the identification ECU 50, for each provisional class label, provisional learning data composed of evaluation data and provisional teacher data and learning data correctly labeled (learning data and identification used in the preliminary learning process) Learning is performed using the evaluation data processed in the learning process and the teacher data of the class label identified for the evaluation data), and the weight of each neuron of the neural network classifier is updated. In this learning process, the identification ECU 50 calculates a square error between the temporary teacher data and the output data according to the equation (2) to generate a learning curve every time learning is completed, and the square error is calculated. Is stored in association with the number of learning times. When the learning ECU satisfies the learning termination condition, the identification ECU 50 returns the updated weight of each neuron of the neural network classifier to the value before learning, and performs learning for the next temporary class label. Such processing is performed for all temporary class labels assigned to the evaluation data.

  In equation (2), m is the number of samples of evaluation data (input data), n is the number of neurons in the output layer of the neural network classifier, p is the number of class labels (teacher data), and t Is teacher data, and o is output data. Incidentally, this square error calculation formula is also used for learning in the preliminary learning process and the relearning process.

  When the learning for all the temporary class labels is completed, the identification ECU 50 generates a learning curve for each temporary class label by associating the square error at the learning frequency with the learning frequency. FIG. 2 is a graph showing the learning curve of the evaluation data shown in FIG. 5 and the five assigned temporary class labels. The horizontal axis represents the number of learnings, and the vertical axis represents the square error. Incidentally, since this process is performed in the identification ECU 50, the learning curve is not actually graphed as shown in FIG. 2, and is configured by associating numerical values between the number of learnings and the square error. Yes.

  The identification ECU 50 determines the convergence characteristics (whether learning has converged globally) of each learning curve generated for each temporary class label, and compares and evaluates all learning curves. Then, the identification ECU 50 extracts a learning curve having the highest convergence characteristic from all the learning curves, and identifies the temporary class label of the learning curve as a valid class label for the evaluation data. In the example shown in FIG. 2, the learning curve of the temporary label 2 has the highest convergence characteristic, and the “normal” class label of the label 2 is identified for the evaluation data. The learning curve of tentative label 2 determines that the learning has converged globally, the evaluation data has been collected in a range that is not inconsistent with the variation in the labeling learning data, and has been correctly labeled as target state data. it can. The other learning curves are determined that the learning is not globally converged, and the evaluation data is outside the range of correctly labeled learning data, or is collected in a partially different state it can.

  Judgment conditions for the learning curve convergence characteristics are determined from three viewpoints: whether the first converges or not (square error at the end of learning); if the second converges, it converges quickly (Convergence speed) Whether the third converges smoothly. As a specific method for determining these three determination conditions by the identification ECU 50, the first is converged by learning a fixed number of times (can be set by learning in advance with correct labeling data) or the second is constant. If the learning converges at a certain number of times, how many times the learning has converged, and if the third converges at a certain number of times, the learning curve gradient is not oscillating greatly (the maximum value of the learning curve gradient is It is possible to use it as an index of vibration by taking the average of about three of the peaks of the gradient or taking the average of the larger one). By creating an evaluation function (using the number of learnings until convergence and the vibration index as inputs) by quantifying and integrating the above three determination methods, the convergence characteristic of the learning curve can be calculated quantitatively.

  When the identification ECU 50 identifies the class label for the evaluation data, the identification ECU 50 outputs the identification result (that is, the driver state) to the driving support ECU 60. The evaluation data and the identified class label (teacher data) are used in the relearning process.

  The relearning process will be described. In the identification ECU 50, after the evaluation data is identified in the identification learning process, the evaluation data is labeled based on the identification result, and learning data including the evaluation data and teacher data is generated. Then, in the identification ECU 50, learning is performed using the learning data as in the preliminary learning, and the weight of each neuron of the neural network classifier is updated. In this way, the identification ECU 50 performs re-learning with new learning data and tunes the neural network classifier.

  The driving support ECU 60 includes a CPU, a ROM, a RAM, and the like. When the predetermined driver state identified by the identification ECU 50 is input, the driving assistance ECU 60 determines driving safety according to the driver state. When the driving safety is lower than the normal level, the driving support ECU 60 generates an image or a message for alerting the driver or presenting advice according to the level, and displays the image on the display 40. Or the voice of the message is output from the speaker 41. Further, when the driving safety is lower than the normal level, the driving support ECU 60 transmits an instruction signal such as changing the control timing to the driving support system 42 according to the level.

  If the neural network discriminator with high discrimination accuracy matched to the driver's personal characteristics can be tuned by discriminating learning processing and relearning processing over a certain period of time, the discriminating learning processing and relearning processing are stopped and the tuning is stopped. The class label (driver state) may be identified by a single evaluation by feedforward calculation of the neural network classifier.

  The operation of the driver state identification device 1 will be described with reference to FIGS. In particular, the preliminary learning process in the identification ECU 50 will be described with reference to the flowchart of FIG. 6, the identification learning process will be described with reference to the flowchart of FIG. 7, and the relearning process will be described with reference to the flowchart of FIG. FIG. 6 is a flowchart showing a flow of preliminary learning processing in the neural network identification ECU of FIG. FIG. 7 is a flowchart showing the flow of identification learning processing in the neural network identification ECU of FIG. FIG. 8 is a flowchart showing the flow of the relearning process in the neural network identification ECU of FIG. In particular, the process shown in the flowchart of FIG. 8 is a process for improving the performance of the neural network classifier, and has a different purpose from the process for status identification in the neural network classifier shown in the flowchart of FIG.

  Before the start of use by the user, a large number of input data in which several pieces of information for identifying the driver state are combined is collected, and a class label is set for each input data. For each input data and each correctly labeled class label, the identification ECU 50 generates learning data including the input label and the teacher data of the class label (S10). Then, the identification ECU 50 inputs the input data to the neural network classifier, performs learning by the error back propagation method, and updates the weight of each neuron of the neural network classifier (S11). Here, before the start of use by the user, a basic neural network classifier is constructed by general learning data.

  The driver information recognition means, environmental information recognition means, and vehicle information recognition means sensors detect and recognize each information necessary for identifying the driver state while the vehicle is being driven by the driver. Information is transmitted to identification ECU50. The identification ECU 50 receives information from each sensor or the like at regular time intervals (S20, S30). The identification ECU 50 combines several pieces of information into evaluation data for each sampling period, and holds evaluation data for each sampling period in a predetermined period (S20, S30). This evaluation data does not have a class label, and it is necessary to identify a valid class label. As described above, the predetermined period and sampling period for collecting the evaluation data are set according to the driver state to be identified.

  The identification ECU 50 generates correctly labeled learning data used in past learning (S21). Further, the identification ECU 50 assigns a temporary class label to the evaluation data, generates learning data composed of the evaluation data and temporary teacher data corresponding to the temporary class label, and adds it as learning data (S22). It is necessary to determine whether or not the temporary class label assigned to the evaluation data is valid as the class label of the evaluation data. The identification ECU 50 performs learning by the error back propagation method using the learning data correctly labeled and the learning data by temporary labeling, and updates the weight of each neuron of the neural network classifier (S23). Until the learning end condition is satisfied, the identification ECU 50 calculates the square error between the temporary teacher data and the output data for each learning, and monitors the square error for each learning in order to monitor the time process of learning. (S24). When the learning end condition is satisfied, the identification ECU 50 ends the learning for the temporary class label, rejects all the weight data updated by the current learning of each neuron of the neural network, and returns to the weight before learning ( S25). Then, the identification ECU 50 determines whether or not all possible temporary labeling has been tried on the evaluation data (S26). If it is determined in S26 that not all temporary labeling has been tried, the identification ECU 50 returns to S21 and performs processing for the next temporary class label. Here, learning is performed for each temporary class label assigned to the evaluation data, and the temporal process of the learning is monitored.

  If it is determined in S26 that all temporary labeling has been tried, the identification ECU 50 generates a learning curve from the number of learnings and its square error for each temporary class label, and quantitatively calculates the convergence characteristics of the learning curve. (S27). Then, the identification ECU 50 compares and evaluates the convergence characteristics of all the learning curves, and extracts the learning curve having the best convergence characteristics (S27). Further, the identification ECU 50 identifies the temporary class label corresponding to the extracted learning curve as an appropriate class label for the evaluation data, and outputs the identification result to the driving support ECU 60 (S28). Here, the temporary class label having the best learning convergence is identified as the class label (predetermined driver state) of the evaluation data from among the plurality of temporary class labels assigned to the evaluation data.

  After the evaluation data is acquired (S20, S30), when the class label identification process for the evaluation data is performed by the processes of S21 to S27 (S31), the identification ECU 50 determines whether the evaluation data is valid as learning data. (Specifically, it is determined whether or not one valid class label is identified for the evaluation data by the process of S31) (S32). If it is determined in S32 that the learning data is not valid, the identification ECU 50 does not perform relearning.

  When it is determined that the learning data is valid in the determination of S32, the identification ECU 50 labels the evaluation data according to the identification result (S33), and generates correctly labeled learning data including the evaluation data and the identified teacher data. (S34). The identification ECU 50 performs learning by the back propagation method using the generated learning data, and updates the weight of each neuron of the neural network classifier (S35). As a result, the neural network classifier is tuned with the learning data reflecting the personal characteristics of the driver.

  When the identification result is input from the identification ECU 50, the driving assistance ECU 60 determines the level of driving safety based on the identified predetermined driver information. When the driving safety is lower than the normal level, the driving support ECU 60 uses the display 40 and the speaker 41 to alert the driver and present advice according to the level, and according to the level, the driving support system 42. An instruction signal such as changing the control timing is transmitted. Thus, appropriate driving assistance for improving safety is performed when the driver's safety level is reduced, such as fatigue, scorching, and sleepiness.

  According to the driver state identification device 1, a plurality of temporary class labels (temporary teacher data) are automatically assigned to the evaluation data, instead of a single evaluation of the evaluation data based on the feedforward calculation of the neural network as in the prior art. Since the evaluation is performed a plurality of times based on the temporal process in the allocation and learning (the convergence characteristic of the learning curve), the identification accuracy of the class label (and thus the driver state) for the evaluation data is high. In particular, since whether or not the learning converges globally in the evaluation is used as a determination index, the discrimination accuracy is higher and the robustness is higher with respect to noise and some data loss. In addition, since general information obtained during driving is used as evaluation data, and a plurality of virtual teacher data is automatically assigned to the evaluation data for identification, the driver is not burdened.

  That is, since a plurality of different temporary class labels are set for the evaluation data, the identification is performed based on much more information than before. Specifically, in the identification learning process, the evaluation for one temporary class label corresponds to one evaluation by the conventional feedforward calculation, and by evaluating and identifying each of the plurality of temporary class labels, The identification is performed based on much more information than before. In addition, by setting all possible temporary class labels for the evaluation data, learning is attempted, and by monitoring the convergence of the learning process for each temporary class label, the temporary class label is assigned to the evaluation data. Since information on whether or not it is appropriate is acquired and identification is performed based on the information, noise or one of the problems that cannot be easily dealt with by evaluating the evaluation data only once using a learned neural network as in the past. It is possible to flexibly cope with evaluation data having some data deficiencies.

  Furthermore, according to the driver status identification device 1, since the learning is performed using the class label (teacher data) identified as the evaluation data as learning data, the neural network classifier is tuned to the personal characteristics of the driver. be able to. As a result, the neural network discriminator dynamically adjusts according to the evaluation data, so that there is more flexibility and higher discrimination accuracy than the conventional discrimination with a fixed weight neural network. Further, since it is not necessary for the driver to input a state in order to perform tuning, no burden is imposed on the driver.

  Further, in the driver state identification device 1, since learning is performed using correctly labeled learning data in addition to temporary learning data including evaluation data and temporary teacher data when performing identification learning, a class label for the evaluation data is used. The identification accuracy is further improved.

  In addition, since the driver status identification device 1 performs appropriate driving support according to the identified driver status, the driver status can be urged to be effectively improved according to the driver status, and driving safety can be improved. Can be improved. At this time, driving support without excess or deficiency that does not bother the driver can be performed, and appropriate driving support from an earlier stage is possible.

  As mentioned above, although embodiment which concerns on this invention was described, this invention is implemented in various forms, without being limited to the said embodiment.

  For example, in the present embodiment, driver information, environment information, and vehicle information are detected as evaluation data, and applied to a driver state identification device that identifies a driver state from the detected information. It is applicable to identification, and as information necessary for identifying the driver state, information other than that exemplified in the embodiment may be used, and the driver state to be identified is not exemplified in the embodiment. The driver status may be identified.

  In the present embodiment, the driver state for the evaluation data is identified and the driving support is performed using the identification result. However, the driver state identification result for the evaluation data may be output, or The identification result may be provided to another system.

  In this embodiment, re-learning (additional learning) is also performed using new learning data based on the identification result for evaluation data. However, re-learning may not be performed after identification for evaluation data. .

  In this embodiment, the learning is performed using correctly labeled learning data other than the learning data based on the evaluation data and the temporary teacher data when performing identification learning. However, the learning data correctly labeled is used. Instead, the evaluation data and the learning data based on the temporary teacher data may be used.

  In this embodiment, the neural network is a three-layer hierarchical type, and the learning method based on the error back propagation method is used. However, the structure of the neural network and the learning method are not particularly limited.

  In the present embodiment, the learning curve is generated using the square error according to Expression (2), but the learning curve is not particularly limited as long as the convergence characteristic of learning can be evaluated.

It is a block diagram of the driver state identification apparatus which concerns on this Embodiment. It is a conceptual diagram of the identification method in the driver state identification device of FIG. It is an example of evaluation data. It is an example of temporary teacher data corresponding to a temporary class label and a temporary class label. It is an example at the time of assigning temporary teacher data (temporary class label) to evaluation data. It is a flowchart which shows the flow of the preliminary learning process in the neural network identification ECU of FIG. It is a flowchart which shows the flow of the identification learning process in the neural network identification ECU of FIG. It is a flowchart which shows the flow of the relearning process in the neural network identification ECU of FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Driver state identification device, 10 ... Face direction / line-of-sight recognition sensor, 11 ... Motion / posture recognition sensor, 12 ... Face / eyeball recognition sensor, 13 ... Foot recognition sensor, 14 ... Heart rate sensor, 20 ... Inter-vehicle distance sensor, 21 ... Lane recognition sensor, 22 ... Traffic signal recognition sensor, 23 ... Sign recognition sensor, 24 ... Pause recognition sensor, 25 ... Pedestrian recognition sensor, 26 ... Communication device for acquiring traffic environment information, 27 ... Car navigation system, 30 ... Vehicle speed sensor, 31 ... Rudder angle sensor, 32 ... Accelerator opening sensor, 33 ... Brake depression sensor, 34 ... Front / rear G sensor, 35 ... Lateral G sensor, 36 ... Vertical G sensor, 37 ... Blinker operation sensor, 40 ... Display , 41 ... Speaker, 42 ... Driving support system, 50 ... Neural network identification ECU, 60 ... Driver state adaptive driving support ECU

Claims (12)

An initial learning step of learning the neural network based on the input data and the teacher data corresponding to the class label of the input data;
A virtual teacher data assignment step for assigning virtual teacher data corresponding to a virtual class label to evaluation data whose class label is unknown;
An evaluation step of performing learning of a neural network based on the evaluation data and virtual teacher data of the evaluation data, and evaluating a convergence characteristic of a learning curve of the learning,
A class label for estimating a virtual class label corresponding to the virtual teacher data of the learning curve having a high convergence characteristic among a plurality of learning curves for a plurality of different virtual teacher data assigned to the evaluation data as a class label of the evaluation data A state estimation method using a neural network, comprising: an estimation step.   The state estimation method using a neural network according to claim 1, further comprising a class label output step of outputting a class label of evaluation data estimated in the class label estimation step.   3. An additional learning step of learning a neural network using virtual teacher data corresponding to a class label of evaluation data estimated in the class label estimation step as teacher data of the evaluation data. The state estimation method using the neural network described in 1.   4. The neural network learning according to claim 1, wherein the evaluation step performs learning of a neural network based on the evaluation data, virtual teacher data of the evaluation data, and the input data and teacher data of the input data. 5. A state estimation method using the neural network according to claim 1. The input data and the evaluation data are values detected by the driving state detection means,
The state estimation method using a neural network according to any one of claims 1 to 4, wherein the class label is a driver state. The neural network is a three-layer hierarchical neural network,
A correspondence relationship between the input data and the teacher data of the input data, a correspondence relationship between the evaluation data and the virtual teacher data of the evaluation data, or a correspondence relationship between the evaluation data and the teacher data estimated with respect to the evaluation data. The state estimation method using a neural network according to any one of claims 1 to 5, wherein learning is performed by an error back propagation method, and a learning curve is generated based on an error by the error back propagation method. . Initial learning means for learning a neural network based on input data and teacher data corresponding to the class label of the input data;
Virtual teacher data assignment means for assigning virtual teacher data corresponding to a virtual class label to evaluation data whose class label is unknown;
Based on the evaluation data and virtual teacher data of the evaluation data, learning of the neural network, and evaluation means for evaluating the convergence characteristics of the learning curve of the learning,
A class label for estimating a virtual class label corresponding to the virtual teacher data of the learning curve having a high convergence characteristic among a plurality of learning curves for a plurality of different virtual teacher data assigned to the evaluation data as a class label of the evaluation data And a state estimation device using a neural network.   8. The state estimation apparatus using a neural network according to claim 7, further comprising class label output means for outputting a class label of evaluation data estimated by the class label estimation means.   9. An additional learning means for learning a neural network using virtual teacher data corresponding to a class label of evaluation data estimated by the class label estimation means as teacher data of the evaluation data. The state estimation apparatus using the neural network described in 1.   The evaluation means performs learning of a neural network based on the evaluation data, virtual teacher data of the evaluation data, and the input data and teacher data of the input data. A state estimation apparatus using the neural network described in item 1. The input data and the evaluation data are values detected by the driving state detection means,
The state estimation apparatus using a neural network according to any one of claims 7 to 10, wherein the class label is a driver state. The neural network is a three-layer hierarchical neural network,
A correspondence relationship between the input data and the teacher data of the input data, a correspondence relationship between the evaluation data and the virtual teacher data of the evaluation data, or a correspondence relationship between the evaluation data and the teacher data estimated with respect to the evaluation data. The state estimation device using a neural network according to any one of claims 7 to 11, wherein learning is performed by an error back propagation method, and a learning curve is generated based on an error by the error back propagation method. .

Images