JP7025123B2 - Emotion estimation device - Google Patents

Description

The present invention relates to a sentiment estimation device.

Conventionally, a playback device that estimates the occupant's mental image based on various information related to vehicle driving, the occupant's biological information, input items by the occupant, etc., and selects a reproduction candidate according to the occupant's mental image from a plurality of content data. It is known (see, for example, Patent Document 1). In this playback device, various information related to vehicle driving is, for example, date and time information, weather information, position information, occupant composition, destination, boarding purpose, driving status, traffic status, content data playback frequency, and content data. Preference, etc.

Japanese Unexamined Patent Publication No. 2007-108134

By the way, according to the reproduction device according to the above-mentioned prior art, the occupant of the vehicle only estimates the mental image of the occupant based on the information on the driving of the vehicle and the biological information detected by various sensors mounted on the vehicle. It is not possible to make an estimation according to the perceived information outside the vehicle and the information of the driving scene. This raises the problem that it is difficult to improve the estimation accuracy of the occupant's mental image.

The present invention has been made in view of the above circumstances, and it is possible to improve the estimation accuracy of the emotion by reflecting the information outside the vehicle and the information of the driving scene perceived by the driver of the vehicle in the emotion estimation. It is intended to provide an estimation device.

In order to solve the above problems and achieve the above object, the present invention has adopted the following aspects.
(1) The emotion estimation device according to one aspect of the present invention includes a data storage unit (for example, a model data storage unit 26 in the embodiment) for storing data relating the feature quantity vector of an image and emotion information, and an image. An image acquisition unit (for example, the vehicle outside camera 12 in the embodiment) for acquiring data, and an image feature amount calculation unit (for example, an image feature in the embodiment) for calculating a feature amount vector from the image acquired by the image acquisition unit. Using the amount calculation unit 22) and the data stored in the data storage unit, an emotion estimation unit (for example, an emotion estimation unit) that estimates emotional information related to the feature amount vector calculated by the image feature amount calculation unit. , The emotional estimation unit27) in the embodiment.

(2) In the emotion estimation device according to (1) above, the image acquisition unit may be mounted on a moving body and acquire an image in front of the moving body in the traveling direction.

(3) In the emotion estimation device according to (1) or (2) above, an input device (for example, a steering switch 11 in the embodiment) for detecting an input operation related to the operator's emotion information and the input device are used. The emotional data generation unit (for example, the emotional data generation unit 21 in the embodiment) that generates the emotional data of the operator according to the detected input operation, and the emotional data generated by the emotional data generation unit. And an estimation model creating unit (for example, estimation in the embodiment) that creates an estimation model that associates the image feature amount calculation unit with the feature amount vector and stores the data related to the estimation model in the data storage unit. The model creation unit 25) may be provided.

(4) The psychological estimation device according to (3) above includes a biological information acquisition unit (for example, a biological sensor 15 and a biological index calculation unit 24 in the embodiment) for acquiring biological information of the operator, and the estimation is performed. In the estimation model, the model creation unit may associate the biometric information acquired by the biometric information acquisition unit with each of the emotional data and the feature amount vector.

According to the emotional estimation device according to the aspect (1) above, the feature amount vector of the image and the emotional condition of the person who visually recognizes the actual scene corresponding to the image are associated with each other, and therefore biometric information is used. Without it, the estimation accuracy of emotions can be improved.

Further, in the case of (2) above, the accuracy of emotional estimation can be improved by reflecting the external information perceived by the occupants of the moving body in the emotional estimation.

Further, in the case of the above (3), since the emotional data which is the objective variable is generated according to the input operation by the operator himself, the estimation model can be created accurately.

Further, in the case of the above (4), the estimation accuracy can be improved by using the biological information having a high direct relationship with the emotional information.

It is a block diagram of the functional structure of the sentiment estimation device which concerns on embodiment of this invention. It is a perspective view which shows the steering switch of the sentiment estimation device which concerns on embodiment of this invention in an enlarged manner. It is a figure which shows an example of the map data which shows the emotional information of the emotional state estimation apparatus which concerns on embodiment of this invention. It is a flowchart which shows the operation of the emotion estimation apparatus which concerns on embodiment of this invention. It is a figure which shows the example of the measured value of the emotional value of the emotional state estimator which concerns on embodiment of this invention, and the time-dependent change of the estimated value. It is a block diagram of the functional structure of the sentiment estimation device which concerns on the modification of embodiment of this invention. It is a figure which shows typically the relationship between the explanatory variable and the objective variable of the sentiment estimation apparatus which concerns on the modification of embodiment of this invention.

Hereinafter, the emotion estimation device according to the embodiment of the present invention will be described with reference to the attached drawings.

The emotion estimation device 10 according to the present embodiment is mounted on a moving body such as a vehicle 1 and estimates the emotions of the occupants of the moving body. Crew sentiment includes, for example, emotions such as psychology, feelings, mood, comfort, and discomfort, as well as physical conditions such as sleepy and not sleepy (awakening).
As shown in FIG. 1, the vehicle 1 includes, for example, a steering switch 11, an external camera 12, a current position sensor 13, a vehicle sensor 14, a biological sensor 15, an in-vehicle device 16, and a processing unit 17. I have.
The psychological estimation device 10 according to the embodiment includes, for example, a steering switch 11, an external camera 12, a current position sensor 13, a vehicle sensor 14, a biological sensor 15, and an in-vehicle device 16.

The steering switch 11 is arranged on the steering wheel 18 as shown in FIG. The steering switch 11 is an input device arranged at a position that can be operated by a driver's finger holding the steering wheel 18 for driving the vehicle 1. The steering switch 11 detects the contact and operation of the operator. The steering switch 11 includes, for example, a cross key 19 as a mechanical switch that accepts operations by the operator's fingers. The cross key 19 includes a first key 19a, a second key 19b, a third key 19c, and a fourth key 19d. The cross key 19 receives a tilting operation by the operator's finger and outputs an operation signal from each of the keys 19a, ..., 19d in response to the tilting operation.

In the steering switch 11, the first key 19a and the second key 19b of the cross key 19 receive, for example, an input operation relating to information on the degree of arousal among the emotions of the operator. The first key 19a outputs an operation signal indicating that the arousal degree is high, and the second key 19b outputs an operation signal indicating that the arousal degree is low. The third key 19c and the fourth key 19d of the cross key 19 receive, for example, an input operation relating to pleasant and unpleasant information in the operator's emotions. The third key 19c outputs an operation signal indicating that it is unpleasant, and the fourth key 19d outputs an operation signal indicating that it is comfortable.

The out-of-vehicle camera 12 is a digital camera including a solid-state image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). For example, the vehicle exterior camera 12 captures an image of the front region of the vehicle 1 in the traveling direction and outputs the data of the captured image.

The current position sensor 13 receives a positioning signal transmitted from an artificial satellite or the like in a navigation satellite system (for example, GNSS: Global Navigation Satellite System or the like) including a positioning system such as GPS (Global Positioning System), and the positioning signal. Is used to detect the current position. The current position sensor 13 outputs a signal of the detection value of the current position.

The vehicle sensor 14 is various sensors mounted on the vehicle 1 and outputs signals of detection values of various vehicle state quantities. The vehicle sensor 14 is, for example, a vehicle speed sensor, an engine rotation sensor, an acceleration sensor, a yaw rate sensor, an accelerator sensor, a throttle opening degree sensor, a brake pressure sensor, a steering sensor, or the like. The vehicle speed sensor detects the speed (vehicle speed) of the vehicle 1. The engine rotation sensor detects the engine rotation speed. The acceleration sensor detects the acceleration of the vehicle 1, for example, front-back acceleration and lateral acceleration. The yaw rate sensor detects the angular velocity around the vertical axis of the vehicle 1. The accelerator sensor detects the displacement amount (accelerator opening) of the accelerator pedal. The throttle opening sensor detects the opening of the throttle valve. The brake pressure sensor detects the oil pressure (brake pressure) of the master cylinder, or the brake caliper or wheel cylinder. The steering sensor detects the steering angle or steering torque of the steering wheel.

The biosensor 15 is each sensor that detects, for example, electroencephalogram (EEG), electrocardiogram (ECG), skin conductance level (SCL), skin conductance response (SCR), skin electrical reflex (GSR), and the like. The biological sensor 15 outputs the detected values of various biological data.

The in-vehicle device 16 is various devices mounted on the vehicle 1. The in-vehicle device 16 is, for example, a driving force output device, a brake device, a steering device, a video and audio device, a navigation device, a display device, a control device for controlling various devices, and the like.

The driving force output device outputs a traveling driving force (torque) for the vehicle 1 to travel to the driving wheels. The driving force output device includes, for example, a combination of an internal combustion engine, a motor, a transmission, and the like, and an ECU (Electronic Control Unit) that controls them. The ECU controls the traveling driving force according to the information input from the control device of the vehicle 1 or the information input from the driving operator.

The brake device includes, for example, a brake caliper or a wheel cylinder, an electric motor for generating hydraulic pressure in the brake caliper or the wheel cylinder, and a brake ECU. The brake ECU controls the electric motor according to the information input from the control device of the vehicle 1 or the information input from the driving operator, and outputs the brake torque corresponding to the braking operation to each wheel. The brake device includes, for example, a mechanism for transmitting the hydraulic pressure generated by the operation of the brake pedal of the operation operator to the cylinder via the master cylinder in addition to the hydraulic pressure generated by the electric motor. The brake device is not limited to the configuration described above, and the actuator may be controlled according to the information input from the control device to transmit the hydraulic pressure of the master cylinder to the brake caliper or the wheel cylinder.

The steering device includes, for example, a steering ECU and an electric motor. The electric motor, for example, exerts a force on the rack and pinion mechanism to change the direction of the steering wheel. The steering ECU drives the electric motor according to the information input from the control device or the information input from the driving operator, and changes the direction of the steering wheel.

The processing unit 17 is a software function unit that functions by executing a predetermined program by a processor such as a CPU (Central Processing Unit). The software function unit is an ECU including a processor such as a CPU, a ROM (Read Only Memory) for storing a program, a RAM (Random Access Memory) for temporarily storing data, and an electronic circuit such as a timer. At least a part of the processing unit 17 may be an integrated circuit such as an LSI (Large Scale Integration).

The processing unit 17 is, for example, an emotional data generation unit 21, an image feature amount calculation unit 22, a vehicle information feature amount calculation unit 23, a biometric index calculation unit 24, an estimation model creation unit 25, and a model data storage unit 26. The emotion estimation unit 27 and the vehicle characteristic correction unit 28 are provided.

The emotional data generation unit 21 generates the operator's emotional data according to the operation signal output from the cross key 19 of the steering switch 11, and inputs the operator's emotional data as an objective variable to the estimation model creation unit 25. .. The emotional data generation unit 21 uses map data showing emotional information, such as Russell's circular emotional map shown in FIG. 3, to generate the emotions of the operator according to the operation signal output from the cross key 19. get. The map data showing the emotional information is the map data that associates the cumulative operation signals output from the cross key 19 in time series with the emotional information. For example, in the Russell's circular emotion map shown in FIG. 3, the high and low levels of arousal associated with the first key 19a and the second key 19b, and the pleasure associated with the third key 19c and the fourth key 19d. And the unpleasant information are set in two coordinate axes of the two-dimensional Cartesian coordinate system. In the two-dimensional Cartesian coordinate system of Russell's ring emotion map, a plurality of types of emotional regions are set, which differ depending on the combination of high and low arousal levels and pleasant and unpleasant information. The emotional data generation unit 21 displaces the coordinate position in the two-dimensional Cartesian coordinate system of Russell's annular emotion map according to the accumulation of the operation signals output from the cross key 19, and is associated with the region including the coordinate position. Get information about your feelings. The emotional data generation unit 21 uses the emotional information acquired from the map data as the emotional data of the operator. The operation signal output from the cross key 19 is, for example, an operator's intuitive recognition of information on the degree of arousal at the present time point or pleasant and unpleasant information as compared with the past time point at an appropriate time interval. It shows a change.

The image feature amount calculation unit 22 calculates an image feature amount vector from the data of the captured image output from the vehicle outside camera 12, and inputs the image feature amount vector as an explanatory variable to the estimation model creation unit 25. The image feature amount calculation unit 22 calculates an image feature amount vector by machine learning such as a convolutional neural network (CNN). The image feature amount calculation unit 22 may shorten the data and time required for training by using, for example, an existing trained CNN model.

The vehicle information feature amount calculation unit 23 calculates the vehicle information feature amount using the signals of the detection values of various vehicle state amounts output from the current position sensor 13 and the vehicle sensor 14, and uses the vehicle information feature amount as an explanatory variable. Input to the estimation model creation unit 25. The vehicle information feature amount calculation unit 23, for example, obtains the detection value itself output from the vehicle sensor 14 or the secondary information obtained by a predetermined calculation using the detection value output from the vehicle sensor 14 for the vehicle. It is an information feature amount. The vehicle information feature amount calculation unit 23 calculates the steering angular velocity by using, for example, the detection value of the steering angle of the steering wheel output from the steering sensor of the vehicle sensor 14.

The biometric index calculation unit 24 calculates the biometric index using the signals of the detection values of various biometric data output from the biometric sensor 15, and inputs the biometric index as an explanatory variable to the estimation model creation unit 25. The biometric index calculation unit 24 uses, for example, the detection value itself output from the biometric sensor 15 or the secondary information obtained by a predetermined operation using the detection value output from the biometric sensor 15 as the biometric index. do. The biometric index calculation unit 24 uses, for example, the detection value of the electroencephalogram (EEG) output from the biosensor 15 to determine the amplitude of a specific frequency band (for example, α wave band, β wave band, θ wave band, etc.). calculate. The biometric index calculation unit 24 calculates the R wave interval (RRI) using, for example, the detection value of the electrocardiogram (ECG) output from the biosensor 15.

The estimation model creation unit 25 creates an emotion estimation model when the execution of the estimation model generation mode is instructed. The estimation model creation unit 25 uses the operator's emotional data output from the emotional data generation unit 21 as the objective variable. The estimation model creation unit 25 includes an image feature amount vector output from the image feature amount calculation unit 22, a vehicle information feature amount output from the vehicle information feature amount calculation unit 23, and a biometric index output from the biometric index calculation unit 24. Is an explanatory variable. The estimation model creation unit 25 creates an emotion estimation model so that the operator's emotional data, which is the objective variable, is uniquely determined from at least one of the image feature amount vector, the vehicle information feature amount, and the biometric index, which are explanatory variables. do.
The estimation model creation unit 25 is, for example, a sparse estimation method based on the Lasso (Least absolute shrinkage and selection operator) regularization method in a multiple regression model in which an image feature quantity vector is input and an operator's emotional data is output. Create a sentiment estimation model by using.
The estimation model creation unit 25 creates an emotional estimation model by using a stepwise method in a multiple regression model that inputs vehicle information features and biomarkers and outputs the operator's emotional data, for example. ..
The estimation model creation unit 25 may set a plurality of groups based on the similarity of information such as the operator's physiological reaction and driving characteristics, and create an emotion estimation model for each of the plurality of groups.

The model data storage unit 26 stores model data based on a combination of the emotional estimation model created by the estimation model creation unit 25 or an appropriate input value and the emotional estimation value estimated by the emotional estimation model for this input value. Remember.

When the execution of the estimation model generation mode is not instructed, the emotion estimation unit 27 uses the emotion estimation model or model data stored in the model data storage unit 26 to obtain an image feature amount vector, a vehicle information feature amount, and the like. And estimate emotional information related to at least one of the biomarkers. The emotion estimation unit 27 acquires the emotion estimation value related to at least one of the image feature amount vector, the vehicle information feature amount, and the biometric index by estimation by the emotion estimation model or by searching the model data.
When a plurality of emotion estimation models or model data are stored in the model data storage unit 26 for each group, the emotion estimation unit 27 stores an appropriate emotion estimation model or model data in response to a selection instruction by the operator or the like. You may choose.

The vehicle characteristic correction unit 28 corrects the vehicle characteristics according to the emotional estimation value acquired by the emotional estimation unit 27. The vehicle characteristics are, for example, information that characterizes the operating state of the vehicle-mounted device 16.

The emotional estimation device 10 according to the present embodiment has the above-mentioned configuration, and next, the operation of the emotional estimation device 10 will be described with reference to FIG.

First, in step S01, the image feature amount calculation unit 22 acquires the data of the captured image captured by the external camera 12.
Next, in step S02, the image feature amount calculation unit 22 calculates the image feature amount vector from the data of the captured image.
Next, in step S03, the vehicle information feature amount calculation unit 23 acquires the detection value of the current position detected by the current position sensor 13.
Next, in step S04, the vehicle information feature amount calculation unit 23 acquires the detection values of various vehicle state quantities detected by the vehicle sensor 14.
Next, in step S05, the vehicle information feature amount calculation unit 23 calculates the vehicle information feature amount based on the detection values of the current position and various vehicle state quantities.

Next, in step S06, the estimation model creation unit 25 determines whether or not the execution of the estimation model generation mode is instructed.
If the determination result is "NO", the estimation model creation unit 25 advances the process to step S13.
On the other hand, if the determination result is "YES", the estimation model creation unit 25 advances the process to step S07.
Next, in step S07, the biological index calculation unit 24 acquires the detected values of various biological data detected by the biological sensor 15.
Next, in step S08, the biometric index calculation unit 24 calculates the biometric index based on the detected values of various biometric data.
Next, in step S09, the emotional data generation unit 21 acquires an operation signal output from the cross key 19 of the steering switch 11.
Next, in step S10, the emotional data generation unit 21 generates the emotional data of the operator in response to the operation signal.

Next, in step S11, the estimation model creation unit 25 uniquely determines the emotional data of the operator, which is the objective variable, by at least one of the image feature amount vector, the vehicle information feature amount, and the biometric index, which are explanatory variables. So, create a sentiment estimation model.
Next, in step S12, the model data storage unit 26 uses the emotion estimation model created by the estimation model creation unit 25, or an appropriate input value, and the emotion estimation value estimated by the emotion estimation model for this input value. Stores model data in combination with. Then, the process proceeds to the end.

Further, in step S13, the biological index calculation unit 24 determines whether or not the detection values of various biological data detected by the biological sensor 15 have been acquired.
If the determination result is "NO", the biometric index calculation unit 24 advances the process to step S15.
On the other hand, when this determination result is "YES", the biometric index calculation unit 24 proceeds to the process in step S14.
Next, in step S14, the biometric index calculation unit 24 calculates the biometric index from the detected values of various biometric data.
Next, in step S15, the emotional estimation unit 27 acquires the emotional estimation value related to at least one of the image feature amount vector, the vehicle information feature amount, and the biometric index by estimation by the emotional estimation model or by searching the model data. do.
Next, in step S16, the vehicle characteristic correction unit 28 corrects the vehicle characteristics according to the emotional estimation value acquired by the emotional estimation unit 27. Then, the process proceeds to the end.

Hereinafter, an example in which a sentiment estimation model is created using predetermined vehicle information feature quantities and biometric index data obtained during actual driving of the vehicle 1 and the sentiment estimated value (estimated value) is estimated will be described. ..
In this embodiment, the predetermined vehicle information feature amount and bioindex data used as explanatory variables are RRI, GSR, vehicle speed, engine speed, accelerator opening, brake pressure, steering angle, steering angular velocity, lateral acceleration, front and rear. Acceleration, yaw rate, brain wave theta band amplitude, brain wave alpha band amplitude, and brain wave beta band amplitude. In the embodiment, the average value of each data acquired at a predetermined time interval (for example, 1 second) over 1 minute is set as 1 trial (trial) data, and the total number of trials is 126, so that the data is 126 minutes. Data of 126 trials (trials) were acquired. .. In the embodiment, in each of the arbitrary N trials (1 ≤ N ≤ 126) between the first trial and the 126th trial, the emotions are estimated by the emotion estimation model while using the Leave one out method. The estimated value was estimated. FIG. 5 is a diagram in which 126 emotional estimated values and 126 emotional measured values (measured values) corresponding to the input operation of the operator synchronized with the acquisition of each data are drawn according to the operating time. ..
According to FIG. 5, the estimated emotional value and the actually measured emotional value are in good agreement, and it is recognized that the estimated accuracy of the estimated emotional value is high.

As described above, according to the sentiment estimation device 10 according to the present embodiment, the captured image in front of the vehicle 1 in the traveling direction is associated with the sentiment of the occupant who visually recognizes the actual scene corresponding to the captured image. , It is possible to improve the estimation accuracy of emotions without using biometric information. By reflecting the external information of the vehicle 1 and the information of the driving scene perceived by the occupant of the vehicle 1 in the emotion estimation, the estimation accuracy of the emotion can be improved.
Further, since the emotional data, which is the objective variable, is generated according to the input operation of the operator himself / herself using the steering switch 11, the emotional estimation model can be accurately created.
Furthermore, the estimation accuracy of the emotional estimation model can be improved by using various biological data having a high direct relationship with the emotional information.

Hereinafter, a modified example of the above-described embodiment will be described.
In the above-described embodiment, when the creation of the emotional estimation model is completed, the emotional data generation unit 21, the biometric index calculation unit 24, and the estimation model creation unit 25 are omitted as in the modification shown in FIG. May be done.
In this modification, the emotion estimation unit 27 may acquire the emotion estimation value related to the image feature amount vector or the vehicle information feature amount by using the emotion estimation model or model data stored in the model data storage unit 26. ..
According to this modification, as the creation of the emotion estimation model is completed, even if the biometric index is omitted as shown in FIG. 7, the emotion estimation related to the image feature vector or the vehicle information feature is estimated. The value can be estimated accurately.

In the above-described embodiment or modification, the emotion estimation model or model data created by the estimation model creation unit 25 may be stored in a server device provided outside the vehicle 1. In this case, the emotion estimation device 10 may include a communication device that communicates with the server device, and may acquire the emotion estimation value by using the emotion estimation model or model data acquired from the server device.

In the above-described embodiment, the emotion estimation device 10 may include another input device instead of the steering switch 11. Other input devices may be, for example, switches, touch panels, keyboards, voice input devices, and the like.

The embodiments of the present invention are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and variations thereof are included in the scope of the invention described in the claims and the equivalent scope thereof, as are included in the scope and gist of the invention.

1 ... Vehicle (moving body), 10 ... Psychological estimation device, 11 ... Steering switch (input device), 12 ... External camera (image acquisition unit), 13 ... Current position sensor, 14 ... Vehicle sensor, 15 ... Biological sensor (living body) Information acquisition unit), 16 ... In-vehicle device, 17 ... Processing unit, 19 ... Cross key, 19a ... 1st key, 19b ... 2nd key, 19c ... 3rd key, 19d ... 4th key, 21 ... Emotional data generation unit , 22 ... Image feature amount calculation unit, 23 ... Vehicle information feature amount calculation unit, 24 ... Biometric index calculation unit (biological information acquisition unit), 25 ... Estimated model creation unit, 26 ... Model data storage unit (data storage unit), 27 ... Emotion estimation department, 28 ... Vehicle characteristic correction department

Claims (4)

An estimation model that outputs emotional information when the feature quantity vector of the image output by the machine learning model is input, and a data storage unit that stores the information of the estimation model generated by multiple regression analysis.
An image acquisition unit that acquires an image of the surroundings of a moving object ,
An image feature amount calculation unit that calculates the feature amount vector by inputting the image acquired by the image acquisition unit into the machine learning model, and an image feature amount calculation unit.
An emotion estimation unit that estimates emotional information related to the image by inputting the feature amount vector calculated by the image feature amount calculation unit into the estimation model stored in the data storage unit.
Equipped with
The data storage unit estimates the emotions of the occupants acquired based on the accumulation of the input operations detected in time series by the input device that detects the input operations related to the emotional information of the occupants of the moving body. An emotional estimation device characterized in that it is associated with a model and stored . The image acquisition unit is mounted on the moving body and acquires an image in front of the moving body in the traveling direction.
The emotional estimation device according to claim 1. An input device that detects input operations related to the operator's emotional information, and
An emotional data generation unit that generates emotional information of the operator in response to the input operation detected by the input device.
The estimation model is obtained by performing multiple regression analysis on a model that outputs the emotional information and inputs the feature amount vector calculated by the image feature amount calculation unit when the input operation is performed. An estimation model creation unit that is created and stores data related to the estimation model in the data storage unit,
The emotion estimation device according to claim 1 or 2. A biometric information acquisition unit for acquiring the biometric information of the operator is provided.
The model further inputs the biometric information.
The emotional estimation device according to claim 3.

Images