KR20240036241A - Method and System for Human-Machine Interaction in Driving Assistant System for Semi-Autonomous Driving Vehicles - Google Patents

Abstract

A human-machine interaction method and system for a driving assistance system in a semi-autonomous vehicle is presented. The human-machine interaction system for the driving assistance system of a semi-autonomous car proposed in the present invention uses a CNN model based on the driver's biosignals to classify the driver's emotions and situations and transmits the situation recognition result data to the ECU board. Semi-autonomous driving is possible through HMI (Human machine interface), ECU (Electronic Control Unit) board that generates multiple scenarios related to the semi-autonomous driving virtual environment using received HMI result data, and CARLA (Car Learning to Act) simulator. ROS (Robot operating system), which manages vehicle information and sensor data related to the semi-autonomous driving virtual environment for execution, and the CARLA client according to a plurality of scenarios related to the semi-autonomous driving virtual environment using the vehicle information and sensor data. Includes CARLA server that runs semi-autonomous driving.

Description

{Method and System for Human-Machine Interaction in Driving Assistant System for Semi-Autonomous Driving Vehicles}

The present invention relates to a human-machine interaction method and system for a driver assistance system in a semi-autonomous vehicle.

Recently, research is being conducted to apply human-machine interaction technology to autonomous vehicles. ADAS (Advanced Driver Assistance Systems) is a system that helps drivers drive in various ways, but there is little research on the interaction between the driver's situation inside the vehicle and the vehicle control system using emotions. To date, by analyzing the driving situation and emotions for the design of autonomous driving and driver assistance systems, as well as research on how changes in the driver's emotions affect vehicle driving and control abilities, it is possible to design a vehicle control system with new aspects, and to design a vehicle control system with new aspects. It can be seen that interaction between vehicles is necessary.

When conducting autonomous driving research with actual vehicles, there are problems such as the risk of fatal accidents and the need for experimental equipment and costs. To solve these problems, simulators in virtual environments are often used. The 3D virtual simulator is not affected by limitations such as objects, weather, space, and experiment costs, and it is easy to set up the situations necessary for research by collecting various scenario data. In particular, virtual autonomous driving simulators based on 3D game engines such as Unreal Engine and Unity are being used in various research fields. Driving simulator types include virtual engine-based CARLA, Sim4CV, and AirSim, and single engine (Unity Engine)-based LGSVL (LG Silicon Valley Lab) were developed. CARLA (Car Learning to Act) simulator, which is free from physical time and location constraints when driving a vehicle, is mainly used. ROS (Robot Operating System) is an open source-based meta operating system that is middleware and is advantageous for heterogeneous devices. It is optimized for application to the real environment in an environment developed using a simulator, and autonomous driving research using ROS is actively underway. As mentioned earlier, when developing autonomous driving modules, modules developed based on a simulator have the advantage of being able to be immediately applied to actual vehicles. When developing ECU (Electronic Control Unit) modules mounted on vehicles, much research is being conducted on simulators based on HILS (Hardware-in-the-Loop). When testing is conducted using HILS, various problems such as safety risks and space constraints do not occur. ECU is a vehicle electronic control microcontroller mounted on a vehicle and is a board that controls the engine and peripheral devices. ADAS (Advanced Driver Assistance Systems) is a development of autonomous vehicle technology, with more than 70 ECU boards installed in vehicles. When developing an ECU module using simulation, the ECU replaces the vehicle-specific object model. Therefore, there is an advantage that the control module on the simulator can be easily developed and reused even without a completed vehicle during the experiment. Electrical devices within a vehicle can be controlled and monitored using the CAN (Controller Area Network) communication protocol.

[1] Sini, Jacopo, Antonio Costantino Marceddu, and Massimo Violante. “Automatic emotion recognition for the calibration of autonomous driving functions.” Electronics 9.3 (2020): 518. [2] Alian, A.A.; Kirk, H.S. Photoplethysmography. Best Practice. Res. Clin. Anaesthesiol. 2014, 28, 395-406. [3] Jacobs, K.W.; Frank, E.H., Jr. Effects of four psychological primary colors on GSR, heart rate and respiration rate. Percept. Mot. Ski.1974, 38, 763-766. [4] Maeng, Jun-Ho, Dong-Hyun Kang, and Deok-Hwan Kim. “Deep Learning Method for Selecting Effective Models and Feature Groups in Emotion Recognition Using an Asian Multimodal Database.” Electronics 9.12 (2020): 1988.

The technical problem to be achieved by the present invention is human-machine interaction for a driving assistance system for a semi-autonomous vehicle equipped with situational awareness based on bio-signal data using a virtual environment CARLA simulator to control driver-assisted driving using the situation and emotions of the occupants. The purpose is to provide an operation method and system.

In one aspect, the human-machine interaction system for a driving assistance system for a semi-autonomous car proposed in the present invention classifies the driver's emotions and situations using a CNN model based on the driver's biosignals, and provides situational recognition result data. HMI (Human machine interface) transmitted to the ECU board, ECU (Electronic Control Unit) board that generates multiple scenarios related to the semi-autonomous driving virtual environment using the received HMI result data, and CARLA (Car Learning to Act) simulator. ROS (Robot operating system) that manages vehicle information and sensor data related to the semi-autonomous driving virtual environment to execute semi-autonomous driving through a plurality of scenarios related to the semi-autonomous driving virtual environment using the vehicle information and sensor data It includes a CARLA server that executes semi-autonomous driving of CARLA clients.

The HMI uses PPG signals and GSR signals to use a CNN model based on the driver's biosignals, and removes low-frequency components of the raw data to directly use the raw data as training data and data for real-time emotion recognition. Butterworth Filter is applied to reduce baseline fluctuations and dynamic noise by using high-order polynomial and moving average filters, and the preprocessed biosignals are divided into waveform units and data is learned through a 1D CNN model. Proceed.

The ROS links data from the CARLA simulator and the ECU board through the ROS Bridge, and the received vehicle information and sensor data are processed for semi-autonomous driving control at each ROS node, enabling semi-autonomous driving. CARLA server manages vehicle information and sensor data to execute semi-autonomous driving using ROS message data transmitted through ROS's API (Application Programming Interface).

The CARLA server converts the vehicle information and sensor data into CAN data to transmit the data to the virtual vehicle, and uses the HMI result data and the vehicle information and sensor data to generate data generated by the ECU board in semi-autonomous driving mode. CARLA client's virtual vehicle is controlled to execute driver control or autonomous control according to multiple scenarios depending on the driver's emotions and situation.

In another aspect, the human-machine interaction method for the driving assistance system of a semi-autonomous car proposed in the present invention is a human machine interface (HMI) that uses a CNN model based on the driver's biosignals to determine the driver's emotions and situation. Classifying and transmitting the situation awareness result data to the ECU board, generating multiple scenarios for the semi-autonomous driving virtual environment using the HMI result data received by the ECU (Electronic Control Unit) board, and ROS (Robot A step in which the operating system) manages vehicle information and sensor data related to the semi-autonomous driving virtual environment to execute semi-autonomous driving through the CARLA (Car Learning to Act) simulator, and the CARLA server uses the vehicle information and sensor data to It includes executing semi-autonomous driving of the CARLA client according to a plurality of scenarios related to the semi-autonomous driving virtual environment.

According to embodiments of the present invention, human-machine interaction for a driving assistance system for a semi-autonomous vehicle equipped with situational awareness based on bio-signal data using a virtual environment CARLA simulator to control driver-assisted driving using the passenger's situation and emotions Methods and systems of operation are provided.

1 is a diagram showing the configuration of a human-machine interaction system for a driving assistance system for a semi-autonomous vehicle according to an embodiment of the present invention.
Figure 2 is a flowchart illustrating a human-machine interaction method for a driving assistance system for a semi-autonomous vehicle according to an embodiment of the present invention.
Figure 3 is a diagram showing a ROS vehicle control Rqt graph for semi-autonomous control according to an embodiment of the present invention.
Figure 4 is a flowchart for explaining a semi-autonomous driving control process according to an embodiment of the present invention.
Figure 5 is a diagram for explaining a DAS communication frame according to an embodiment of the present invention.
Figure 6 is a configuration diagram for DAS according to an embodiment of the present invention.
Figure 7 is a diagram dividing a two-dimensional area into four areas to control a vehicle based on the driver's situation according to an embodiment of the present invention.
Figure 8 is a diagram showing the proposed 1D CNN model for driver's situation and emotion recognition according to an embodiment of the present invention.

Current vehicle-centric semi-autonomous modules do not take into account the driver's situation and emotions. When changing from autonomous driving to manual driving, HMI (Human-Machine Interface) and ADAS (Advanced Driver Assistance Systems) are essential to assist vehicle driving.

In the present invention, vehicle speed control and information are monitored through driver's situation and emotion recognition, and a total of four situations are classified to confirm the possibility of vehicle driving scenarios. As a result of the experiment, a multi-modal biosignal-based 1D CNN (1D Convolutional Neural Networks) model was applied to a semi-autonomous vehicle. The study was conducted in a virtual environment similar to a real vehicle using a driving simulator and HILS (Hardware-in-the-Loop Simulation) equipment. The controllability of the new driver assistance system was confirmed in each situation, emotion, and situation of the driver, and the response speed of the driver assistance system was 351.75 ms through 4 bio-signal data situations and 8 emotion recognition, confirming that it was applicable.

In the present invention, a human-machine interaction method and system for a driving assistance system of a semi-autonomous vehicle equipped with situational awareness based on biosignal data using a virtual environment CARLA simulator to control driver assistance driving using the occupant's situation and emotions is provided. For situation and emotion recognition, simple data processing and 1D CNN (1D Convolutional Neural Networks) emotion recognition are performed. Objects such as vehicles and cities were constructed based on a driving simulator, which is a game engine-based open source platform, with a configuration similar to a real environment vehicle, and data of the vehicle was monitored and controlled using ROS.

Recently, driver-machine interaction research has applied various biosignals. According to the prior art, biometric signals based on facial expressions were used to smoothly transition from manual driving to autonomous driving. By conveying the passengers' intentions and emotions to the system, comfort during driving was provided through decisions that were closest to the passengers' intentions. Another prior art proposed a warning system to prevent accidents using the driver's voice-based biometric signals. As a result of comparing various driver behavior states (DBS) by applying the proposed system, it was confirmed that improvement of the existing vehicle control system is possible. However, there are many difficulties in switching vehicle control during semi-autonomous driving.

Another prior art measured the vehicle control transition time according to the driver's emotions in semi-autonomous driving, and conducted research on manual driving and the driver's emotions and situations. A study was also conducted on the effects of high and low driver emotional states on manual driving performance. When the driver's emotions were positive, concentration during driving was high, but reaction speed was slow. Negative emotional states showed low concentration and high reaction speed. And when drivers listened to happy music and sad music, the driving speed showed greater changes in shifting and steering control for the happy music compared to the sad music. According to survey responses from the AAA Foundation for Traffic Safety, 80% of drivers reported anger and aggression while driving. Analysis of driver driving data showed that emotional states, including anger, sadness, crying and emotional instability, increase the likelihood of a vehicle crash by 9.8 times, and when emotions are in a state of anger, speeding, violating traffic rules and driving dangerously aggressively. There have been confirmed cases of accidents occurring due to drivers' inability to control their emotions while driving a vehicle, but this has not been applied due to difficulties in accuracy of emotion recognition and access to the vehicle control system. Recently, self-driving car technology has been actively researched with methods for extracting high accuracy for emotional recognition based on various human biosignals and detailed vehicle control systems. Research on how to prevent and reduce accidents using the emotions of passengers is being used as a new method to achieve complete control of intelligent vehicles in which driver's behavior and emotion recognition is in the human-machine interaction of semi-autonomous vehicles. The studies mentioned above often conduct virtual environment simulator experiments when conducting research on driver-based vehicle driving. It is easy to obtain reliable research data in a driving simulator, and biosignals that are greatly influenced by the surrounding environment take advantage of this advantage to study the driver's condition depending on the situation. In autonomous driving research linking middleware ROS and game engines, experiments without actual vehicles have been actively conducted. Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

1 is a diagram showing the configuration of a human-machine interaction system for a driving assistance system for a semi-autonomous vehicle according to an embodiment of the present invention.

The present invention proposes a driver assistance module 110 that includes a human machine interface (HMI) 111 that operates according to the driver's situation and emotions during semi-autonomous driving.

The proposed human-machine interaction system for the driving assistance system of a semi-autonomous car includes a human machine interface (HMI) 111 of the driver assistance module 110, an electronic control unit (ECU) board 112, It includes ROS (Robot operating system) 121, CARLA server 122, and CARLA client 123 of the driving simulator module 120.

The HMI (111) classifies the driver's emotions and situations through a 1D CNN (1D Convolutional Neural Networks) model based on the driver's biometric signals, and sends the situation recognition result data to the ECU (Electronic Control Unit) board (112) using serial communication. will be transmitted to.

The HMI 111 may use the PPG signal and the GSR signal to use a 1D CNN model based on the driver's biometric signals. In order to directly use raw data as training data and data for real-time emotion recognition, Butterworth Filter is applied to remove low-frequency components of the raw data, and high-order polynomial and moving average filters are used to determine baseline fluctuation and Dynamic noise can be reduced. In this way, the preprocessed biosignals are divided into waveform units, and then data learning is performed through a 1D CNN model.

The ECU board 112 uses the received HMI result data to configure four scenarios regarding the semi-autonomous driving virtual environment in the CARLA (Car Learning to Act) simulator, which is the driving simulator module 120.

ROS (121) manages vehicle information and sensor data about the semi-autonomous driving virtual environment to execute semi-autonomous driving through CARLA (Car Learning to Act) simulator.

The ROS 121 inputs the throttle and brake values of the driving scenario vehicle configured using the received data to the CARLA server 122 to control the speed of the virtual vehicle of the CARLA client 123. At this time, the throttle and brake data sent to the virtual vehicle are converted into ROS TOPIC messages and used in the vehicle. If there is no HMI result data, the ECU board 112 does not transmit data. Therefore, the vehicle switches to manual driving, and the vehicle speed is controlled using the throttle and brake values.

ROS 121 links data from the CARLA simulator and the ECU board 112 through the ROS Bridge, and the received vehicle information and sensor data are processed for semi-autonomous driving control at each ROS node. , CARLA server 122 manages vehicle information and sensor data to execute semi-autonomous driving using ROS message data transmitted through ROS' API (Application Programming Interface) for semi-autonomous driving.

The CARLA server 122 uses the vehicle information and sensor data to execute semi-autonomous driving of the CARLA client 123 according to a plurality of scenarios related to the semi-autonomous driving virtual environment.

The CARLA server 122 converts the vehicle information and sensor data into CAN data to transmit it to the virtual vehicle. Thereafter, the HMI result data, vehicle information, and sensor data are used to execute driver control or autonomous control according to a plurality of scenarios according to the driver's emotions and situations generated by the ECU board 112 in semi-autonomous driving mode. The virtual vehicle of the CARLA client 123 can be controlled.

Figure 2 is a flowchart illustrating a human-machine interaction method for a driving assistance system for a semi-autonomous vehicle according to an embodiment of the present invention.

The proposed human-machine interaction method for the driving assistance system of a semi-autonomous car is that HMI (Human Machine Interface) classifies the driver's emotions and situations using a CNN model based on the driver's biosignals, and sends the situation recognition result data to the ECU. A step of transmitting to the board (210), a step of generating a plurality of scenarios regarding the semi-autonomous driving virtual environment using the HMI result data received by the ECU (Electronic Control Unit) board (220), and the Robot operating system (ROS) A step (230) of managing vehicle information and sensor data regarding a semi-autonomous driving virtual environment for executing semi-autonomous driving through a CARLA (Car Learning to Act) simulator, and the CARLA server uses the vehicle information and sensor data to It includes a step (240) of executing semi-autonomous driving of the CARLA client according to a plurality of scenarios regarding the semi-autonomous driving virtual environment.

In step 210, the HMI classifies the driver's emotions and situations through a 1D Convolutional Neural Networks (1D CNN) model based on the driver's biosignals, and sends the situation recognition result data to the ECU (Electronic Control Unit) board using serial communication. will be transmitted to.

The HMI can use the PPG signal and GSR signal to use a 1D CNN model based on the driver's biometric signals. In order to directly use raw data as training data and data for real-time emotion recognition, Butterworth Filter is applied to remove low-frequency components of the raw data, and high-order polynomial and moving average filters are used to determine baseline fluctuation and Dynamic noise can be reduced. In this way, the preprocessed biosignals are divided into waveform units, and then data learning is performed through a 1D CNN model.

In step 220, the ECU board uses the received HMI result data to configure four scenarios regarding the semi-autonomous driving virtual environment in CARLA (Car Learning to Act) simulator, a driving simulator module.

In step 230, ROS manages vehicle information and sensor data regarding the semi-autonomous driving virtual environment for executing semi-autonomous driving through the CARLA simulator.

ROS inputs the throttle and brake values of the driving scenario vehicle configured using the transmitted data to the CARLA server to control the speed of the CARLA client's virtual vehicle. At this time, the throttle and brake data sent to the virtual vehicle are converted into ROS TOPIC messages and used in the vehicle. If there is no HMI result data, the ECU board does not transmit data. Therefore, the vehicle switches to manual driving, and the vehicle speed is controlled using the throttle and brake values.

ROS links data from the CARLA simulator and the ECU board through the ROS Bridge, and the received vehicle information and sensor data are processed for semi-autonomous driving control at each ROS node. Using ROS message data sent to ROS' API (Application Programming Interface), CARLA server manages vehicle information and sensor data to enable semi-autonomous driving.

In step 240, the CARLA server executes semi-autonomous driving of the CARLA client according to a plurality of scenarios regarding the semi-autonomous driving virtual environment using the vehicle information and sensor data.

The CARLA server uses the vehicle information and sensor data to execute semi-autonomous driving of the CARLA client according to a plurality of scenarios related to the semi-autonomous driving virtual environment.

The CARLA server converts the vehicle information and sensor data into CAN data to transmit it to the virtual vehicle. Afterwards, the CARLA client uses the HMI result data, vehicle information, and sensor data to execute driver control or autonomous control according to a plurality of scenarios according to the driver's emotions and situations generated by the ECU board in semi-autonomous driving mode. You can control a virtual vehicle.

Figure 3 is a diagram showing a ROS vehicle control RQT graph for semi-autonomous control according to an embodiment of the present invention.

Vehicle information and sensor data according to an embodiment of the present invention are managed using ROS. ROS Bridge is used to link CARLA simulator and ECU data. Data generated from the CARLA simulator is transmitted through the CARLA ROS bridge (310), and the received vehicle information and sensor data (320) are processed like several APIs for autonomous driving in each ROS NODE. Semi-autonomous driving is performed using ROS MSG (ROS Message) data sent to the API of the module for semi-autonomous driving.

Figure 3 is an RQT graph supporting ROS showing the process of manual driving and semi-autonomous driving control, where the ellipse is expressed as ROS NODE and the square is expressed as ROS MSG. Control and information confirmation of CARLA client objects on the CARLA simulator server are managed using ROS CARLA Message. At this time, Make_node (330) is a ROS NODE that controls manual driving and semi-autonomous driving and is an API module related to vehicle control. When controlling a manual driving vehicle, vehicle_control_cmd Message (340) is used, and control can be made by entering the throttle, brake, and steering values using the corresponding message values. Each data value has a real number from 0 to 100. The throttle value receives the physical accelerator value and transmits it to the throttle to control the vehicle speed. Semi-autonomous driving classifies the driver's emotions and situations and executes a driving scenario appropriate to the situation. As a control message, use akermann_cmd (350) and carla_akemann_control_ego_vehicle ROS node (360) for throttle, brake, and steering values by settings (Steering_angle, Steering_angle_velocity, Speed, Acceleration, Jerk). PID control vehicle_control_cmd. The processed value is sent to the virtual vehicle on the CARLA simulator as a ROS Message.

Figure 4 is a flowchart for explaining a semi-autonomous driving control process according to an embodiment of the present invention.

DAS (Driver Assistance Systems) according to an embodiment of the present invention is configured based on HLIS [1]. It receives HMI result data (410) to control the virtual engine and vehicle through the ECU. Afterwards, the data 421 is converted into CAN data 420 to be transmitted to the virtual vehicle. CAN data 420 includes HMI result data 410 and ECU accelerator and brake values 421, which are data for the virtual vehicle. In semi-autonomous driving mode, it is divided into driver control and autonomous control (430).

When controlling the driver, the physical accelerator and brake values configured by hardware are received as ADC data (441). At this time, the received data serves to control (442) the throttle and brake values of the virtual vehicle.

In autonomous control, autonomous driving (454) is executed according to the scenario that matches situations 1, 2, and 3 (453) based on situation data, and in situation 4 (452), the driver is controlled with normal driving ability. It is executed with (442).

Figure 5 is a diagram for explaining a DAS communication frame according to an embodiment of the present invention.

Communication between hardware used for vehicle control according to an embodiment of the present invention uses serial (510) and CAN (520). A frame in serial communication transmitted from HMI consists of a total of 3 bytes.

The 0 index of Serial (510) is the 0xFF value and is a synchronization signal, the 1 index is the HMI result data value, and the 2 index is the data end value using LF (Line Feed).

The CAN data frame 520 uses a total of 3 bytes, and each 1 byte contains different information. The 0 index data has the HMI result data value, and the CAN 1 and 2 index values have the Excel Throttle value and brake, so when controlling the driver, the data is used based on the value included in the 1.2 index.

Figure 6 is a configuration diagram for DAS according to an embodiment of the present invention.

According to an embodiment of the present invention, the CAN communication environment used in an actual vehicle environment is configured through an ECU board. In order to implement the communication method between the HMI and the simulator vehicle, the environment was configured as shown in Figure 6. In the HMI environment, the ECU board 620 converts the result data 610 received through serial communication and the Accel and Brake values configured by hardware into CAN data and transmits it to the driving simulator. The receiving driving simulator PC was based on the Linux operating system. Through Socket CAN, an open source Linux kernel, CAN data can be received from a driving simulator PC. The driving simulator PC processes the control stage with the CAN received using the Semi-Autonomous Driving API through ROS (630) and sends control commands as ROS topic messages to the vehicle (640) in the CARLA simulator. send to

Figure 7 is a diagram dividing a two-dimensional area into four areas to control a vehicle based on the driver's situation according to an embodiment of the present invention.

In the present invention, we designed a system that recognizes the driver's situation when switching between manual driving and autonomous driving and prevents additional traffic rule violations and accidents. As in the study case mentioned above, emotional state often has a significant impact on a driver's driving ability. Additionally, drivers' emotions are correlated with surrounding situations, and they lack the ability to calm themselves down when they feel frustrated or angry. Therefore, in order to control the vehicle based on the driver's situation, such as certain emotions, the two-dimensional area was divided into four areas.

The area in Figure 7(a) recognizes the driver's speed perception ability to prevent vehicle traffic accidents caused by excessive acceleration. If the driver is in a state of excessive happiness while driving the vehicle, it can have a negative impact on driving ability and causes the driver to drive at higher speeds without being able to concentrate on the speedometer and speed control. For accurate comparison, the experimenters were asked to drive while listening to exciting music and calm music. As a result of measuring the vehicle's average speed and TCL (Traction Control System), it was confirmed that while driving while listening to happy and exciting music, drivers were distracted and their concentration was weakened.

The area in Figure 7(b) recognizes the driver's perception and response ability when an unexpected situation occurs while driving. Anger and feelings of anger that arise from the environment outside the vehicle (e.g., traffic jams, other drivers' arguments, etc.) can be seen to increase the driver's aggressiveness, dangerous behavior, and the time it takes to crash. In addition, we investigated whether the causes and factors related to anger while driving can affect driver behavior, and the results of the experiment showed that driver emotions (e.g., anger, anger) were related and specific to social deviance and driving violation behavior. It was composed.

The area in Figure 7(c) is the driver's awareness of situational judgment and recognition ability regarding vehicle situations and driving inattention while driving. The problem of fatigue and drowsiness while driving a vehicle is a major research area to date, and various investigations and experiments have been conducted. In addition, driver drowsiness that occurs while driving was analyzed based on EEG signals, and the impact on the driver's condition and driving performance was derived through real-time monitoring, resulting in inattentive driving as a result of analysis including biometric and physical signals for drowsiness. The categories were divided into distraction and boredom.

The region in Figure 7(d) is the recognition of driving ability in a normal state in which the driver is not affected by any emotions. When switching to manual driving normally, the driver drives the vehicle while changing the throttle value to directly control the accelerator. At this time, stable driving is possible because the driver's situation can be normally reflected in vehicle control when the control method is switched.

Figure 8 is a diagram showing the proposed 1D CNN model for driver's situation and emotion recognition according to an embodiment of the present invention.

In order to recognize the driver's situation and emotions while driving, fast emotional recognition based on biometric signals is required. However, autonomic nervous system signals that humans cannot control have few characteristic changes due to emotional changes, and many signals lack regularity. Therefore, existing research conducted emotion recognition by extracting features from raw biosignals. In the prior art, features were extracted from biosignals using power spectrum density (PSD), and in another prior art, topography was used. However, it is difficult to apply it to real-time emotion recognition due to the time and performance required in the feature extraction process.

The biosignals used in the present invention are PPG[2] and GSR[3] signals that are easy to acquire and have certain regularities. Additionally, a method to extract signal features was not used for fast data preprocessing and calculation. Since raw data is difficult to use directly as training data and data for real-time emotion recognition, a Butterworth Filter is applied to remove low-frequency components of the data, and a high-order polynomial and moving average filter are used to determine baseline fluctuation and Dynamic noise was reduced. The preprocessed biosignals were divided into short 1.1-second waveform units, and data characteristics and learning were performed through a 1D CNN model.

An artificial neural network (ANN) consists of three layers: input, hidden, and output layers. However, existing artificial neural networks have difficulty finding optimal values for parameters and are vulnerable to distortion due to movement and change. 1D CNN, an improved model, is based on the weights and biases of the previous layer, just like ANN, but has a structure that extracts data features and identifies rules. Therefore, recent research is applying a method to extract and classify signal features using a 1D CNN model for various voice and bio signals [4]. The configuration diagram of the multimodal one-dimensional convolutional neural network used in the present invention is shown in Figure 8.

With the result data of the multimodal biosignal-based 1D CNN model, a study on the driving assistance control module when the driver drives the vehicle was conducted on a simulator. Even in autonomous driving, it is possible to develop an autonomous driving system through human emotions, and it can be seen that interaction between the driver and the vehicle is necessary when manually controlling the vehicle and switching modes. The experimental results confirmed the possibility of real-time vehicle control through the driver's situational awareness, but unlike robots, human emotions have the characteristic of not changing rapidly. Therefore, stored bio-signal data was used for accurate vehicle control in the simulator. If the system is studied for each situation by using it with various autonomous driving sensors in the future, development and research will be conducted on various driving assistance systems and autonomous driving systems. It can be confirmed that the new module equipped with the driver's situation and emotion recognition proposed in the present invention is possible to converge various fields and autonomous driving research by approaching it as a center of interaction between passengers and vehicles rather than a vehicle-centered existing autonomous driving system module.

In the present invention, the possibility of a module for vehicle driving speed control based on multimodal biosignals was confirmed. To analyze the driver's emotions, a vehicle speed control and driving assistance system module was proposed using input data as short as 1.1 seconds and a 1D CNN model without separate feature extraction. To create an environment similar to a real vehicle, a virtual city and vehicle environment was configured on the CARLA simulator server, and an ECU board was used to configure the same communication system as the actual vehicle. In addition, CAN communication, which is widely used for in-vehicle communication, situation scenarios, and Excel data were transmitted to the virtual environment vehicle. As a result of real-time monitoring of data managed by middleware ROS and comparison of measured values and target values, it is possible to see that the shape of the target graph and the graph derived from the experimental results match. In addition, the average reaction time for controlling autonomous vehicles is 830ms, which has a stable distribution in commercialization and actual application. In the experimental results, the average reaction speed was 351.75ms, confirming the possibility of use when controlling a vehicle.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (8)

HMI (Human Machine Interface) that classifies the driver's emotions and situations using a CNN model based on the driver's biometric signals and transmits the situation recognition result data to the ECU board;
An ECU (Electronic Control Unit) board that generates multiple scenarios regarding a semi-autonomous driving virtual environment using the received HMI result data;
ROS (Robot operating system), which manages vehicle information and sensor data about the semi-autonomous driving virtual environment to execute semi-autonomous driving through CARLA (Car Learning to Act) simulator; and
CARLA server that executes semi-autonomous driving of the CARLA client according to a plurality of scenarios related to the semi-autonomous driving virtual environment using the vehicle information and sensor data
A human-machine interaction system including. According to paragraph 1,
The HMI is,
Butterworth uses PPG signals and GSR signals to use the CNN model based on the driver's biometric signals, and removes low-frequency components of the raw data to directly use the raw data as training data and data for real-time emotion recognition. By applying a filter (Butterworth Filter) and using a high-order polynomial and moving average filter, baseline fluctuations and dynamic noise are reduced, and the preprocessed biosignals are divided into waveform units and data is learned through a 1D CNN model.
Human-machine interaction systems. According to paragraph 1,
The ROS is,
Data from the CARLA simulator and ECU board are linked through the ROS Bridge, and the received vehicle information and sensor data are processed for semi-autonomous driving control at each ROS node, CARLA server manages vehicle information and sensor data to execute semi-autonomous driving using ROS message data sent through API (Application Programming Interface).
Human-machine interaction systems. According to paragraph 1,
The CARLA server is,
In order to transmit the vehicle information and sensor data as data to the virtual vehicle, it is converted into CAN data, and the HMI result data and the vehicle information and sensor data are used to generate driver's emotions and Controls the CARLA client's virtual vehicle to execute driver control or autonomous control according to multiple scenarios depending on the situation.
Human-machine interaction systems. A human machine interface (HMI) classifies the driver's emotions and situations using a CNN model based on the driver's biosignals and transmits the situation recognition result data to the ECU board;
Generating a plurality of scenarios related to a semi-autonomous driving virtual environment using HMI result data received by an Electronic Control Unit (ECU) board;
A step in which ROS (Robot operating system) manages vehicle information and sensor data regarding a semi-autonomous driving virtual environment to execute semi-autonomous driving through a CARLA (Car Learning to Act) simulator; and
A CARLA server executing semi-autonomous driving of the CARLA client according to a plurality of scenarios related to the semi-autonomous driving virtual environment using the vehicle information and sensor data.
A human-machine interaction method including. According to clause 5,
The step where the HMI classifies the driver's emotions and situations using a CNN model based on the driver's biometric signals and transmits the situation recognition result data to the ECU board,
Butterworth uses PPG signals and GSR signals to use the CNN model based on the driver's biometric signals, and removes low-frequency components of the raw data to directly use the raw data as training data and data for real-time emotion recognition. By applying a filter (Butterworth Filter) and using a high-order polynomial and moving average filter, baseline fluctuations and dynamic noise are reduced, and the preprocessed biosignals are divided into waveform units and data is learned through a 1D CNN model.
Human-machine interaction methods. According to clause 5,
The step in which the ROS manages vehicle information and sensor data about the semi-autonomous driving virtual environment to execute semi-autonomous driving through the CARLA (Car Learning to Act) simulator is,
Data from the CARLA simulator and ECU board are linked through the ROS Bridge, and the received vehicle information and sensor data are processed for semi-autonomous driving control at each ROS node, CARLA server manages vehicle information and sensor data to execute semi-autonomous driving using ROS message data sent through API (Application Programming Interface).
Human-machine interaction methods. According to clause 5,
The step of the CARLA server executing semi-autonomous driving of the CARLA client according to a plurality of scenarios related to the semi-autonomous driving virtual environment using the vehicle information and sensor data,
In order to transmit the vehicle information and sensor data as data to the virtual vehicle, the vehicle information and sensor data are converted into CAN data, and the HMI result data and the vehicle information and sensor data are used to generate the driver's emotions and Controls the CARLA client's virtual vehicle to execute driver control or autonomous control according to multiple scenarios depending on the situation.
Human-machine interaction methods.

Images