JP2024071249A - Vehicle control device - Google Patents

Abstract

[Problem] To provide a vehicle control device that can prevent collisions in vehicle control based on emotions and situations where it is not possible to respond to any of the occupants. [Solution] The vehicle control device includes a processor that uses a trained model to estimate the emotions of each occupant of the vehicle, calculates the rate of change in the emotions of each occupant based on time-series data of the estimated emotions, selects a target occupant from multiple occupants based on the calculated rate of change in emotions, and executes a predetermined vehicle control based on the estimation result of the emotion of the selected target occupant. [Selected Figure] Figure 1

Description

This disclosure relates to a vehicle control device.

Patent document 1 proposes a technology that estimates the emotions of the driver or passengers and controls the vehicle's driving based on the estimation results.

JP 2017-136922 A

In the technology proposed in Patent Document 1, it is possible to control the vehicle using, for example, a trained machine learning model (trained model). In the technology proposed in Patent Document 1, when there are multiple occupants in the vehicle, the emotions of at least some of the occupants may differ from those of the other occupants. In this case, a collision of vehicle control based on emotions may occur, or the vehicle may fall into a situation where it is not possible to respond to any of the occupants.

The present disclosure has been made in consideration of the above, and aims to provide a vehicle control device that can prevent collisions caused by emotion-based vehicle control, or situations in which the vehicle is unable to respond to any of the occupants.

The vehicle control device according to the present disclosure includes a processor that uses a trained model to estimate the emotions of each occupant of the vehicle, calculates the rate of change in the emotions of each occupant based on time series data of the estimated emotions, selects a target occupant from among multiple occupants based on the calculated rate of change in emotions, and executes a predetermined vehicle control based on the estimation result of the emotion of the selected target occupant.

According to the present disclosure, the emotional change rate of each vehicle occupant is calculated, and vehicle control is given priority to occupants with a high emotional change rate, thereby preventing collisions in vehicle control based on emotions and situations where the vehicle cannot respond to any of the occupants.

FIG. 1 is a block diagram showing a schematic configuration of a vehicle control device according to an embodiment. FIG. 2 is a diagram showing the flow of information in each unit of the vehicle control device according to the embodiment. FIG. 3 is a graph for explaining a case where an occupant is selected from a differential value of emotion data in the control target selection unit of the vehicle control device according to the embodiment. FIG. 4 is a graph for explaining a case in which an occupant is selected from the differential value and integral value of emotion data in the control target selection unit of the vehicle control device according to the embodiment. FIG. 5 is a flowchart illustrating an example of a processing procedure of a vehicle control method executed by the vehicle control device according to the embodiment.

A vehicle control device according to an embodiment of the present disclosure will be described with reference to the drawings. Note that the components in the following embodiments include those that are easily replaceable by a person skilled in the art, or those that are substantially identical.

(Vehicle control device)
The vehicle control device 1 is for controlling a vehicle. The vehicle control device 1 may be mounted on the vehicle, or may be realized by a server device or the like separate from the vehicle. In this embodiment, the description will be given on the assumption that the vehicle control device 1 is mounted on the vehicle.

When the vehicle control device 1 is realized by a server device, the vehicle and the server device are connected by a network, such as an Internet network or a mobile phone network. The server device, which is the vehicle control device 1, remotely controls the vehicle by communicating through the vehicle's communication unit (Data Communication Module: DCM).

As shown in FIG. 1, the vehicle control device 1 includes a control unit 10, a storage unit 20, and a sensor group 30. The control unit 10 includes a processor and a memory (main storage unit). Specifically, the processor includes a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), etc. The memory includes a RAM (Random Access Memory), a ROM (Read Only Memory), etc.

The control unit 10 loads the programs stored in the memory unit 20 into the working area of the main memory unit, executes them, and realizes functions that meet a predetermined purpose by controlling each component part through the execution of the programs. Through the execution of the programs stored in the memory unit 20, the control unit 10 functions as an emotion estimation unit 11, a control target selection unit 12, an occupant behavior detection unit 13, and a vehicle control unit 14.

The emotion estimation unit 11 estimates the emotion of each occupant in the vehicle. The emotion estimation unit 11 estimates the emotion of each occupant based on, for example, sensor data (image data, biometric data) from the sensor group 30. The emotion estimation unit 11 estimates the emotion of each occupant in the vehicle from the above sensor data, for example, by using a trained machine learning model (trained model).

The "emotions of each occupant" estimated by the emotion estimation unit 11 is, for example, information about the comfort or discomfort of the occupant, and can be expressed, for example, as a numerical value (probability) of the comfort or discomfort. The estimation of the emotions of each occupant by the emotion estimation unit 11 is repeatedly performed at a predetermined control cycle.

As shown in FIG. 2, the emotion estimation unit 11 acquires image data of the occupant, for example, from a camera in the sensor group 30. The emotion estimation unit 11 then estimates the emotion of the occupant based on the acquired image data. In this case, the emotion estimation unit 11 calculates a numerical value (probability) of the comfort or discomfort of the occupant, for example, based on the facial expression of the occupant included in the image data.

As shown in FIG. 2, the emotion estimation unit 11 acquires biometric data of the occupant, for example, from a biosensor in the sensor group 30. The emotion estimation unit 11 then estimates the emotion of the occupant based on the acquired biometric data. In this case, the emotion estimation unit 11 calculates the numerical value (probability) of the comfort or discomfort of the occupant based on information such as the occupant's body temperature, heart rate, pulse rate, blood pressure, and brain waves contained in the biometric data.

The emotion estimation unit 11 can perform the above emotion estimation, for example, by using a trained machine learning model (trained model). In this case, the input data input to the machine learning model during emotion estimation is, for example, image data, biometric data, etc. of each occupant. Moreover, the output data output from the machine learning model is, for example, a numerical value (probability) of the occupant's comfort or discomfort.

The method for constructing the machine learning model used in the emotion estimation unit 11 is not particularly limited, and various methods such as deep learning using neural networks, support vector machines, decision trees, naive Bayes, and k-nearest neighbor methods can be used.

The control target selection unit 12 calculates the rate of change in the emotion data of each occupant based on the time series data of emotions (hereinafter referred to as "emotion data") estimated by the emotion estimation unit 11. Then, the control target selection unit 12 selects, from among the multiple occupants, an occupant to be controlled by the vehicle control unit 14 (an occupant of interest) based on the calculated rate of change in the emotion data. Specifically, the control target selection unit 12 calculates the rate of change in the emotion data for each occupant, and selects, from among the occupants, the occupant with the largest rate of change in the emotion data as the occupant of interest.

Here, the "change rate of emotion data" refers to, for example, the change rate of the occupant's emotion from "pleasant to unpleasant." The change rate of emotion data can be indicated by the differential value of the emotion data of each occupant, as shown in FIG. 3, for example. This figure is a graph for explaining a case in which the control target selection unit 12 selects an occupant from the differential value of the emotion data. In this figure, the vertical axis represents the numerical value (probability) of the occupant's pleasantness or unpleasantness estimated by the emotion estimation unit 11, and the horizontal axis represents time.

In the example of FIG. 3, the emotion data of occupant 1 drops sharply from the "comfortable" side to the "uncomfortable" side. Therefore, the control target selection unit 12 determines that occupant 1 is the one who feels more uncomfortable, and selects this occupant 1 as the control target. Note that when selecting the control target, a predetermined evaluation period and threshold value may be set and used for the judgment, as shown in the figure.

As the rate of change of emotion data, for example, as shown in FIG. 4, in addition to the differential value of the emotion data of each occupant, the integral value may be used. This figure is a graph for explaining a case where the control target selection unit 12 selects an occupant from the differential value and integral value of the emotion data. In this figure, the vertical axis represents the numerical value (probability) of the comfort or discomfort of the occupant estimated by the emotion estimation unit 11, and the horizontal axis represents time.

In the example of Figure 4, when looking only at the differential value of the emotion data, the rate of change for occupant 2 is higher than that for occupant 1. On the other hand, the integral value of the emotion data is larger for occupant 1 than for occupant 2. Therefore, the control target selection unit 12 determines that occupant 1 feels more uncomfortable, and selects occupant 1 as the control target.

The occupant behavior detection unit 13 detects the behavior (situation) of each occupant based on, for example, sensor data (image data, sensor data) from the sensor group 30. For example, the occupant behavior detection unit 13 detects the situation of each occupant in the vehicle, such as "started eating or drinking," "looks sleepy," "looks hot," or "looks cold." Note that, like the emotion estimation unit 11, the occupant behavior detection unit 13 may detect the behavior of each occupant using a trained machine learning model (trained model).

The vehicle control unit 14 executes a predetermined vehicle control based on the estimation result of the emotion of the occupant (occupant of interest) selected by the control target selection unit 12. "Predetermined vehicle control" refers to control according to the emotion of the selected occupant, and examples of the control include the following:

(1) Implement a vehicle operation that has a high rate of change from "uncomfortable to comfortable" for the selected occupant (so that the occupant becomes comfortable). Data on this vehicle operation is collected in advance from the occupant.
(2) Do not execute (stop) a vehicle operation that has a high rate of change from "comfortable to uncomfortable" for the selected occupant (a vehicle operation that makes the occupant uncomfortable). Data on this vehicle operation is collected in advance from the occupant.

The vehicle control unit 14 may also execute vehicle control for the occupant selected by the control target selection unit 12 in response to the behavior detected by the occupant behavior detection unit 13. In this case, for example, the following control can be performed.

(1) When a selected occupant begins to eat or drink, vehicle operations such as reducing vehicle acceleration, adjusting the curvature of curves, and notifying the driver of the location of restrooms are performed.
(2) If the selected occupant appears sleepy, a vehicle operation such as lowering the volume of music in the vehicle is performed.
(3) If the selected occupant looks hot, a vehicle operation is performed, such as turning on the air conditioner (cooling) or turning off the air conditioner (heating) or opening the windows.
(4) If the selected passenger looks cold, a vehicle operation is performed such as turning off the air conditioner (cooling) or turning on the air conditioner (heating) or closing the windows.

In addition to the above, the vehicle control unit 14 may also perform vehicle operations that are tailored to the seating position and personality of the selected occupant. Note that "personality of the occupant" refers to, for example, the age, adult/child status, and gender of the occupant. In this case, the seating position and personality of the occupant may be detected, for example, by a camera or the like in the sensor group 30, or data may be collected in advance from the occupant.

The vehicle control unit 14 can perform the above vehicle control using a trained machine learning model (trained model) or based on predetermined rules. When a machine learning model is used, the input data input to the machine learning model is, for example, information related to the emotion estimation result of the occupant selected by the control target selection unit 12, the behavior (situation) detected by the occupant behavior detection unit 13, the seating position of the occupant, the personality of the occupant, etc. Also, the output data output from the machine learning model is, for example, information related to the above vehicle operation for the occupant.

The method for constructing the machine learning model used by the vehicle control unit 14 is not particularly limited, and various methods such as deep learning using neural networks, support vector machines, decision trees, naive Bayes, and k-nearest neighbor methods can be used.

The storage unit 20 is realized by a recording medium such as an EPROM (Erasable Programmable ROM), a hard disk drive (HDD), and a removable medium. Examples of removable media include disk recording media such as a Universal Serial Bus (USB) memory, a Compact Disc (CD), a Digital Versatile Disc (DVD), and a Blu-ray (registered trademark) Disc (BD).

The storage unit 20 can store an operating system (OS), various programs, various tables, various databases, and the like. The storage unit 20 may also store, for example, emotion estimation results from the emotion estimation unit 11, information about the occupant selected by the control target selection unit 12, information about the behavior of the occupant detected by the occupant behavior detection unit 13, and the like. The storage unit 20 may also store machine learning models (trained models) used by the emotion estimation unit 11 and the vehicle control unit 14, emotion data estimated by the emotion estimation unit 11, and the like.

The sensor group 30 acquires data for detecting the status of each occupant. Examples of the sensor group 30 include a camera for capturing an image of each occupant, a temperature sensor and an infrared sensor for detecting the status of each occupant, and a biosensor for acquiring biometric data of each occupant. Examples of biosensors include a temperature sensor, a heart rate sensor, a pulse sensor, a blood pressure sensor, and an electroencephalogram sensor.

(Vehicle control method)
An example of a processing procedure of a vehicle control method executed by the vehicle control device according to the embodiment will be described with reference to FIG.

First, the emotion estimation unit 11 estimates the emotion of each occupant based on, for example, sensor data (image data, biometric data) from the sensor group 30 (step S1). Next, the control target selection unit 12 calculates the rate of change (e.g., differential value) of the emotion data of each occupant estimated in step S1 (step S2).

Next, the control target selection unit 12 selects the occupant with the highest rate of change in emotion data from among the occupants of the vehicle (step S3). Next, the vehicle control unit 14 implements a predetermined vehicle control for the occupant selected in step S3 (step S4), and completes this process.

According to the vehicle control device of the embodiment described above, the rate of change in the emotion data of each vehicle occupant is calculated, and vehicle control is given priority to occupants with a high rate of change. This allows vehicle control to be performed for each occupant based on a clear priority order, making it possible to prevent collisions in emotion-based vehicle control and situations in which it is not possible to respond to any occupant.

That is, the vehicle control device according to the embodiment uses a trained model to estimate the emotions of each occupant and records the estimation results in a time series. Next, the rate of change of the emotion data is calculated from the time series records, and an occupant of interest is selected from the multiple occupants based on the calculated rate of change of the emotion data. In this way, the vehicle control device according to the embodiment can appropriately determine the priority of each occupant based on the rate of change of the emotion data, thereby suppressing problems such as collisions in vehicle control based on emotions and an inability to respond to multiple occupants.

In addition, in the vehicle control device according to the embodiment, by using the rate of change of emotion data, it is possible to identify vehicle control caused by, for example, a rapid change from "pleasant to unpleasant" or "unpleasant to pleasant." For example, if the change is "pleasant to unpleasant," the vehicle control is corrected in a direction that suppresses such vehicle control, and if the change is "unpleasant to pleasant," the vehicle control is corrected in a direction that promotes such vehicle control, thereby preventing all occupants from becoming excessively uncomfortable. As a result, it is possible to expect comfortable vehicle control for all occupants.

Further advantages and modifications may readily occur to those skilled in the art. Thus, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof.

REFERENCE SIGNS LIST 1 vehicle control device 10 control unit 11 emotion estimation unit 12 control target selection unit 13 occupant behavior detection unit 14 vehicle control unit 20 storage unit 30 sensor group

Claims (1)

A processor is provided.
The processor,
Use the trained model to estimate the emotions of each occupant in the vehicle,
Calculate the rate of change of each passenger's emotion based on the estimated emotion time series data,
Selecting a target occupant from the plurality of occupants based on the calculated change rate of emotion;
Executing a predetermined vehicle control based on the estimation result of the emotion of the selected occupant of interest.
Vehicle control device.

Images