JP2020064492A - Operation support method and operation support device - Google Patents

Abstract

To provide an operation support method and an operation support device which appropriately support an operation even when an object to be operated is not specified.SOLUTION: An operation support method allows execution of a processor 11 of an operation support device for outputting support information related to operation of a user using a function provided on a device of a vehicle, in which the processor 11 acquires operation information on a user, estimates an operation of a function to be executed by the user on the basis of the operation information on the user, at least temporarily stores the operation of the function in a storage device 320, refers to operation history information 321 including an operation required time required for completion of input of an operation command for executing the function, determines a support mode of the support information supporting the estimated operation of the function to be executed by the user on the basis of the operation required time for each of the users, and outputs the support information in the support mode.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an operation support method and an operation support device.

  Regarding this type of device, a system is known in which, when a user uses a function provided in an information terminal, a ratio of the user's operation completion is calculated as an operation achievement rate, and operation support is provided based on the operation achievement rate. (Patent Document 1).

JP, 2014-142744, A

  In the conventional technology, operation support is provided only for the functions of the information terminal. However, it may be difficult to provide appropriate support information when the operation target of the function to be executed by the user cannot be specified, as in a vehicle including various devices.

  The problem to be solved by the present invention is to provide an operation support method and an operation support apparatus that provide appropriate operation support even when an operation target is not specified.

  The present invention estimates the operation of a function to be executed by the user based on the acquired operation information of the user, is at least temporarily stored in the storage device, and requires the operation command by the user to execute the function. By referring to the operation history information including the operation required time, based on the operation required time for each user, by determining the support mode of the support information for supporting the estimated operation, by outputting the support information in the support mode, The above problems are solved.

  According to the present invention, since the function to be executed by the user is estimated, appropriate operation support can be provided even when the operation target is not specified.

It is a block configuration diagram of an operation support system according to the present embodiment. It is a functional block diagram of the operation support device of this embodiment. It is a flow chart which shows a procedure of output processing of operation support information concerning this embodiment. It is a figure which shows an example of the operation corresponding information of this embodiment. It is a figure which shows an example of the operation log information containing the operation required time of this embodiment. It is a figure which shows an example of the operation log information containing the operation evaluation time of this embodiment. It is a figure which shows an example of the operation part of the vehicle-mounted apparatus of this embodiment. It is a figure which shows the 1st example of the support information of this embodiment. It is a figure which shows the 2nd example of the support information of this embodiment. It is a figure which shows the 3rd example of the support information of this embodiment. It is a figure which shows the 4th example of the assistance information of this embodiment. It is a figure which shows the 5th example of the assistance information of this embodiment. It is a flow chart which shows the 1st procedure of creation processing of operation history information concerning this embodiment. It is a figure for demonstrating the 1st example of the operation history information of this embodiment. It is a flow chart which shows the 2nd procedure of creation processing of operation history information concerning this embodiment. It is a figure for demonstrating the 2nd example of the operation log information of this embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, a case where the operation support method and the operation support apparatus according to the present invention are applied to an operation support system that cooperates with an in-vehicle device 200 mounted on a vehicle and a server 300 will be described as an example. The in-vehicle device 200 is a device (group) mounted in the vehicle.

  FIG. 1 is a diagram showing a block configuration of the operation support system 1. The operation support system 1 of the present embodiment includes an operation support device 100, an in-vehicle device 200, and a server 300. The embodiment of the operation support device 100 of the present invention is not limited, and may be mounted in a vehicle or may be applied to a portable terminal device capable of exchanging information with the in-vehicle device 200. The terminal device includes devices such as a smartphone and a PDA. The operation support system 1, the operation support device 100, the in-vehicle device 200, and each device included in these devices are computers that include an arithmetic processing device such as a CPU and execute arithmetic processing.

First, the vehicle-mounted device 200 will be described.
The in-vehicle device 200 of the present embodiment includes various in-vehicle devices. The in-vehicle devices include the communication device 210, the input / output device 220, the detection device 230, the vehicle controller 240, the automatic driving device 250, the in-vehicle sensor 260, the navigation device 270, the accessory 280, and the airbag device 291. At least one or more is included. The in-vehicle devices included in the in-vehicle apparatus 200 are connected by a CAN (Controller Area Network) or other in-vehicle LAN in order to exchange information with each other. The in-vehicle device 200 can exchange information with the operation support device 100 via the in-vehicle LAN.

  The communication device 210 exchanges information between the on-vehicle devices. The communication device 210 exchanges information with the in-vehicle device 200, the operation support device 100, and the server 300. The communication device 210 communicates with an external customer service center.

  The input / output device 220 includes at least one of a display 221, a microphone 222, a switch 223, a speaker 224, and a camera 226. The input / output device 220 is an operation target when executing a function desired by the user. The display 221 presents various information including operation support information including an image including text. The display 221 is a touch panel display, presents information, and accepts a user's input command. The microphone 222 acquires the voice uttered by the user. The switch 223 receives the input operation of the user. The switch 223 can be configured by incorporating a light emitting element 225 such as an LED. By providing the switch 223 with a light emitting function, the location of the switch 223 can be indicated to the user by light emission. The speaker 224 outputs various information including the operation support information by uttering a text. The camera 226 is arranged inside and outside the vehicle. The camera 226 is an imaging device including an imaging device such as a CCD. The camera 226 may be an infrared camera. The camera 226 receives an input operation by a user's gesture. The camera 226 images an object around the vehicle. The objects include pedestrians, vehicles such as two-wheeled vehicles and four-wheeled vehicles (other vehicles), road structures such as guardrails, median strips, and curbs. These pieces of information are used for automatic driving or driving support.

The detection device 230 acquires detection information on the inside and outside of the vehicle. The detection device 230 includes any one or more of an image processing device 231, a voice processing device 232, a door sensor 233, a seating sensor 234, a thermometer 235, a position sensor 236, and a shift position sensor 237. Including.
The image processing device 231 analyzes the captured image captured by the camera 226. The image processing device 231 sets the coordinates of the imaging area based on the arrangement position of the camera 226. The image processing device 231 extracts the user's hand, leg, or head from the captured images acquired over time by using a pattern matching technique or the like, extracts the movement, and extracts the user's hand, leg, or Alternatively, the position of head movement is calculated. The image processing device 231 extracts the direction of the line of sight from the head, face, and eye movements of the user from the captured images acquired over time, and calculates the position at which the user's line of sight is directed and the transition thereof. These pieces of information are sent to the operation support device 100. The user includes both a driver and an occupant.
Further, the image processing device 231 may analyze the image data of the camera 226 provided outside the vehicle and identify the type of the target object based on the analysis result. The image processing device 231 identifies whether the target object included in the image data is a vehicle, a pedestrian, or a sign by using a pattern matching technique or the like. The image processing device 231 processes the acquired image data, and acquires the distance from the vehicle to the object based on the position of the object existing around the vehicle. The image processing device 231 acquires the time when the vehicle reaches the target object based on the position and time of the target object existing around the vehicle.
The voice processing device 232 analyzes the voice data acquired by the speaker 224. The voice processing device 232 analyzes a user or a utterance between the users. A speech recognition process of cutting out a word / phrase from the utterance content and extracting a target word / phrase by referring to a dictionary uses a method known at the time of application. The voice processing device 232 acquires the utterance content of the user as text data. The voice processing device 232 acquires the conversation content between the users as text data.
The door sensor 233 acquires information from the vehicle controller 240 whether the door is unlocked or locked.
The seating sensor 234 detects the number of users where in the vehicle the users are located.
The thermometer 235 detects the temperature inside the vehicle compartment and the temperature outside the vehicle.
The position sensor 236 acquires the attribute of the area including the current position of the vehicle. The area attribute is acquired from the navigation device 270 described later.
The shift position sensor 237 acquires information on the shift position of the vehicle. The shift position information includes the identification information of the driving mode of the vehicle and the identification information of the parking mode.

  The detection device 230 may acquire the detection information from an external device via the communication device 210. If the communication device 210 can perform inter-vehicle communication between another vehicle and the vehicle, the detection device 230 indicates the vehicle speed and acceleration of the other vehicle detected by the vehicle speed sensor of the other vehicle to the effect that the other vehicle exists. You may acquire as object information. Of course, the detection device 230 can also acquire object information including the position, speed, and acceleration of another vehicle from the external device of the Intelligent Transport Systems (ITS) via the communication device 210. The detection device 230 outputs the detection result to one or more of the operation support device 100, the server 300, and the vehicle controller 240.

  The vehicle controller 240 is an in-vehicle computer such as an electronic control unit (ECU), and electronically controls driving / operation of the vehicle. The vehicle controller 240 centrally manages information regarding driving of the vehicle. The vehicle controller 240 manages control information of the automatic driving device 250. The vehicle controller 240 manages the operations of a drive device 251, a braking device 252, and a steering device 253 described later. The vehicle controller 240 acquires vehicle state information from the sensor 260. The sensor 260 has a steering angle sensor 261, a vehicle speed sensor 262, and a radar device 263.

  The automatic driving device 250 makes a driving plan and sends a control command to move the vehicle according to the made driving plan to the vehicle controller 240, and the vehicle controller 240 executes this. The operation plan includes control commands to the drive device 251, the braking device 252, and the steering device 253. The automatic driving device 250 includes a driving support function such as a lane keep support system. The functions of the automatic driving device 250 include a fully automatic driving function, a semi-automatic driving function, and a driving support function.

  The sensor 260 includes a steering angle sensor 261, a vehicle speed sensor 262, and a radar device 263. The steering angle sensor 261 detects information such as a steering amount, a steering speed, and a steering acceleration, and outputs the information to the vehicle controller 240. The vehicle speed sensor 262 detects the speed and / or acceleration of the vehicle and outputs it to the vehicle controller 240. The radar device 263 measures the distance to the object.

  The vehicle-mounted device 200 of this embodiment includes a navigation device 270. The navigation device 270 calculates the route from the current position of the vehicle to the destination by using the method known at the time of application. The calculated route is sent to the vehicle controller 240 for use in controlling the driving of the vehicle. The navigation device 270 includes a position detection device 271. The position detection device 271 includes a receiver of a Global Positioning System (GPS) and detects the traveling position (latitude / longitude) of the vehicle that is traveling.

  The navigation device 270 includes an accessible position detection device 271, map information 272, facility information 273, and an output device 274. The map information 272 and the facility information 273 need only be readable by the navigation device 270, may be physically configured separately from the navigation device 270, or may be stored in the server 300 that can be read via the communication device 210. May be stored. The map information 272 is a so-called electronic map, and is information in which latitude and longitude and map information are associated with each other. The facility information 273 includes an attribute in which the position is associated with the area including the position. The navigation device 270 can refer to the facility information 273 and determine the attribute of the area to which the current position belongs based on the current position. For example, it can be determined whether or not the host vehicle is located in the parking lot. The determination result is sent to the operation support device 100.

  The vehicle-mounted device 200 of this embodiment has an accessory 280 as a peripheral device. The accessory 280 includes at least one of the audio device 281, the air conditioner 282, and the external communication device 283. Each device is a device having a function known at the time of filing this application. The audio device 281 is a device that provides information including music. The operations of the audio device 281 include on / off operations, channel operations, and volume operations. The operation of the air conditioner 282 includes an on / off operation, a temperature adjusting operation, and an air volume adjusting operation.

  The vehicle-mounted device 200 of this embodiment includes an airbag device 291 and a self-diagnosis device 292. Each device is a device having a function known at the time of filing this application. These devices function autonomously, and the information on the deployment of the airbag is sent to the operation support device 100. Information in which a defect is detected in the self-diagnosis is sent to the operation support device 100.

Next, the operation support device 100 will be described.
The operation support device 100 includes a control device 10, a storage device 20, and an output device 30. The output device 30 includes a communication device 31. The output device 30 sends a control command to the input / output device 220 of the vehicle-mounted device via the communication device 31. The control device 10, the storage device 20, and the output device 30 can exchange information with each other via a wired or wireless communication line. The control device 10 exchanges information with the storage device 320 of the server 300 via the communication device 31. The communication device 31 exchanges information with the in-vehicle device 200, exchanges information inside the operation support device 100, and exchanges information between the external device and the operation support system 1.

The control device 10 will be described.
The control device 10 includes a processor 11. The processor 11 is an arithmetic unit that generates and outputs support information regarding the operation of the in-vehicle device to a user who uses the function of the in-vehicle device.
Specifically, the processor 11 functions as the control device 10 by executing a program (ROM) that stores a program that causes the output control process of the support device to be executed and a program that is stored in this ROM. The computer includes a CPU (Central Processing Unit) as a circuit and a RAM (Random Access Memory) that functions as an accessible storage device.

The processor 11 according to the present embodiment executes processing according to the following method.
(1) Acquire user motion information,
(2) Guess the operation of the function to be executed by the user based on the operation information of the user,
(3) Refer to operation history information that is stored at least temporarily in the storage device and that includes the operation required time required for the operation command by the user to execute the function,
(4) Determine the support mode of the support information for supporting the estimated operation based on the operation time required for each user,
(5) The support information is output in the determined support mode.

FIG. 2 shows functional blocks of the processor 11 of the operation support device 100.
As illustrated in FIG. 2, the processor 11 includes a motion information acquisition unit 111, an operation estimation unit 112, a support mode determination unit 113, and an output control unit 114.

  The motion information acquisition unit 111 acquires the motion information of the user. The operation estimating unit 112 estimates the operation of the function to be executed by the user based on the operation information of the user. The support mode determination unit 113 refers to the operation history information 321 that is at least temporarily stored in the storage device 320 and includes the operation required time required by the user to perform an operation command to execute the function, and the operation required time for each user. Based on, the support mode of the support information for supporting the estimated operation is determined. In this example, the processor 11 refers to the operation history information 321 stored in the server 300. The operation history information 321 may be stored in the storage device 20 of the operation support device 100. The output control unit 114 outputs the support information in a predetermined support mode.

FIG. 3 is a flowchart for explaining the outline of the support information output processing.
In step S101, the processor 11 identifies a user. The user may be identified from the identification information of the vehicle key. The user may be identified from the user ID included in the application information for using the vehicle. The user may be identified using a pattern matching technique based on a captured image of the user captured by the camera 226 installed in the vehicle. The template used for the pattern matching process may be acquired from the data at the time of application for vehicle use, or may be acquired from the captured image at the time of first boarding.

  In step S101, the processor 11 acquires vehicle state information. The vehicle status information includes the type of vehicle driving mode, the speed of the vehicle, the traffic jam condition of the route on which the vehicle travels, the presence or absence of damage to the vehicle, the presence or absence of a failure or abnormality in the vehicle equipment (vehicle-mounted device), the vehicle position, the vehicle One or more of the indoor temperature, the outside air temperature of the vehicle, and the operation of the wiper of the vehicle.

  The type of vehicle driving mode is acquired from the vehicle controller 240. The vehicle speed is acquired from the vehicle speed sensor 262 or the vehicle controller 240. The traffic jam condition of the route on which the vehicle travels may be estimated from a change in vehicle speed or may be acquired from an external information providing device via the communication device 210. Whether the vehicle is damaged is acquired from the airbag device 291 or the vehicle controller 240. The presence / absence of a failure or abnormality of the vehicle device (vehicle-mounted device) is acquired from the self-diagnosis device 292 or the vehicle controller 240. The position of the vehicle is acquired from the position detection device 271 or the vehicle controller 240. The indoor temperature of the vehicle and the outside air temperature of the vehicle are acquired from the thermometer 235 or the air conditioner 282. The operation of the vehicle wiper is acquired from the vehicle controller 240.

In step S102, the processor 11 acquires the operation information of the user.
The processor 11 acquires motion information based on the movement of the user's hand, leg, or head extracted from the captured images captured by the camera 226 over time. The processor 11 sets coordinates in the vehicle interior space based on the position of the camera 226. The processor 11 detects, as “motion information”, a change over time in the position of the user's hand, leg, or head in the vehicle interior space from the captured images over time. The processor 11 detects, as “motion information”, a change over time in the position in the vehicle interior space where the user's head / face and line of sight are directed from the captured images over time. The technology at the time of filing can be applied to detect the direction of the line of sight of a person. For example, a method of detecting the position of the eye from the face to track the position of the iris of the eye, a method of detecting the position of the eye from the face, or a pattern matching method using the position of the iris as a reference can be used. The processor 11 determines the movement of the user. The processor 11 acquires motion information such as where the user's hand, finger, or arm is located and where the line of sight is directed based on image information and the like. Further, the processor 11 determines that the user is having trouble with the operation based on the motion information based on the motion of the user. The operation of the user when the operation takes time can be obtained in advance as a pattern. It can be determined that the user is taking time and effort based on the movement patterns of the hands, fingers, and arms, the frequency of repeated movements, and the like. Similarly, the movement of the user's line of sight when the operation is time-consuming can be obtained in advance as a pattern. It can be determined that the user is taking time and effort based on the position to which the line of sight is directed, the pattern of eye / head / face movement, the frequency with which the line of sight movement is repeated, and the like.

In the present embodiment, an example of a scene in which a user takes time to operate a function to be executed is shown.
(1) I want to switch to the automatic driving mode while traveling in the manual driving mode, but the driver or driving supervisor needs to monitor the front or the surroundings of the vehicle, so the automatic driving switch cannot be operated immediately, A scene that takes time to operate.
(2) In a vehicle used for car sharing, the position of the switch for releasing the seat belt is unknown, and it takes time to remove the seat belt.
(3) In a vehicle used for car sharing, etc., it takes time to open the door without knowing the position of the door open / close switch.
(4) A scene in which it is troublesome to operate the air conditioner, audio, and car navigation without knowing which switch should be operated by operating a device having a plurality of switches arranged side by side, such as an air conditioner, audio, or car navigation.
(5) A scene in which it takes time to operate the air conditioner, the audio, and the car navigation without knowing which hierarchy should be selected by operating a device having a large number of layers selected by the touch panel type switch, for example, car navigation.

The motion information includes information obtained by the user's utterance motion.
The processor 11 acquires the motion information based on the user's utterance information extracted from the voice information collected by the microphone 222. The processor 11 performs a voice recognition process on the collected voice information and acquires the uttered content. Based on the motion information, it is determined that the user is having trouble with the operation. The processor 11 determines that the user is having trouble performing the operation based on the motion information based on the utterance of the user.

In step S102, the processor 11 acquires the traveling environment information of the vehicle. The processor 11 determines the traveling environment of the vehicle based on the vehicle state information and changes in the vehicle state information. The traveling environment of a vehicle includes the type of road on which the vehicle travels, traffic congestion on the road on which the vehicle travels, attributes of the location of the vehicle (parking lot, etc.), temperature inside and outside the vehicle, damage and breakdown of the vehicle, etc. .
The vehicle state information acquired in step S101 and the traveling environment information acquired in step S102 are used in the subsequent step S105 in the process of estimating the operation of the function to be executed by the user.

  In step S103, the processor 11 detects the "operation search state" in which the user is having trouble performing the operation based on the "motion information" of the user. The processor 11 determines whether or not the operation state of the user is the “operation search state”. The mode of "motion information" for determining the "operation search state" can be defined in advance.

  In step S104, the processor 11 determines that the “operation search state” has continued for a predetermined time. In the processes of steps S103 to S104, the processor 11 does not input the operation command, even though the user's hand stays in the predetermined area for the predetermined time or longer based on the operation state of the user (YES in step S103). If the state has continued for the predetermined time (YES in step S104), it is determined that the user has trouble in the operation. When the movement of the finger of the user's hand is common to a predetermined pattern and the movement is repeated a plurality of times or repeated at a predetermined frequency, the processor 11 does not input an operation command for a predetermined time. First, it is determined that the user has trouble in operating the device. The processor 11 takes time for the user to operate the device when the state in which the operation command is not input continues for a predetermined time even though the frequency or the number of times the user's head tilts in the predetermined direction is equal to or more than the predetermined number. To judge. The processor 11 determines that the user takes time to operate the device when the state in which the operation command is not input continues for a predetermined time even if the frequency or the number of times the user's line of sight turns to the predetermined region is equal to or more than the predetermined number. To do.

  In steps S <b> 103 to S <b> 104, the processor 11 determines that the operation command is not input for a predetermined time or more, based on the operation state of the user, although the utterance content of the user includes a keyword that evokes a troublesome operation. It is determined that the user is having trouble with the operation when the operation is continued for a long time. Keywords that evokes time-consuming work are defined in advance. For example, a word relating to “lost” or “confused” such as “where is the switch?”, “Where is it?”, “Don't know” or “that?” Is defined in advance as a keyword that reminds the user's time and effort.

  If the determinations in step S103 and step S104 are YES and it is determined that the user is in the "operation search state" in which the operation instruction is input, the processor 11 inputs the operation instruction by the user in step S103. The timing at which the trial is started (timing at which time is detected) is at least temporarily stored as the “timing at which the operation of the function is started”. The judgment of the timing when the user starts to input the operation command can be defined as appropriate. For example, the “timing at which the operation of the function is started” may be calculated when the number of times the user's hand has been detected in a predetermined area in the vehicle interior within a predetermined time is a predetermined number or more. In this processing, the first timing when the user's hand is detected in the predetermined area may be set as "timing when the operation of the function is started", or the last timing when the user's hand is detected in the predetermined area is set as " The timing when the operation of the function is started ”may be used. The "timing at which the operation of the function is started" may be calculated when it is detected that the user's line of sight is directed to a predetermined area in the vehicle compartment a predetermined number of times within a predetermined time. In this process, the first timing when it is detected that the user's line of sight is directed to the predetermined region may be set as the “timing when the operation of the function is started”, or the last time when the user's line of sight is detected in the predetermined region. The timing may be set to “the timing when the operation of the function is started”. The predetermined area may be set in an area where an input unit such as a switch is arranged. The “operation required time” is from the “timing when the operation of the function is started” to the “timing when the operation of the function is completed” when the operation command is input.

  From the timing when it is determined that the operation of the function is started, if the elapsed time is a predetermined time or more without the operation command being input, the processor 11 indicates that the user takes time to perform the operation and needs assistance. to decide.

Then, the processing proceeds to step S105, and the processor 11 estimates the operation of the function to be executed by the user. The processor 11 provides the user with assistance information regarding the estimated operation.
The "predetermined time" that is the threshold for determining whether or not the user is taking a lot of time is the operation required time from the start of the operation of the function to the input of the operation command based on the past operation history information. It is preferable to set based on the above. The “predetermined time” is preferably set for each user. The setting method of the “predetermined time” is not particularly limited, but for example, a table of appropriate time of operation required time is stored for each function, and a deviation from a default value set based on the appropriate time is learned for each user, The appropriate time may be customized and the “predetermined time” may be calculated based on this appropriate time. The “predetermined time” for each function of each user may be a customized appropriate time (operation required time). The "predetermined time" is timed after the operation scene is detected, and the support information is output when the appropriate time has elapsed.
By setting a threshold value for determining the presence / absence of the effort based on the time actually taken by the user, it can be determined with high accuracy that the user took the effort.

  The processor 11 refers to the operation correspondence information and estimates the operation of the function to be executed by the user based on the acquired operation information. The processor 11 may download the operation correspondence information to the storage device 20 in a predetermined cycle, or may read the storage device 320 of the server 300. FIG. 4 shows an example of the operation correspondence information according to this embodiment.

The “operation correspondence information” shown in FIG. 4 includes user's motion information.
The processor 11 refers to the “operation correspondence information” in which the detected position of the motion information and the position of the operation unit of the vehicle-mounted device are associated with each other, and refers to the position of the operation unit that is the target of the user's motion extracted from the motion information. Based on that, the user infers the operation of the function to be executed.

  The operation correspondence information includes position information of the operation unit of the in-vehicle device included in the in-vehicle apparatus 200. The position of each operation unit at a predetermined coordinate is stored. The operation unit includes a display 221, a switch 223, a speaker 224, and a camera 226. The operation of the function accepted by each operation unit is appropriately set for each vehicle. The input operation unit of the display 221 includes operation of the navigation device 270 and input of the audio device 281. The switch 223 includes an operation mode switching command, an air conditioner 282, and an input operation unit of the external communication device 283. The speaker 224 and the camera 226 include an input operation unit of the external communication device 283.

The processor 11 extracts the target of the user's action from the action information. The target of the operation to be extracted is the operation unit (switch) of the function that the user intends to execute.
The processor 11 identifies the position of the user's motion target from the captured image at the coordinates with the camera 226 as a reference. The position of the user's motion target is, for example, the pattern of the user's movement, the position or range where the user's hand is present, or the position or range where the user's finger is present. The processor 11 refers to the operation correspondence information and determines the operation unit existing at the position of the target of the user's action.
Since the operation of the function to be executed by the user can be inferred from the position of the target of the user's action, it is possible to provide appropriate support information. By analyzing the movements of the user's hand and head, the operation unit of the function to be executed by the user can be estimated with high accuracy.

The processor 11 identifies the position of the target to which the user's line of sight is directed from the captured image at the coordinates with respect to the camera 226. The target position to which the user's line of sight is directed is, for example, the direction or range in which the user's face or head is directed, or the direction or range in which the black eye of the user's eye is directed. The processor 11 refers to the operation correspondence information and determines the operation unit existing in the direction in which the user's line of sight is directed.
Since the operation of the function to be executed by the user can be inferred from the operation information of the user, it is possible to provide appropriate support information. By analyzing the direction of the user's line of sight, it is possible to estimate with high accuracy the operation unit of the function that the user intends to execute.

  The processor 11 identifies the operation unit based on the word extracted from the voice information collected by the microphone 222. Keywords associated with the functions are defined in the operation correspondence information. For example, keywords such as “hot” and “cold” are associated with the operation unit of the air conditioner 282. When words similar to “hot” and “cold” are extracted from the user's utterance, the operation unit of the air conditioner 282 is specified based on the operation correspondence information. When a word similar to “arrival” or “stop” is extracted from the user's utterance, the switch 223 that is the door opening / closing operation unit is specified based on the operation correspondence information. When a word similar to “crowded” or “traffic jam” is extracted from the user's utterance, the switch 223 that is the operation unit for automatic driving is specified based on the operation correspondence information. By voice-recognizing the words included in the user's utterance, the operation unit of the function to be executed by the user can be estimated with high accuracy.

The processor 11 determines the state of the vehicle based on the vehicle detection information.
The processor 11 refers to the “operation correspondence information” in which the vehicle state information regarding the state of the vehicle and the operation unit of the function according to the traveling environment predicted from the vehicle state information are associated in advance, and the processor 11 determines Based on the state, guess the operation of the function that the user is trying to perform.
Since the operation of the function to be executed by the user can be inferred from the state of the vehicle, it is possible to provide appropriate support information.

  The vehicle state information includes the type of vehicle driving mode, the speed of the vehicle, the traffic jam condition of the route on which the vehicle travels, the presence or absence of damage to the vehicle, the presence or absence of vehicle equipment failure or abnormality, the vehicle position, the vehicle interior temperature, At least one of the outside temperature of the vehicle and the operation of the wiper of the vehicle is included.

The “operation correspondence information” shown in FIG. 4 includes vehicle state information.
(1) The processor 11 determines that the vehicle is running at a vehicle speed lower than a predetermined value (low speed) and continues for a predetermined time or more, or the traffic congestion in which a pattern of the vehicle speed change over time is defined in advance. If the matching rate with the vehicle speed pattern at that time is equal to or higher than a predetermined value, it is determined that the traveling environment of the host vehicle is congested, and the function to be executed by the user is assumed to be automatic driving mode switching. In this case, the operation unit supported is a changeover switch for the automatic driving mode.
(2) If the type of the vehicle driving mode is the manual driving mode (non-automatic driving mode and it is determined that the vehicle has traveled at a low vehicle speed less than the predetermined value for a predetermined time or more, It is determined that the traveling environment of the host vehicle is congested and that the function that the user is trying to perform is automatic driving mode switching, and in this case, the operation unit supported is a switching switch for automatic driving mode. Whether the operation mode is the manual operation mode or the automatic operation mode is acquired from the vehicle controller 240.
(3) When the vehicle has a vehicle speed pattern at the time of traffic congestion obtained from traffic information from an external device (for example, VICS (registered trademark): Vehicle Information and Communication System), the processor 11 determines that the traveling environment of the vehicle is traffic congestion. Therefore, it is presumed that the function that the user intends to execute is the automatic operation mode switching. In this case, the operation unit supported is a changeover switch for the automatic driving mode.
(4) When the difference between the vehicle compartment temperature and the outside air temperature is equal to or more than a predetermined value, the processor 11 determines that the temperature control in the vehicle compartment of the host vehicle is necessary, and the function that the user tries to execute is Infer that it is temperature control. The operation unit supported in this case is the operation switch of the temperature control device.
(5) When the shift position is parking, the processor 11 determines that it is necessary to get off the vehicle, and the function to be executed by the user is a seat belt releasing function or a door opening / closing function. I guess. In this case, the operation unit supported is a seat belt release switch or a door operation switch.
(6) When the current position belongs to the area of the parking lot, the processor 11 determines that the host vehicle is in a parking situation, and estimates that the function to be executed by the user is the parking assist function. The operation unit supported in this case is a parking assist operation switch. When the current position belongs to the area of the parking lot, the processor 11 determines that it is necessary to get off the vehicle, and the function to be executed by the user is a seat belt releasing function or a door opening / closing function. I guess. In this case, the operation unit supported is a seat belt release switch or a door operation switch.
(7) When the current position of the host vehicle is the highway, the processor 11 determines that the automatic driving mode is appropriate, and estimates that the function to be executed by the user is the automatic driving function. In this case, the operation unit supported is a changeover switch for the automatic driving mode.
(8) When the processor 11 obtains the execution information of the airbag deployment operation, it determines that a notification to the outside is necessary, and presumes that the function that the user is trying to execute is the emergency notification function. . The operation unit supported in this case is an emergency call switch.
(9) When the processor 11 obtains the failure detection information about the vehicle-mounted device, the processor 11 determines that the maintenance is necessary, and presumes that the function to be executed by the user is the calling function of the operator. The operation unit supported in this case is the operator's call switch.
(10) Note that the functions to be executed by the user may be predicted by combining a plurality of the determination methods described above. For example, the processor 11 (A) the vehicle is in the manual driving mode, (B) the traveling position is an expressway, and (C) the vehicle intermittently travels at a speed lower than a predetermined speed (low speed region) (repeating deceleration operation). , (D) The hand of the user is detected near the switch for changing the automatic driving function of the steering wheel at a predetermined time / predetermined frequency, (E) A keyword related to automatic driving (such as "tired" or "automatic") is detected, When a predetermined number or more of the determination factors (A) to (E) are detected, it is predicted that the operation of the function to be executed by the user is “switching to the automatic operation mode”.

In the present embodiment, an example of a method for estimating the operation of the function that the user intends to execute will be shown.
(1) The camera 226 determines the user's line of sight, motion, and posture at the time of motion, and calculates the target area that the user is trying to operate. The map information including the position information of the operation unit (switch) in the vehicle compartment prepared in advance is referred to, and the operation unit (switch) located in the area common to the target area is narrowed down. When the user's utterance is detected, a word is extracted, and when the extracted word includes a keyword associated with a predefined function, the device associated with the keyword is It is presumed to be the operation unit of the function to be executed. Specifically, when the user's motion of moving his / her hand near the steering wheel is detected and the keyword related to the automatic driving is extracted from the voice information collected by the microphone 222, the user intends to press the switch button to the automatic driving. It is determined to be operating. Further, when the road on which the vehicle is traveling is an expressway, the accuracy of this determination is improved, so the road type of the traveling road may be added to the determination condition.
(2) Based on the position information of the position detection device 271 of the own vehicle and the region of interest (ROI) extracted from the image captured by the camera 226, it is determined that the current position of the own vehicle is a parking lot. If so, it is determined that the user is removing the seat belt with the intention of getting off the vehicle.
(3) When it is determined that the passenger compartment temperature of the host vehicle is outside the predefined comfortable temperature range and the user's line of sight is directed to the operation unit (switch panel) of the air conditioner 282, an operation command If is not input, it is determined that the user is operating the air conditioner 282 with the intention of adjusting the temperature inside the vehicle.
(4) When the user's exit is detected and it is detected that the line of sight of another user present in the passenger compartment is directed to the vicinity of the door, the door opening / closing switch is operated with the intention of the user to exit. It is determined that you are operating. The user's exit information is acquired via the vehicle controller 240 by the seating sensor.

It returns to step S105 of FIG.
When the operation to be supported is determined, the process proceeds to step S106. In step S106, the processor 11 refers to the operation history information. The operation history information 321 is at least temporarily stored in the storage device 320, and includes the operation required time required by the user for an operation command to execute the function. The operation history information 321 may be stored in the storage device 20 of the operation support device 100.

  The operation history information 321 includes the operation required time. The operation required time is the time required until the input of the operation command for executing the function is completed. Specifically, the operation required time is the time from “the timing when the operation of the function is started” to “the timing when the input of the operation command is completed” to the operation unit.

The processor 11 selects, based on the identification result of the user who has issued the operation command, a support mode associated with a relatively short operation required time among the operation required times of the identified users. From the operation history information of the user, the support mode associated with the shortest operation required time is selected.
Based on the identification result of the user who has issued the operation command and the estimation result of the operation of the vehicle-mounted device that the user intends to execute, the processor 11 selects, from among the operation required times stored for each function, the identified user. A support mode associated with a relatively short required operation time is selected. Of the operation history information for each function of the user, the support mode associated with the shortest required operation time measured when operating that function is selected.
The selected support mode is determined as the support mode when outputting the support information.

  FIG. 5 shows an example of the operation history information 321. The operation history information 321 includes an operation required time and a support mode of support information. The operation history information 321 includes the identification information of the user, the identification information of the operation unit to be operated, the recorded date and time, the vehicle information, and the time and effort level (evaluation) according to the length of the operation required time. Any one or more of are included. The vehicle information includes the type of vehicle, the in-vehicle function, and the like. The information is that the vehicle is a one-box car, the vehicle is a sliding door, the vehicle has an automatic driving function, and the vehicle has an automatic parking function. The support mode is an output mode when the support information is presented. The operation history information 321 is sorted for each user. The time-consuming operation tends to be caused by the habit of the user. For this reason, it is preferable to organize by user and refer to the operation history information 321 in which the operation targets of the same user are common. When the operation history information 321 of the same user does not exist, it is preferable to refer to the operation history information 321 having common vehicle information (vehicle type).

In step S107, the processor 11 determines a support mode for supporting the estimated operation based on the operation required time of the operation history information 321.
The processor 11 adopts the support mode of the operation history information 321 in which the required operation time is short as the support mode of the current support information. The processor 11 adopts a support mode in which the effort level is highly evaluated according to the length of the operation required time as the support mode of the current support information.

  Although not particularly limited, it is preferable to refer to the common user operation history information 321. Although not particularly limited, it is preferable to refer to the operation history information of the operation target common to the common user and the estimated operation.

A method of determining the support mode when referring to the operation history information 321 shown in FIG. 5 will be described.
The processor 11 refers to the operation history information 321 when the “user A” operates the “automatic operation mode switching” when it is estimated that the function to be executed by the user A is the “automatic operation function”. To do. The processor 11 extracts a history with a shortest operation required time and a high evaluation among them, and determines blinking light emission using the light emitting element 225 such as an LED, which is the assistance mode in the history, as the current assistance mode.

The support mode includes at least the type of the output device and the type of the information transmission medium. The transmission medium includes at least sound, images, and light emission.
As the input / output device 220 for outputting the support information, a voice output device including a speaker 224, an image output device including a display 221, and a light emitting device including a light emitting element 225 such as an LED can be used. The support information output using the speaker 224 as a voice output device is voice information, and the support mode is a voice output mode. The support information output via the display 221 as an image output device is an image (including a figure and a text), and the support mode is a display output mode. The support information output through the light emitting element 225 as the light emitting device is visible light emission information, and the support mode is the light emission mode. The light emitting device including the light emitting element 225 may be incorporated in the operation portion of the switch so that the location of the switch can be indicated by light emission.

The voice output mode provides assistance information by voice so as to speak to the user. This is a support mode effective when the user cannot look at the guidance screen of the device. This is effective when driving and paying particular attention to the surroundings. It is also possible to exchange information bidirectionally and request a detailed explanation from the user, or instruct to repeat the explanation.
The display output mode outputs support information using a figure, a text, and a moving image. Suitable for detailed explanation when the operation is complicated. It is also possible to exchange information bidirectionally by using an operation using a GUI (Graphical User Interface) technology including buttons, check boxes, and slide bars to request a detailed explanation from the user, or to instruct repetition of the explanation. .
The light emission mode indicates the position of the operation unit (switch) by illuminating the operation unit itself. Although it is not suitable for explaining detailed instructions and operating procedures, it is possible to directly point to the operating unit. This is especially effective when the operation target cannot be found due to darkness at night.
These support modes can be switched appropriately. The support information may be presented by combining a plurality of support modes. In the present embodiment, main support information is defined in advance, and the support mode of the main support information is determined. It is also possible to use other support modes for the sub support information that complements the main support information.
The support mode may be changed over time. In this embodiment, the support information to be output first is defined in advance, and the support mode of the support information to be output first is determined. It is also possible to use other support modes for the complementary support information output thereafter.
For example, when it is determined that the main support information to be output first is output in a display output mode or a light emission mode as a visual notification / display, and a user operation command is not input (operation is not completed). Alternatively, the support information may be output using a voice output mode. The support mode may be switched in consideration of the effectiveness of the support mode.

  Since the support information can be output by at least one of audio information that can be heard by humans, display (image) information that can be visually recognized by humans, and light emission information that can be perceived by humans, the support information can be output in a support mode suitable for the user. Can be provided.

The processor 11 refers to the operation history information 321 when the “user A” operates the “door operation” when the function to be executed by the user A is estimated to be the “door operation”. The processor 11 extracts a history with the shortest required operation time and a high evaluation among them, and determines the presentation of the switch position by the image information using the display, which is the assistance mode in the history, as the current assistance mode.
When it is estimated that the function to be executed by the user A is “air conditioning operation”, the processor 11 refers to the operation history information 321 when the “user A” operates “air conditioning operation”. The processor 11 extracts a history with the shortest required operation time and a high evaluation among them, and determines the presentation of the assistance information by voice using a speaker, which is the assistance aspect in the history, as the assistance aspect this time.

  Since the function to be executed by the user is estimated, it is possible to provide effective operation support to the user even when the operation target is not specified. Since the support mode is determined based on the estimated function with reference to the operation history information 321, the support information can be provided in the support mode suitable for the user.

The required operation time is the time from the "timing at which the operation of the function is started" to the "timing at which the input of the operation command is completed" for the operation unit. Therefore, if the required operation time is short, assistance at that time will be provided. If the support information is provided in the form, it can be evaluated that the user does not have to take the trouble. By determining the support mode based on the operation history information including the operation required time stored for each user, the support information can be provided in the support mode effective for the user.
Further, by determining the support mode based on the operation history information including the operation required time stored for each user and function, the support information is provided in the effective support mode for the function predicted to be executed by the user. be able to.

In this embodiment, the operation history information 321 is defined from different viewpoints.
The operation history information 321 includes the operation evaluation time. The operation evaluation time is a value for evaluating the effectiveness (degree of support effect) of the support information output in the past. The operation evaluation time is the time from “completion of output of support information” to “completion of input of operation command for executing a function”.
The shorter operation evaluation time means that the support mode is suitable for the user.
Based on the identification result of the user who has issued the operation command, the processor 11 preferentially selects the support mode associated with the operation evaluation time of the identified user, which is associated with the relatively short operation evaluation time. , Set the weighting.
In addition, the processor 11 compares the operation evaluation time stored for each function of the identified user based on the identification result of the user and the estimation result of the operation of the vehicle-mounted device that the user intends to execute. The weighting is set so that the support mode associated with the short operation evaluation time is preferentially selected.
For example, when the difference between the operation required times associated with the operations of different functions in the plurality of accumulated data is within a predetermined value, the operation evaluation time is further referred to (the evaluation time is short). The weighting is set so that the support mode is preferentially selected. Specifically, a positive coefficient of less than 1 is added so that the operation required time associated with the support mode having a short operation evaluation time or a high evaluation is shortened, and the operation required time is calculated to be short. As a result, the required operation time is shortened, and the operation mode is preferentially selected.
In addition, in the plurality of accumulated data, when the difference between the operation required times associated with the operations of different functions is within a predetermined value, the evaluation of the operation evaluation time is further referred to, and the reduction degree of the operation evaluation time is reduced. The weighting is set so that a support mode that is large (the ratio of the current operation required time to the previous operation required time is small) is preferentially selected. Specifically, a positive coefficient of less than 1 is attached so that the required operation time associated with the support mode having a large degree of reduction in the operation evaluation time is shortened, and the required operation time is calculated to be short. As a result, the required operation time is shortened, and the operation mode is preferentially selected.
The selected support mode is determined as a support mode for outputting support information.

  FIG. 6 shows an example of the operation history information 321. The operation history information 321 includes an operation evaluation time and a support mode of support information. The operation history information 321 includes the identification information of the user, the identification information of the operation unit to be operated, the support mode when the support information is presented, the operation evaluation time, the vehicle information, and the length of the operation evaluation time. It includes any one or more of the floor plan level (evaluation) according to the above and the recorded date and time. In FIG. 6, the operation required time is not shown in order to mainly describe the example including the operation evaluation time, but the operation history information 321 includes the operation required time.

  The evaluation is an evaluation determined according to the length of the operation evaluation time from “completion of output of support information” to “completion of input of operation command for executing function”. When the operation evaluation time is long (for example, 20 seconds or more), a low evaluation is given (Z evaluation). Although the support information is output, it takes a long time to complete the input of the operation command, and it is evaluated that the response of the user is slow. When the operation evaluation time is short (for example, less than 10 seconds), a high evaluation is given (X evaluation). The output of the support information shortens the time required to complete the input of the operation command, and the response of the user is evaluated to be quick. When the operation evaluation time is between the X evaluation and the Z evaluation (for example, 10 seconds or more and less than 20 seconds), a standard evaluation is given (Y evaluation).

  Further, regarding the evaluation of a certain support mode, when the evaluation with the new recorded date and time is higher than the evaluation with the recorded date and time is older, the support mode is a support mode in which the user can easily learn. You can judge. A change in the operation evaluation time over time may be calculated for each support mode, and the calculation result may be reflected in the evaluation. For example, when the operation evaluation time decreases with time, the evaluation may be calculated based on the operation evaluation time weighted so that the evaluation becomes higher. The processor 11 determines the support mode based on the operation required time in consideration of these evaluations. When the evaluation result of the operation evaluation time is good, weighting may be performed so that the operation required time becomes short. As a result, the operation required time can be corrected based on the fact that the operation evaluation time is short or has been shortened over time, and the support mode can be determined based on the corrected operation required time.

  The operation history information 321 can be organized for each user as shown in FIG. In addition, it is possible to sort by operation target. The operation history information 321 has a high tendency that the troublesome operation is caused by the habit of the user. For this reason, it is preferable to organize by user and refer to the operation history information 321 in which the operation targets of the same user are common. When the operation history information 321 of the same user does not exist, it is preferable to refer to the operation history information 321 having common vehicle information (vehicle type).

  The processor 11 sets the weighting so that the support mode in the operation history information 321 having a short operation evaluation time is preferentially selected. The processor 11 sets the weighting so that the support mode in which the effort level is highly evaluated (the effort time is short) according to the length of the operation evaluation time is preferentially selected. Although not particularly limited, the weighting is preferably set by referring to the common user operation history information 321. Although not particularly limited, it is preferable that the weighting be set with reference to operation history information of an operation target common to the common user and the estimated operation.

The processor 11 refers to the operation history information 321 when the “user A” operates the “automatic operation mode switching” when it is estimated that the function to be executed by the user A is the “automatic operation function”. To do. The processor 11 extracts the history with the shortest operation evaluation time and the highest evaluation, and the blinking light emission, which is the support mode in the history, is changed to the flashing light emission so as to be preferentially selected as the current support mode. Weight.
The processor 11 refers to the operation history information 321 when the “user A” operates the “door operation” when the function to be executed by the user A is estimated to be the “door operation”. The processor 11 extracts the history with the shortest operation evaluation time and the highest evaluation, and the presentation of the support information by the image information using the display, which is the support mode in the history, is given priority as the support mode of this time. The display output mode is weighted so as to be selected.
When it is estimated that the function to be executed by the user A is “air conditioning operation”, the processor 11 refers to the operation history information 321 when the “user A” operates “air conditioning operation”. The processor 11 extracts the history with the shortest operation evaluation time and the highest evaluation, and presents the support information by the voice using the speaker or the light emission using the LED, which is the support mode in the history. The voice output mode or the light emission mode is weighted so that the support mode is preferentially selected.

Furthermore, weighting may be performed according to changes in the operation evaluation time.
In the operation history information 321 shown in FIG. 6, regarding the air conditioning operation when the user A uses the vehicle of type V1, the shortest operation evaluation time when the speaker is used and when the LED is used is the same 5 seconds. In such a case, the effect of the support mode will be further examined. Referring to the operation history information 321, when the voice guidance using the speaker is performed, the operation evaluation time recorded at the previous date and time T12 is 5 seconds, and the operation evaluation time recorded at the later date and time T22 is also. 5 seconds. Further, when the guide is performed by blinking light emission using the LED, the operation evaluation time recorded at the previous date and time T11 is 10 seconds, and the operation evaluation time recorded at the later date and time T212 is shortened to 5 seconds. Has been done. When the operation evaluation time is shortened in this way, it can be inferred that the support mode is effective (effective) for the user.

  In the present embodiment, even if the latest operation evaluation time is the same 5 seconds, the operation evaluation time of support by blinking light emission using the LED whose operation evaluation time is shortened to 50% is further preferentially selected. The weighting is set as described above, and the weighted operation evaluation time is calculated. As a result, the operation evaluation time of support by blinking light emission is less than 5 seconds, and the shortest operation evaluation time is No. It becomes 9 data. The support aspect in this data is the LED. In the processing this time, when it is estimated that the function to be executed by the user A is “air conditioning operation”, the processor 11 preferentially selects blinking light emission as a support mode instead of voice. , Weighting larger than that of the voice output mode (easy to be selected) is set.

  Since the operation support device 100 automatically estimates the function to be executed by the user, it is possible to provide effective operation support for the user even when the operation target is not specified. Since the support mode is determined based on the estimated function with reference to the operation history information 321, the support information can be provided in the support mode suitable for the user.

The operation evaluation time is the time from “completion of output of support information” to “completion of input of operation command for executing a function”. Therefore, if the operation evaluation time is short, the time and effort required for the user will be short due to the output of support information. It can be evaluated that it has become. By determining the support mode based on the operation history information including the operation evaluation time stored for each user, the support information can be provided in a support mode effective for the user.
If there is a tendency that the current operation evaluation time is shorter than the previous operation evaluation time, then the operation mode is considered to be suitable for the user. In the support mode in which the operation evaluation time tends to be shortened over time, it can be expected that the time and effort required for the user will be shortened by outputting the support information. By determining the support mode based on the operation history information including the change over time of the operation evaluation time stored for each user, the support information can be provided in a support mode effective for the user.
Further, by determining the support mode based on the operation history information including the operation evaluation time stored for each user and function, the support information is provided in the effective support mode for the function predicted to be executed by the user. be able to.

After the support mode is determined, the processor 11 performs a support mode change process based on the user's motion information and / or vehicle state information.
As described above, the processor 11 of the present embodiment determines the support mode based on the operation required time or the operation evaluation time.
However, in a scene where the user wants to concentrate on the surrounding situation monitoring for driving or autonomous driving monitoring, a voice output mode that allows the user to listen rather than providing support information in a visually supported support mode such as a display output mode or a light emission mode May be more effective.
In addition, when the user other than the driving administrator is sleeping, the passengers are having a conversation with each other, or the music is being listened to at a high volume, the support information is provided in a voice output mode. In some cases, it may be preferable for the user to provide the support information in a display support mode or a light emission mode for visually supporting.
In such a case, the display mode once determined is changed.
According to this processing, even after the support mode is determined, by performing the support mode change process based on the user's motion information and / or vehicle state information, the support mode can be easily understood by the user. Support information can be provided.

Hereinafter, a specific processing method for changing the support mode will be described.
First, a process of changing from a visually recognized support mode such as a display output mode or a light emission mode to a voice output mode based on voice information will be described.
The support mode determined based on the operation required time or the operation evaluation time is the display support mode or the light emission mode, and during the output process of the support information, the vehicle state information of the own vehicle is the predefined surrounding monitoring degree. When it is determined that the state is high, the display support mode or the light emission mode by visual recognition is changed to the voice support mode by listening. The surrounding monitoring degree is a degree according to the degree of caution that the user monitors the surroundings of the own vehicle. When the surroundings are carefully monitored relatively, the degree of surroundings monitoring is set to be relatively high. For example, this may be the case when an obstacle is present at the time of traveling at an intersection or the vehicle speed is high.
In a state where the degree of surroundings monitoring is high, the user concentrates on driving and surroundings monitoring, and therefore, even if the supporting information is presented in the display supporting mode or the light emitting mode, it is highly likely that the user does not notice. Also, even if the location of the switch is indicated by light emission, the location may not be understood. In such a case, the support information that the user hears is easier for the user to understand than the support information that the user visually recognizes.
The state where the surrounding monitoring degree is high means that the position of the own vehicle is within a predetermined distance from the intersection, the width of the road on which the own vehicle is traveling is less than a predetermined value, the distance between the own vehicle and obstacles, and the required arrival. A state in which the time is less than a predetermined value is included. The road width of the traveling road is acquired from the map information 272 of the navigation device 270.

For example, the support mode determined based on the operation required time or the operation evaluation time is a display support mode or a light emission mode, and at the time of output processing of the support information, the vehicle state information of the own vehicle is a predefined surrounding monitor. When it is determined that the frequency is high, the display support mode or the light emission mode by visual recognition is changed to the voice support mode by listening.
Note that this process may be executed when the operation command is not input for a predetermined time after the support information is output in the support mode determined based on the operation required time or the operation evaluation time. This is because it can be determined that the user cannot use the support information.

Next, a description will be given of processing in the case of changing from the audio output mode such as audio information to be heard to the visual support mode such as the display output mode and the light emission mode.
The support mode determined based on the operation required time or the operation evaluation time is a voice output mode,
When it is determined that the user (an occupant other than the driver or the driving manager) boarding the vehicle is sleeping during the output process of the support information, it is determined that the occupants of the vehicle are talking to each other. If it is performed, the voice support mode is changed to the visual display support mode or the light emission mode.
When the occupant is sleeping, it is preferable that the passenger compartment be quiet. Further, when the passengers are having a conversation with each other, it is preferable that the passenger compartment be quiet so as not to disturb the conversation. Further, when the volume of the audio device 281 is equal to or higher than the predetermined value, the support information may not be heard.
In such a case, the support information visually recognized by the user is easier for the user to understand than the support information to be heard by the user.
Whether or not the user is sleeping is determined based on the image captured by the camera 226 inside the vehicle, by identifying the face, identifying the position of the eyes, and determining that the user is sleeping by the absence of black eyes. Can be detected. The biometric information of each user may be acquired over time and compared with the biometric information pattern during sleep to determine whether or not the user is sleeping. The user is not limited to the user sitting in the driver's seat. The camera 226 is provided on the ceiling of the passenger compartment and can capture a wide-angle captured image. Regarding the face recognition technology, a method known at the time of application can be used.
Whether or not the users are having a conversation can be detected based on the voice information collected by the microphone 222 inside the vehicle, and the features of the voice can be extracted to detect that the different users are having a conversation.
The volume of music is acquired from the audio device 281.

For example, the support mode determined based on the operation required time or the operation evaluation time is a voice output mode, and during the output process of the support information, the occupants are sleeping, the users are in conversation, or the audio is output. When it is determined that the output sound volume of is equal to or higher than the predetermined value, the audio support mode by listening is changed to the display support mode or light emission mode by visual recognition.
Note that this process may be executed when the operation command is not input for a predetermined time after the support information is output in the support mode determined based on the operation required time or the operation evaluation time. This is because it can be determined that the user cannot use the support information.

  It returns to step S107 of FIG. As described above, when the support mode for outputting the support information is determined, the process proceeds to step S108. In step S108, the processor 11 generates a support information output instruction. The support information output command includes a command for designating the support mode determined in step S107.

The support mode of the support information will be described.
FIG. 7 shows an operation unit around the steering wheel ST of the vehicle. The switch SW1 is a switch for an automatic driving function. The switch SW1a is the main switch for the automatic driving function. The switch SW2 is a switch for navigation function. The switch SW3 is a switch for opening and closing the slide door. The switch SW4 is a switch for a driving support function. The switch SW5 is an air conditioning function switch. Hereinafter, an output example of the support information according to each support mode will be described.

  When the display output mode is determined as the support mode, the processor 11 outputs the support information of the image information (including the text information) via the display 221. The display 221 presents the support information G of the image information. The support information is image information indicating the position / operation of the switch SW to be operated, and image information and / or text information indicating an operation procedure. These are sequentially displayed according to the procedure. Specific support information is shown in FIGS. 8A to 10.

  8A and 8B show a first example of the support information G. The support information G1a illustrated in FIG. 8A is image information indicating the position of the switch SW1 which is the operation unit of the automatic driving function. The support information G1a includes support information G11 displayed in a balloon. The support information G11 enlarges and displays the switch SW1 which is the operation unit, and emphasizes the main switch SW1a predicted to be currently operated by the user (indicated by a thin black mark in the drawing). The support information G1b illustrated in FIG. 8B is image information that provides guidance on the operating procedure of the automatic driving function. The support information G1b displays an icon (image information) imitating a vehicle and text information "Please press the Start button" to prompt the user to press the start button for the automatic driving function that the user intends to execute. To do. The contents of the image and text of the operation support are defined appropriately.

  9A and 9B show a second example of the support information G. The support information G2A shown in FIG. 9A and the support information G2B shown in FIG. 9B are presented on the display 221 shown in FIG. The support information G2A and G2B shown in FIGS. 9A and 9B are image information indicating the position of the switch SW3 which is the operation unit of the opening / closing function of the slide door. The figure shown by a balloon is an enlarged view of the opening / closing switch SW3 of the left and right doors of the vehicle. 9A emphasizes and displays the opening / closing switch SW3R of the right side door of the vehicle, and FIG. 9B emphasizes and displays the opening / closing switch SW3L of the left side door of the vehicle. Which of the support information in FIG. 9A or FIG. 9B is to be displayed on the display 221 may be determined based on the positional relationship between the road and the vehicle, the support information regarding the opening / closing switch SW3R / L of the roadside door. .

FIG. 10 shows a third example of the support information G. The support information G3 shown in FIG. 10 is presented on the display 221 shown in FIG. FIG. 10 is image information showing the position of the switch SW7 which is the operation unit for the opening / closing function of the slide door provided in the rear seat. The figure shown by a balloon is an enlarged view of the switch SW7 relating to the opening / closing function of the door on the side where the switch is provided.
The users in this embodiment include a driver and an occupant. It is also possible for a user located in the rear seat to guess the function to be executed by the user and assist the input of an operation command to execute the function.

Subsequently, a storage process of the operation history information 321 will be described. First, the operation history information 321 including the operation required time will be described.
The processor 11 associates the identification information of the user, the estimation result of the operation of the function to be executed by the user, and the operation time required until the input of the operation command for executing the function by the user is completed. The operation history information 321 is transmitted to the storage device 320 via the communication network and stored in the storage device 320. The operation history information 321 may be stored in the storage device 20 of the operation support device 100. As described above, since the support mode can be determined by referring to the operation history information 321 in which the user identification information, the operation required time, and the operation content are associated with each other, the support information can be output in a mode suitable for the user who receives the support. .

  The output example of the assistance information of a voice output mode is shown. As an example, an output example in the case of providing support information in a voice output mode in the case of “turning on the automatic driving function” in a scene where the function of switching the automatic driving mode is executed will be shown. The assistance information guides the operation procedure for the operation unit to be operated in the automatic driving mode switching by voice for each action. When the processor 11 deduces that the user desires to turn on the automatic driving function, "I will explain the operating procedure." "Operation 1, the set switch of the pro-pilot installed on the steering wheel. "Turn it on.", "Operation 2, when the ProPilot function enters the standby state, turn on the ProPilot set switch.", "Operation 3, check the start of the ProPilot function." Output a voice announcement such as. A voice content of support information for supporting the operation is stored in association with each function. When the support mode of the support information for the operation of the estimated function is determined to be the voice output mode, the voice support information associated with the estimated function is read and the voice is output via the speaker 224. .

FIG. 11 is a flowchart showing the control procedure of the learning process of accumulating the operation history information 321.
In step S1 of FIG. 11, the processor 11 identifies the user. The specific processing content is the same as step S101 in FIG.

  In step S2, the processor 11 acquires the motion information of the user, and determines, based on the motion information based on the motion of the user, that the user is in an operation search state in which the operation is time-consuming. The specific processing content is the same as step S103 in FIG.

When it is determined that the user is in the operation search state, the process proceeds to step S3.
In step S3, the timing is started with the timing when the user determines that the operation is in the search state as the starting point. The timing at which the user is determined to be in the operation search state is pretended to be "timing at which the operation of the function is started". In step S4, it is determined whether or not the input of the operation instruction by the user is completed. When the input information regarding the predicted function is obtained, it is determined that the input of the operation command is completed. In step S5, the timing at which the input information is obtained is set as the end point, and the timing is ended. In step S6, the processor 11 calculates the operation required time. The operation required time is from the start point of the time counting start in step S3 to the end point of the time counting end in step S5. With this, the processor 11 acquires the operation required time from the “timing when the operation of the function is started” to the “timing when the input of the operation command is completed” to the operation unit. In step S7, the processor 11 sends to the server 300 operation history information in which the user identification information, the operation target, the support mode, and the operation required time are associated with each other. At the same time, the vehicle type as the vehicle information and the determined time-saving level may be transmitted. The support mode may include the type of the output device and the type of the transmission medium.

In step S8, the server 300 stores the received information in the operation history information 321 in the storage device 320.
FIG. 12 is an example of the operation history information accumulated in the storage device 320. The operation history information 321 may be sorted by user or by function. In step S9, the server 300 updates the operation history information 321. In the present embodiment, the processing from step S8 is executed by the control device 310 of the server 300. The operation support device 100 may transmit the operation history information including the operation required time to the server 300 by including the write command to the storage device 320.

Subsequently, the accumulation processing of the operation history information 321 including the operation evaluation time will be described.
The processor 11 identifies the user, the function executed in response to the operation command, the support mode when the support information is output, and the completion of the output of the support information until the input of the operation command for executing the function is completed. The operation history information including the operation evaluation time of is transmitted to the storage device 320 via the communication network and stored in the storage device 320. The operation history information 321 may be stored in the storage device 20 of the operation support device 100. Since the support mode can be determined by referring to the operation history information 321 including the operation evaluation time and the support mode thus accumulated, the support information can be output in a mode suitable for the user who receives the support.

FIG. 13 is a flowchart showing the control procedure of the learning process of accumulating the operation history information 321 including the operation evaluation time.
In step S11 of FIG. 13, the processor 11 identifies the user. The specific processing content is the same as step S101 in FIG.

  In step S12, the processor 11 acquires the user's motion information, and determines, based on the motion information based on the user's movement, that the user is in the operation search state in which the operation is troublesome. The specific processing content is the same as step S103 in FIG.

When it is determined that the user is in the operation search state, the process proceeds to step S13.
When it is determined in step S12 that the user is in the operation search state, and when the predetermined time has elapsed in step S13, the process proceeds to step S14. If not, the process returns to step S11. In step S14, the processor 11 acquires the target of the function with which the user inputs an operation command. In step S15, the processor 11 acquires a support mode. The support mode is the mode determined in step S107 of FIG. In step S16, the processor 11 generates support information. The support information is support information related to the output command generated in step S108 of FIG. In step S17, the processor 11 outputs assistance information.

  In step S18, the processor 11 starts timing with the output start timing of the support information as a starting point. In step S19, the processor 11 determines whether or not the input of the operation command is completed. When the input information regarding the predicted function is obtained, it is determined that the input of the operation command is completed. In step S20, the timing at which the input information is obtained is set as the end point, and the timing is ended. In the same step S20, the processor 11 calculates the operation evaluation time. The operation evaluation time is from the start point of time counting start in step S18 to the end point of time counting end in step S19. With this, the processor 11 obtains the operation required time from “completion of output of support information” to “timing of completion of input of operation command” to the operation unit. In step S21, the processor 11 sends to the server 300 the operation history information 321 in which the user identification information, the operation target, the support mode, and the operation evaluation time are associated with each other. At the same time, the vehicle type as the vehicle information and the determined time-saving level may be transmitted. The support mode may include the type of the output device and the type of the transmission medium. The operation evaluation time may include support start timing (support information output start timing) and operation end timing (operation command input timing).

In step S22, the server 300 stores the received information in the operation history information 321 in the storage device 320.
FIG. 14 is an example of the operation history information accumulated in the storage device 320. The operation history information 321 may be sorted by user or by function. In step S23, the server 300 updates the operation history information 321. In the present embodiment, the processing from step S8 is executed by the control device 310 of the server 300. The operation support device 100 may transmit the operation history information including the operation required time to the server 300 by including the write command to the storage device 320.

  Since the operation support device 100 according to the embodiment of the present invention is configured and operates as described above, it has the following effects.

[1] The operation support method of the present embodiment estimates the operation of a function to be executed based on the acquired user's operation information, and stores the operation at least temporarily in the storage device 320 or the storage device 20 to execute the function. In order to allow the user to refer to the operation history information including the operation required time required for the operation command, based on the operation required time for each user, determine the support mode of the support information for supporting the estimated operation, the support mode To output support information.
According to the operation support method of the present embodiment, since the function to be executed by the user is estimated, it is possible to provide appropriate operation support even when the operation target is not specified. Further, since the support mode is determined based on the estimated function by referring to the operation history information 321, the support information can be provided in the support mode suitable for the user. As a result, support information can be provided in a support mode effective for the user.

[2] In the operation support method of the present embodiment, the support modes are (1) a voice output mode in which support information is output by voice using a voice output device such as a speaker 224, and (2) an image output device such as a display 221. One of the display output mode in which the support information is presented by using (3) and the light emission mode in which the support information is output by using the light emitting device (3) is determined as the support mode.
The support information output using the speaker 224 as a voice output device is voice information, and the support mode is a voice output mode. The support information output via the display 221 as an image output device is an image (including a figure and a text), and the support mode is a display output mode. The support information output through the light emitting element 225 as the light emitting device is visible light emission information, and the support mode is the light emission mode.
According to the operation support method of the present embodiment, the support information can be output by any one or more of audio information that can be heard by humans, display (image) information that can be visually recognized by humans, and light emission information that can be perceived by humans. The support information can be provided in a support mode suitable for the user.

[3] The operation support method of the present embodiment refers to operation history information including an operation required time required until the input of an operation command for executing a function, which is associated with the identified user, is referred to, and the operation is performed. Based on the identification result of the user who issued the command, among the operation required times of the identified users, the support mode associated with the relatively short operation required time is preferentially selected, and the support information is provided in the support mode. Output.
The required operation time is the time from the "timing at which the operation of the function is started" to the "timing at which the input of the operation command is completed" for the operation unit. Therefore, if the required operation time is short, assistance at that time will be provided. If the support information is provided in the form, it can be evaluated that the user does not have to take the trouble.
According to the operation support method of the present embodiment, by determining the support mode based on the operation history information including the operation required time stored for each user, it is possible to provide the support information in a support mode effective for the user. it can.
Further, by determining the support mode based on the operation history information including the operation required time stored for each user and function, the support information is provided in the effective support mode for the function predicted to be executed by the user. be able to.
Since the support information is provided in the support mode in which the operation required time is short, the operation required time can be shortened.

[4] The operation support method of the present embodiment provides operation history information including an operation required time required until the input of an operation command for executing a function, which is associated with the function executed by the identified user, is completed. Based on the identification result of the user who issued the operation instruction and the estimation result of the operation of the in-vehicle device which the user intends to execute, the relative operation time among the operation required times stored for each function of the identified user The support mode associated with the relatively short operation required time is preferentially selected, and the support information is output in the selected support mode.
According to the operation support method of the present embodiment, by determining the support mode based on the operation history information including the operation required time stored for each user and each function, it is effective for the function predicted to be executed by the user. Support information can be provided in various support modes.

[5] The operation support method of the present embodiment refers to the operation history information including the operation evaluation time, which is associated with the identified user, and identifies the identified user based on the identification result of the user who has issued the operation command. Of the operation evaluation times of 1, the support mode associated with the relatively short operation evaluation time is preferentially selected, and the support information is output in the support mode.
The operation evaluation time is the time from “completion of output of support information” to “completion of input of operation command for executing a function”. Therefore, if the operation evaluation time is short, the time and effort required for the user will be short due to the output of support information. It can be evaluated that it has become.
According to the operation support method of the present embodiment, by determining the support mode based on the operation history information including the operation evaluation time stored for each user, it is possible to provide the support information in a support mode effective for the user. it can.

[6] The operation support method of the present embodiment refers to the operation history information including the operation evaluation time, which is associated with the function performed by the identified user, and identifies the user who has issued the operation command and the operation history information. Based on the estimation result of the operation of the vehicle-mounted device to be executed, of the operation evaluation times stored for each function of the identified user, the support mode associated with the relatively short operation evaluation time is prioritized. To select and output the support information in the support mode.
According to the operation support method of the present embodiment, by determining the support mode based on the operation history information including the operation evaluation time stored for each user and function, it is effective for the function predicted to be executed by the user. Support information can be provided in various support modes.

[7] In the operation support method of the present embodiment, the elapsed time from the timing at which the user who issued the operation command is identified and the operation of the function is determined to be started is equal to or longer than the predetermined time set based on the operation required time. When it becomes, the support mode is selected and the support information is output according to the support mode.
The "predetermined time" as a threshold for determining that the user is taking a lot of time is based on the operation required time from the start of the operation of the function to the input of the operation command based on the past operation history information. To set.
According to the operation support method of the present embodiment, it is possible to determine with high accuracy that a user has taken time by setting a threshold value for determining whether or not the user has taken time, based on the time actually taken by the user.
Even if the elapsed time from the timing when it is determined that the operation of the function is started exceeds the predetermined time set based on the operation required time, the support information can be provided when the operation command is not input. Assistance information can be provided during the actual hassle.

[8] The operation support method according to the present embodiment refers to operation correspondence information in which the detected position of the motion information and the position of the operation unit of the vehicle-mounted device are referred to in advance, and the user's motion extracted from the motion information is displayed. The user infers the operation of the function to be executed based on the position of the target operation unit.
According to the operation support method of the present embodiment, it is possible to infer the operation unit of the function to be executed by the user from the position of the target of the user's action, and thus it is possible to provide appropriate support information.
Since it is possible to infer the operation of the function that the user is going to perform based on the operation information, determine the timing when the user cannot input the operation command and is troublesome even if the user has not actually completed the input. However, the user can guess the function to be performed.

[9] The operation support method of the present embodiment refers to operation correspondence information in which an operation unit of a function predicted from a user's motion and motion information are associated in advance, and the user tries to execute the operation based on the motion information. Guess the operation of the function to be performed.
According to the operation support method of the present embodiment, the operation unit of the function to be executed by the user can be inferred from the operation information of the user, so that appropriate support information can be provided.

[10] The operation support method of the present embodiment is intended to be executed by the user on the basis of the user's motion information including the information about the movement of the user's hand or head or the direction of the line of sight extracted from the captured image. Guess the function to do.
According to the operation support method of this embodiment, by analyzing the movements of the user's hand and head, the operation unit of the function to be executed by the user can be estimated with high accuracy. By analyzing the direction of the user's line of sight, it is possible to estimate with high accuracy the operation unit of the function that the user intends to execute.

[11] In the operation support method of the present embodiment, the utterance information of the user extracted from the collected voice information is included in the utterance information of the user.
According to the operation support method of the present embodiment, by recognizing the words included in the utterance of the user by voice, the operation unit of the function to be executed by the user can be estimated with high accuracy.

[12] The operation support method according to the present embodiment determines a vehicle state from vehicle detection information, vehicle state information regarding the vehicle state, and an operation unit having a function according to a traveling environment predicted from the vehicle state information. Refers to the operation correspondence information associated in advance, and estimates the operation unit of the function to be executed by the user based on the determined vehicle state.
According to the operation support method of the present embodiment, it is possible to estimate the operation of the function to be executed by the user from the state of the vehicle, and thus it is possible to provide appropriate support information.

[13] In the operation support method of the present embodiment, the vehicle state information includes the type of vehicle driving mode, the speed of the vehicle, the traffic jam condition of the route on which the vehicle travels, the presence / absence of damage to the vehicle, and the equipment installed in the vehicle. It includes any one or more of presence or absence of failure or abnormality, vehicle position, vehicle interior temperature, vehicle outside air temperature, and vehicle wiper operation.
According to the operation support method of the present embodiment, it is possible to infer the function to be executed by the user from various vehicle state information.

[14] In the operation support method of the present embodiment, after the support mode is determined, the support mode change process is performed based on the user's motion information and / or vehicle state information. Even after the support mode is determined based on the operation required time or the operation evaluation time, the user understands by performing the support mode change process based on the user's motion information and / or vehicle state information. The support information can be provided in an easy support mode.

[15] In the operation support method of the present embodiment, the identification information of the user, the estimation result of the operation of the function to be executed by the user, and the operation required until the input of the operation command for executing the function by the user is completed. The operation history information 321 associated with the required time is transmitted via the communication network and stored in the storage device 320 of the server 300 or the storage device 20 of the operation support device 100.
According to the operation support method of the present embodiment, the support mode can be determined with reference to the accumulated operation history information 321, so that the support information can be output in a mode suitable for the user who receives the support.

[16] The operation support method according to the present embodiment causes the function to be executed after the identification information of the user, the function executed according to the operation command, the support mode when the support information is output, and the completion of the output of the support information. The operation history information including the operation evaluation time until the input of the operation command is transmitted via the communication network and stored in the storage device 320 of the server 300 or the storage device 20 of the operation support device 100.
According to the operation support method of the present embodiment, since the support mode can be determined by referring to the operation history information 321 including the accumulated operation evaluation time and the support mode, the support information is in a mode suitable for the user who receives the support. Can be output.

[17] The operation support device 100 of this embodiment has the same operation and effect as the operation support method described above.

  Note that the embodiments described above are described to facilitate understanding of the present invention, and are not described to limit the present invention. Therefore, each element disclosed in the above-described embodiment is intended to include all design changes and equivalents within the technical scope of the present invention.

1 ... Operation support system 100 ... Operation support device 10 ... Control device 11 ... Processor 20 ... Storage device 321 ... Operation history information 30 ... Output device 200 ... In-vehicle device, vehicle-mounted device 210 ... Communication device 220 ... Input / output device 221 ... Display 222 ... Microphone 223 ... Switch 224 ... Speaker 225 ... Light emitting element 226 ... Camera 230 ... Detection device 231 ... Image processing device 232 ... Voice processing device 233 ... Door sensor 234 ... Seating sensor 235 ... Thermometer 236 ... Position sensor 237 ... Shift position Sensor 240 ... Vehicle controller 241 ... Timer 250 ... Automatic driving device 251 ... Drive device 252 ... Braking device 253 ... Steering device 260 ... Sensor 261 ... Steering angle sensor 262 ... Vehicle speed sensor 263 ... Radar device 270 ... Navigation device 271 ... Position detection Device 272 ... Map information 273 ... Facility information 274 ... Output device 280 ... Accessory 281 ... Audio device 282 ... Air conditioner 283 ... External communication device 291 ... Air bag device 292 ... Self-diagnosis device

Claims (17)

An operation support method for causing a processor of an operation support device to output support information regarding an operation of a user who uses a function included in a device of a vehicle,
The processor is
Obtaining the user's motion information,
Guessing the operation of the function to be executed by the user based on the operation information of the user,
At least temporarily stored in the storage device, referring to the operation history information including the operation required time required to complete the input of the operation command for executing the function,
Based on the operation required time for each user, to determine the support mode of the support information for supporting the operation of the function that the user is supposed to perform,
An operation support method for outputting the support information in the support mode. In the process of determining the support mode,
The processor is
A voice output mode in which the support information is voice output using a voice output device;
A display output mode in which the support information is presented using an image output device,
The operation support method according to claim 1, wherein any one of a light emission mode for outputting the support information using a light emitting device is determined as the support mode. In the process of determining the support mode,
The processor is
With reference to the operation history information including the operation required time required until the input of the operation command for executing the function, which is associated with the identified user,
Based on the identification result of the user who has issued the operation command, preferentially selects the support mode associated with the relatively short operation required time from the operation required time of the identified user. ,
The operation support method according to claim 1, wherein the selected support mode is determined as the support mode for outputting the support information. In the process of determining the support mode,
The processor is
The operation history information including the operation required time required until the input of the operation command for executing the function, which is associated with the function executed by the identified user, is referred to,
The operation required time stored for each of the functions of the identified user based on the identification result of the user who has issued the operation command and the estimation result of the operation of the device which the user intends to execute. Of these, preferentially selecting the support mode associated with the relatively short operation time,
The operation support method according to claim 1, wherein the selected support mode is determined as the support mode for outputting the support information. In the process of determining the support mode,
The processor is
Referring to operation history information including operation evaluation time from completion of output of the support information to completion of input of the operation command to execute the function, which is associated with the identified user.
Based on the identification result of the user who has issued the operation command, the support mode associated with the relatively short operation evaluation time is preferentially selected from the operation evaluation times of the identified user. Like so,
The operation support method according to any one of claims 1 to 4, wherein the selected support mode is determined as the support mode for outputting the support information. In the process of determining the support mode,
The processor is
Referenced to operation history information including operation evaluation time from completion of output of the support information to completion of input of the operation command to execute the function, which is associated with the function executed by the identified user.
The operation evaluation time stored for each of the functions of the identified user based on the identification result of the user who has issued the operation command and the estimation result of the operation of the device that the user intends to execute. Of these, weighting is performed so that the support mode associated with the relatively short operation evaluation time is preferentially selected.
The operation support method according to any one of claims 1 to 4, wherein the selected support mode is determined as the support mode for outputting the support information. The processor is
Identifying the user who issued the operation command,
When the elapsed time from the timing when it is determined that the operation of the function is started is equal to or more than the predetermined time set based on the operation required time,
The operation support method according to claim 1, wherein the support mode is selected, and the support information is output according to the selected support mode. In the process of inferring the operation of the function to be executed by the user,
The processor is
Based on the position of the operation unit that is the target of the user's motion extracted from the motion information, by referring to the operation correspondence information in which the detected position of the motion information and the position of the operation unit of the device are associated in advance. The operation support method according to any one of claims 1 to 7, wherein the operation of the function to be performed by the user is estimated. In the process of inferring the operation of the function to be executed by the user,
The processor is
The operation unit of the function predicted from the operation of the user and the operation information are referred to in advance to refer to operation correspondence information, and the operation of the function to be executed by the user is estimated based on the operation information. The operation support method according to any one of claims 1 to 7.   The operation support method according to any one of claims 1 to 9, wherein the motion information includes information regarding a movement of a hand or a head of the user or a direction of a line of sight extracted from a captured image.   The operation support method according to any one of claims 1 to 10, wherein the motion information includes utterance information of the user extracted from collected voice information. In the process of inferring the operation of the function to be executed by the user,
The processor is
Judging the state of the vehicle from the vehicle detection information,
The state of the vehicle determined by referring to the operation correspondence information in which the vehicle state information regarding the state of the vehicle and the operation unit of the function according to the traveling environment predicted from the vehicle state information are associated in advance. The operation support method according to claim 1, wherein an operation unit of the function to be executed by the user is estimated based on the above.   The vehicle state information includes the type of the driving mode of the vehicle, the speed of the vehicle, the traffic jam condition of the route on which the vehicle travels, the presence or absence of damage to the vehicle, the presence or absence of failure or abnormality of the equipment of the vehicle, the vehicle. 13. The operation support method according to claim 12, further comprising any one or more of the position, the indoor temperature of the vehicle, the outside air temperature of the vehicle, and the operation of the wiper of the vehicle. The processor is
The method according to any one of claims 1 to 13, wherein the determined support mode is changed when the operation state information of the user and / or the vehicle state information regarding the state of the vehicle is in a predetermined state. Operation support method. When the start of operation of the function is detected,
The processor is
Identification information of the user,
An inference result of an operation of the function that the user intends to perform,
The operation history information in which the operation required time required for the user to complete the input of the operation command to execute the function is associated with the operation history information is transmitted to the storage device via a communication network,
The operation support method according to claim 1, wherein the operation history information is stored in the storage device. When the start of operation of the function is detected,
The processor is
Identification information of the user,
The function executed according to the operation command,
An operation evaluation time from completion of output of the support information to completion of input of the operation command for executing the function, which is associated with the function executed by the user,
Operation history information in which the support mode at the time of outputting the support information is associated with the operation history information is transmitted to the storage device via a communication network,
The operation support method according to claim 1, wherein the operation history information is stored in the storage device. An operation support device for outputting support information regarding an operation to a user who uses a function of a vehicle device,
The processor provided in the operation support device,
Obtaining the user's motion information,
Guessing the operation of the function to be performed by the user based on the operation information,
At least temporarily stored in a storage device, with reference to operation history information including an operation required time required by the user to perform an operation command to execute the function,
Based on the operation required time for each user, determine the support mode of the support information for supporting the estimated operation,
An operation support device that outputs the support information in the support mode.

Images