JP2023083085A - Operator support method - Google Patents

Abstract

To support an operator who operates or remotely operates a mobile object in earlier timing.SOLUTION: An operator support method supports an operator who operates or remotely operates a mobile object. The operator support method comprises processing of: (1) acquiring a visual line direction of the operator; (2) setting a target visual line direction requiring gaze of the operator according to a scene in which the mobile object is placed; (3) determining whether or not a divergence degree between the visual line direction of the operator and the target visual line direction exceeds a threshold; and (4) notifying the operator of a guide direction from the visual line direction of the operator toward the target visual line direction through at least one of tactile sensation, hearing sensation, and visual sensation of the operator when the divergence degree exceeds the threshold.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to technology for assisting an operator who operates or remotely operates a mobile object.

Patent Literature 1 discloses a vehicle warning device that makes a driver aware of an approaching obstacle. A vehicle warning device includes an obstacle sensor that detects an obstacle approaching a vehicle, a tactile information transmission means, and a control unit. The tactile information transmission means transmits tactile information to the driver by vibrating the seat surface on which the driver sits. When an obstacle is detected by the obstacle sensor, the controller determines whether the driver is performing an avoidance operation. Here, the avoidance operation is a steering operation in a direction away from an obstacle or a braking operation for decelerating the vehicle. When the driver is not performing an avoidance operation, the control unit controls the tactile information transmission means so that the seat surface in the direction in which the obstacle exists vibrates.

In addition, Patent Document 2 and Patent Document 3 are known as techniques for giving a warning to the driver through the sense of touch.

JP 2006-119840 A JP 2016-024509 A JP 2009-129052 A

Consider an operator who operates or remotely operates a mobile object such as a vehicle. Due to the limited field of view of the operator, the operator may miss events requiring attention. Therefore, operator assistance techniques such as issuing a notification prompting the operator to pay attention are useful.

However, if a notification is issued whenever an event requiring attention is detected, the operator feels annoyed by the notification. This is because the operator may already be aware of the event requiring attention. Therefore, it is desirable to issue a notification prompting the operator to pay attention as necessary.

In general, an operator's operation of a moving object proceeds in order of "1. Recognition", "2. Judgment", and "3. Operation". According to the technique disclosed in Patent Document 1, the seat surface is vibrated only after detecting that there is no "3. operation". Therefore, the timing at which the operator receives the notification is delayed. In that case, the effect of operator assistance is not necessarily sufficient.

One object of the present disclosure is to provide a technology capable of quickly assisting an operator who operates or remotely operates a mobile object.

A first aspect relates to an operator assistance method for assisting an operator who operates or remotely operates a mobile object.
The operator assistance method is
a process of acquiring the line-of-sight direction of the operator;
A process of setting a target line-of-sight direction that requires an operator's gaze according to a scene in which the moving object is placed;
a process of determining whether or not the degree of divergence between the line-of-sight direction of the operator and the target line-of-sight direction exceeds a threshold;
and a process of notifying the operator of the guidance direction from the operator's line-of-sight direction to the target line-of-sight direction through at least one of the operator's tactile sense, auditory sense, and visual sense when the degree of deviation exceeds a threshold.

A second aspect relates to an operator assistance system that assists an operator who operates or remotely operates a mobile object.
The operator assistance system comprises one or more processors.
The one or more processors are
Get the line-of-sight direction of the operator,
setting a target line-of-sight direction that requires the operator's attention according to the scene in which the moving object is placed;
Determining whether the degree of divergence between the operator's line-of-sight direction and the target line-of-sight direction exceeds a threshold,
It is configured to notify the operator of the guiding direction from the operator's line-of-sight direction to the target line-of-sight direction through at least one of the operator's tactile, auditory, and visual senses when the degree of deviation exceeds a threshold.

According to the present disclosure, the operator gaze direction and the target gaze direction requiring the operator's gaze are considered. If the degree of divergence between the operator line-of-sight direction and the target line-of-sight direction exceeds the threshold, there is a high possibility that the operator has not recognized an event requiring the operator's gaze. Therefore, in that case, the operator is notified of the guiding direction from the operator line-of-sight direction to the target line-of-sight direction. As a result, it is expected that the operator will turn his or her line of sight toward the target line-of-sight direction and recognize an event requiring attention.

In general, an operator's operation of a moving object proceeds in order of "1. Recognition", "2. Judgment", and "3. Operation". According to the present disclosure, the notification is issued with the absence of "1. Recognition" as a trigger. As a result, it becomes possible for the operator to recognize events that require attention at an earlier timing. Therefore, the accuracy of the operation/remote control of the moving body by the operator is improved.

Furthermore, according to the present disclosure, when the degree of divergence between the operator line-of-sight direction and the target line-of-sight direction exceeds a threshold, line-of-sight guidance processing is performed. Since the line-of-sight guidance process is performed not unconditionally but as needed, it is possible to prevent the operator from feeling annoyed.

FIG. 4 is a conceptual diagram for explaining operator assistance processing according to the embodiment of the present disclosure; 1 is a block diagram showing a configuration example of an operator support system according to an embodiment of the present disclosure; FIG. 6 is a flowchart showing operator support processing according to the embodiment of the present disclosure; 4 is a block diagram showing an example of driving environment information according to the embodiment of the present disclosure; FIG. FIG. 10 is a flowchart for explaining an example of target line-of-sight direction setting processing (step S20) according to the embodiment of the present disclosure; FIG. FIG. 10 is a flowchart for explaining an example of target line-of-sight direction setting processing (step S20) according to the embodiment of the present disclosure; FIG. FIG. 10 is a flowchart for explaining an example of target line-of-sight direction setting processing (step S20) according to the embodiment of the present disclosure; FIG. FIG. 10 is a flowchart for explaining an example of target line-of-sight direction setting processing (step S20) according to the embodiment of the present disclosure; FIG. FIG. 5 is a conceptual diagram for explaining an example of visual guidance processing (step S40) according to the embodiment of the present disclosure; FIG. 7 is a conceptual diagram for explaining another example of the visual guidance processing (step S40) according to the embodiment of the present disclosure; FIG. 10 is a conceptual diagram for explaining still another example of the visual guidance processing (step S40) according to the embodiment of the present disclosure; FIG. FIG. 10 is a conceptual diagram for explaining still another example of the visual guidance processing (step S40) according to the embodiment of the present disclosure; FIG. 1 is a block diagram showing a configuration example of a vehicle control system according to an embodiment of the present disclosure; FIG. 1 is a conceptual diagram for explaining a remote operation system according to an embodiment of the present disclosure; FIG. 1 is a block diagram showing a configuration example of a remote operation device according to an embodiment of the present disclosure; FIG.

Embodiments of the present disclosure will be described with reference to the accompanying drawings.

1. Overview FIG. 1 is a conceptual diagram for explaining operator support processing according to the present embodiment. An operator O operates or remotely operates a mobile object M. FIG. For example, the mobile object M is a vehicle V. When the mobile object M is a vehicle V, the operator O is a driver who drives the vehicle V or a remote operator who remotely drives the vehicle V. FIG. Vehicle V may be a self-driving vehicle. The mobile object M is not limited to the vehicle V, and may be, for example, a robot, a flying object, or the like. Examples of robots include physical distribution robots, work robots, and the like. Examples of flying objects include airplanes, drones, and the like.

Since the operator O has a limited field of view, there is a possibility that the operator O will overlook an event requiring attention. Therefore, an operator support process is useful, such as issuing a notice to prompt the operator O to pay attention. In particular, this embodiment provides an operator support process (line-of-sight guidance support process) for issuing a notification for guiding the line of sight of the operator O as necessary.

In FIG. 1, the "target line-of-sight direction LSt" is the direction that the operator O needs to gaze at. This target line-of-sight direction LSt is set according to the scene in which the moving body M is placed. For example, in a scene in which the moving object M turns leftward, the target line-of-sight direction LSt is set leftward from the front. Various examples are conceivable as the policy for setting the target line-of-sight direction LSt. Various setting examples of the target line-of-sight direction LSt will be described later.

On the other hand, the “operator line-of-sight direction LSo” is the line-of-sight direction of the operator O. FIG. The operator line-of-sight direction LSo does not necessarily match the target line-of-sight direction LSt. When the operator line-of-sight direction LSo and the target line-of-sight direction LSt are greatly deviated, there is a high possibility that the operator O has not recognized an event that requires the operator's O gaze. Therefore, in that case, the "line-of-sight guidance process" is executed. Specifically, the operator O is notified of the guiding direction from the operator line-of-sight direction LSo to the target line-of-sight direction LSt. The guidance direction is notified to the operator O through at least one of the operator's sense of touch, hearing, and sight.

FIG. 2 is a block diagram showing a configuration example of an operator support system 1 that executes operator support processing (line-of-sight guidance support processing) according to this embodiment. The operator support system 1 includes an operator monitor 10 , a target line-of-sight direction setting section 20 , an operating condition determination section 30 , a line-of-sight guidance processing section 40 and a line-of-sight guidance device 50 .

The operator monitor 10 detects the state of the operator O. FIG. For example, operator monitor 10 includes a camera that captures operator O's face. The operator line-of-sight direction LSo can be calculated based on the detection result by the operator monitor 10 . That is, by using the operator monitor 10, the operator line-of-sight direction LSo can be obtained.

FIG. 3 is a flowchart showing operator support processing by the operator support system 1 according to this embodiment. The processing flow shown in FIG. 3 is repeatedly executed at fixed cycles.

In step S<b>10 (line-of-sight direction acquisition processing), the operator's line-of-sight direction LSo is obtained based on the detection result of the operator monitor 10 .

In step S20 (target line-of-sight direction setting process), the target line-of-sight direction setting unit 20 sets a target line-of-sight direction LSt that requires the operator's O gaze. More specifically, the target line-of-sight direction setting unit 20 sets the target line-of-sight direction LSt according to the scene in which the moving object M is placed.

In step S30 (operation condition determination process), the operation condition determination unit 30 determines whether or not the operation condition for the line-of-sight guidance process is satisfied based on the operator line-of-sight direction LSo and the target line-of-sight direction LSt. Here, the operating condition of the line-of-sight guidance processing is that the degree of divergence D between the operator line-of-sight direction LSo and the target line-of-sight direction LSt exceeds a threshold value Dth. Various examples are conceivable as a method for calculating the degree of divergence D. FIG. Various examples of the degree of divergence D are described below. If the deviation D exceeds the threshold Dth (step S30; Yes), the process proceeds to step S40. Otherwise (step S30; No), step S40 is not executed, and the process in this cycle ends.

In step S40 (line-of-sight guidance processing), the line-of-sight guidance processing unit 40 notifies the operator O of the guidance direction from the operator line-of-sight direction LSo to the target line-of-sight direction LSt. The line-of-sight guidance device 50 is used for this notification. The visual guidance device 50 is an operator interface that transmits information to the operator O. FIG. For example, the visual guidance device 50 is a haptic device that transmits information to the operator O through the operator's sense of touch. As another example, the visual guidance device 50 may be a speaker that transmits information to the operator O through the operator's hearing. As yet another example, the visual guidance device 50 may be a display that conveys information to the operator O through the operator's O vision. The visual guidance processing unit 40 notifies the operator O of the guidance direction through at least one of tactile, auditory, and visual sensations by controlling the visual guidance device 50 .

As described above, in the operator support processing according to the present embodiment, the operator line-of-sight direction LSo and the target line-of-sight direction LSt requiring the operator's O gaze are considered. If the degree of divergence D between the operator line-of-sight direction LSo and the target line-of-sight direction LSt exceeds the threshold value Dth, there is a high possibility that the operator O has not recognized an event that requires the operator's attention. Therefore, in that case, the operator O is notified of the guiding direction from the operator line-of-sight direction LSo to the target line-of-sight direction LSt. As a result, it is expected that the operator O will turn his or her line of sight toward the target line-of-sight direction LSt and recognize an event requiring attention.

In general, the operation of the moving object M by the operator O proceeds in the order of "1. Recognition", "2. Judgment", and "3. Operation". As a comparative example, according to the technology disclosed in Patent Document 1 above, a notification is issued only after detecting that there is no "3. operation". On the other hand, according to the present embodiment, the notification is issued with the absence of "1. Recognition" as a trigger. That is, according to the present embodiment, assistance to the operator O is provided earlier than in the comparative example. As a result, the operator O can recognize an event requiring attention at an earlier timing. Therefore, the accuracy of operation/remote control of the moving body M by the operator O is improved. This means that operator assistance is effective.

As another comparison, consider a notification whenever an event requiring attention is detected. In this comparative example, the operator O feels annoyed by the notification. This is because the operator O may already be aware of an event requiring attention. On the other hand, according to the present embodiment, the line-of-sight guidance process is performed when the degree of divergence D between the operator line-of-sight direction LSo and the target line-of-sight direction LSt exceeds the threshold value Dth. Since the line-of-sight guidance process is performed not unconditionally but as needed, the operator O is prevented from feeling annoyed.

The operator support system 1 and operator support processing according to the present embodiment will be described in more detail below. In addition, below, the case where the mobile body M is the vehicle V is considered. The same applies to mobile bodies M other than vehicles V. FIG.

2. Example of Target Line-of-Sight Direction Setting Processing (Step S20) As described above, the target line-of-sight direction LSt that requires the gaze of the operator O is set according to the scene in which the vehicle V is placed. The scene in which the vehicle V is placed can be grasped based on the driving environment of the vehicle V. FIG. The target line-of-sight direction setting unit 20 sets the target line-of-sight direction LSt based on the driving environment information 200 indicating the driving environment of the vehicle V. FIG.

FIG. 4 is a block diagram showing an example of the driving environment information 200. As shown in FIG. The driving environment information 200 includes surrounding situation information 210, vehicle state information 220, map information 230, vehicle position information 240, target route information 250, and the like.

The surrounding situation information 210 is information indicating the surrounding situation of the vehicle V. FIG. The surrounding situation information 210 is obtained by a recognition sensor mounted on the vehicle V. FIG. Examples of the recognition sensor include a camera, LIDAR (Laser Imaging Detection and Ranging), radar, and the like. For example, surroundings information 210 includes image information captured by a camera. As another example, the surroundings information 210 includes point cloud information obtained by LIDAR.

Surroundings information 210 further includes object information 215 about objects in the vehicle V's surroundings. Examples of objects include pedestrians, bicycles, other vehicles (leading vehicle, following vehicle, parked vehicle, etc.), road structure (white lines, crosswalks, curbs, guardrails, etc.), traffic lights, signs, falling objects, etc. be. The object information 215 indicates the relative position (relative direction) and relative speed of the object with respect to the vehicle V. FIG. For example, by analyzing image information obtained by a camera, an object can be identified and the relative position of the object can be calculated. Also, based on the point cloud information obtained by LIDAR, an object can be identified and the relative position and relative velocity of the object can be obtained. The object information 215 may include the moving direction and moving speed of the object.

The vehicle state information 220 is information indicating the state of the vehicle V. FIG. The state of the vehicle V is exemplified by speed, acceleration, yaw rate, steering angle, winker state, and the like. The vehicle state information 220 may include the driving mode of the vehicle V (automatic driving/manual driving).

Map information 230 includes a general navigation map. Map information 230 may indicate lane layout, road geometry, and the like. Map information 230 may include location information for landmarks, traffic lights, signs, and the like. Map information 230 is obtained from a map database. The map database may be stored in the storage device of the vehicle V, or may be stored in an external management server.

The vehicle position information 240 is information indicating the position of the vehicle V. FIG. For example, the vehicle position information 240 is obtained by a position sensor mounted on the vehicle V (eg, GNSS (Global Navigation Satellite System)). Further, highly accurate vehicle position information 240 may be acquired by well-known self-position estimation processing (localization) using object information 215 and map information 230 .

The target route information 250 indicates the target route of the vehicle V. FIG. For example, the target route is set by the navigation system mounted on the vehicle V. FIG. As another example, the target route may be determined by an automated driving system that controls automated driving of the vehicle V. FIG.

FIG. 5 is a flowchart for explaining an example of the target line-of-sight direction setting process (step S20).

In step S200, the target line-of-sight direction setting unit 20 determines whether or not the signal ahead (nearest) of the vehicle V is a red/yellow signal. A traffic light and its lighting state can be recognized based on surrounding situation information 210 (object information 215, image information obtained by a camera). If the signal in front of the vehicle V is a red light or yellow light (step S200; Yes), the target line-of-sight direction setting unit 20 sets the direction LSt_1 of the red light or yellow light as the target line-of-sight direction LSt (step S201 ). Otherwise (step S200; No), the process proceeds to step S202.

In step S<b>202 , the target line-of-sight direction setting unit 20 determines whether or not the vehicle V is approaching an intersection based on the vehicle position information 240 and the map information 230 . If the vehicle V is approaching the intersection (step S202; Yes), the process proceeds to step S203. Otherwise (step S202; No), the process proceeds to the straight-ahead scene process (C; see FIG. 8).

In step S203, the target line-of-sight direction setting unit 20 determines the traveling direction (left turn, right turn, or straight ahead) of the vehicle V passing through the intersection. For example, the direction of travel is obtained from the target route information 250 . As another example, the direction of travel can be obtained from the winker state (operating direction of the winker lever) indicated by the vehicle state information 220 . If it is a left turn, the process proceeds to left turn scene processing (A; see FIG. 6). If it is a right turn, the process proceeds to the right turn scene process (B; see FIG. 7). In the case of straight ahead, the process proceeds to straight ahead scene processing (C; see FIG. 8).

FIG. 6 is a flow chart showing an example of processing for a left turn scene.

In step S210, the target line-of-sight direction setting unit 20 determines whether or not there is a "trapped vehicle" that the vehicle V may be involved in when turning left. The involved vehicle can be recognized based on the object information 215 . If there is a vehicle involved (step S210; Yes), the target line-of-sight direction setting unit 20 sets the direction LSt_2 of the vehicle involved as the target line-of-sight direction LSt (step S211). Otherwise (step S210; No), the process proceeds to step S212.

In step S212, the target line-of-sight direction setting unit 20 determines whether or not there is a pedestrian crossing in the left turn direction. Crosswalks can be recognized based on object information 215 . If a crosswalk exists (step S212; Yes), the target line-of-sight direction setting unit 20 sets the direction LSt_3 of the crosswalk as the target line-of-sight direction LSt (step S213). Otherwise (step S212; No), the process proceeds to step S214. Comparing the direction LSt_2 of the trapped vehicle and the direction LSt_3 of the pedestrian crossing, the direction LSt_2 of the trapped vehicle is further to the left.

In step S<b>214 , the target line-of-sight direction setting unit 20 detects the intersection angle (the angle at which the roads intersect at the intersection) based on the map information 230 and the vehicle position information 240 . Based on the intersection angle, the target line-of-sight direction setting unit 20 sets the direction LSt_4 of the left turn road as the target line-of-sight direction LSt (step S215).

FIG. 7 is a flow chart showing an example of processing for a right turn scene.

In step S220, the target line-of-sight direction setting unit 20 determines whether or not there is an oncoming vehicle in the oncoming lane. Oncoming vehicles can be recognized based on object information 215 . If there is an oncoming vehicle (step S220; Yes), the target line-of-sight direction setting unit 20 sets the direction LSt_5 of the oncoming vehicle as the target line-of-sight direction LSt (step S221). Otherwise (step S220; No), the process proceeds to step S222.

In step S220, the target line-of-sight direction setting unit 20 determines whether or not there is a pedestrian crossing in the right-turn direction. Crosswalks can be recognized based on object information 215 . If there is a crosswalk (step S222; Yes), the target line-of-sight direction setting unit 20 sets the direction LSt_6 of the crosswalk as the target line-of-sight direction LSt (step S223). Otherwise (step S222; No), the process proceeds to step S224.

In step S224, the target line-of-sight direction setting unit 20 determines whether or not an obstacle exists in the right turn direction. Examples of obstacles include pedestrians, bicycles, preceding vehicles, parked vehicles, falling objects, and the like. Obstacles can be recognized based on object information 215 . If an obstacle exists (step S224; Yes), the target line-of-sight direction setting unit 20 sets the direction LSt_7 of the obstacle as the target line-of-sight direction LSt (step S225). Otherwise (step S224; No), the process proceeds to step S226.

In step S<b>226 , the target line-of-sight direction setting unit 20 detects the intersection angle (the angle at which the roads intersect at the intersection) based on the map information 230 and the vehicle position information 240 . Then, the target line-of-sight direction setting unit 20 sets the direction LSt_8 of the right turn destination road as the target line-of-sight direction LSt based on the intersection angle (step S227).

FIG. 8 is a flow chart showing an example of processing for a straight-ahead scene.

In step S<b>230 , the target line-of-sight direction setting unit 20 determines whether or not an obstacle exists in front of the vehicle V. Examples of obstacles include pedestrians, bicycles, preceding vehicles, parked vehicles, falling objects, and the like. Obstacles can be recognized based on object information 215 . If an obstacle exists (step S230; Yes), the process proceeds to step S231. Otherwise (step S230; No), the process proceeds to step S233.

In step S<b>231 , the target line-of-sight direction setting unit 20 calculates the distance to each obstacle based on the object information 215 . In step S232, the target line-of-sight direction setting unit 20 sets the target line-of-sight direction LSt based on the distance to each obstacle. For example, the target line-of-sight direction setting unit 20 sets the direction LSt_9 of the obstacle closest to the vehicle V as the target line-of-sight direction LSt.

In step S233, the target line-of-sight direction setting unit 20 determines whether or not there is a following vehicle. A following vehicle can be recognized based on the object information 215 . If there is a following vehicle (step S233; Yes), the target line-of-sight direction setting unit 20 sets the direction LSt_10 of the following vehicle as the target line-of-sight direction LSt (step S234). Otherwise (step S233; No), the process proceeds to step S235.

In step S<b>235 , the target line-of-sight direction setting unit 20 determines whether or not the adjacent lane is a merging lane based on the map information 230 and the vehicle position information 240 . If the adjacent lane is a merging lane (step S235; Yes), the target line-of-sight direction setting unit 20 sets the direction LSt_11 of the merging lane as the target line-of-sight direction LSt (step S236). Otherwise (step S235; No), the target line-of-sight direction setting unit 20 sets the front direction LSt_12 as the target line-of-sight direction LSt (step S237).

As another example, the line-of-sight direction of the model driver may be used as the target line-of-sight direction LSt. For example, the line-of-sight direction of the exemplary driver in each scene is modeled in advance through machine learning. For example, the line-of-sight direction of the model driver is modeled by using image information obtained by a camera and the line-of-sight direction of the model driver as learning data. In step S20, the target line-of-sight direction setting unit 20 acquires the line-of-sight direction of the exemplary driver according to the scene in which the vehicle V is placed, based on the image information obtained by the camera and the model. Then, the target line-of-sight direction setting unit 20 sets the acquired line-of-sight direction of the model driver as the target line-of-sight direction LSt.

3. Example of Operating Condition Determining Process (Step S30) As described above, the operating condition of the line-of-sight guidance process is that the divergence D between the operator line-of-sight direction LSo and the target line-of-sight direction LSt exceeds the threshold value Dth. The operating condition determination unit 30 calculates the divergence D based on the operator line-of-sight direction LSo and the target line-of-sight direction LSt for each predetermined condition determination period Tc.

For example, the operating condition determination unit 30 calculates the angle θ (absolute value) formed by the operator line-of-sight direction LSo and the target line-of-sight direction LSt. The operating condition determination unit 30 may extract only the angles θx exceeding a predetermined threshold θth from the angles θ. The operating condition determination unit 30 may measure the duration Tx during which the angle θ exceeds the threshold θth. As the degree of divergence D, the following various examples can be considered.
<Example 1> Sum or average value of angles θ during condition determination period Tc <Example 2> Sum or average value of (θx−θth) during condition determination period Tc <Example 3> Sum of duration Tx during condition determination period Tc < Example 4> sum of (θx−θth)×Tx in condition determination period Tc

By adjusting the threshold Dth or the threshold θth, the sensitivity of the operating conditions can be adjusted. By lowering the threshold value and increasing the sensitivity, it is possible to shorten the time until the line-of-sight guidance process and allow the operator O to respond quickly. Conversely, by raising the threshold value and lowering the sensitivity, it is possible to suppress erroneous operations and unnecessary operations of the visual guidance processing.

As an example, consider a case where the vehicle V is compatible with both the automatic driving mode and the remote driving mode. When switching from the automatic driving mode to the remote driving mode, the remote driving operator needs to quickly recognize and confirm the surrounding conditions of the vehicle V. In this regard, it can be said that the target line-of-sight direction LSt, which is the direction requiring gaze, indicates the direction to be checked first. Therefore, it is conceivable to increase the sensitivity by lowering the threshold for a certain period of time after switching (transition) from the automatic operation mode to the remote operation mode, compared to other periods. This makes it easier for the remote operator's line of sight to turn to the target line-of-sight direction LSt more quickly, so that the remote operator can quickly recognize and confirm events that require his or her attention.

4. Example of Line-of-Sight Guidance Processing (Step S40) The line-of-sight guidance processing unit 40 notifies the operator O of the guidance direction from the operator line-of-sight direction LSo to the target line-of-sight direction LSt by controlling the line-of-sight guidance device 50 . The visual guidance processing unit 40 may increase the control amount of the visual guidance device 50 as the degree of divergence D increases. Various examples of eye guidance treatments and eye guidance devices 50 are described below.

In the examples shown in FIGS. 9 and 10, the visual guidance device 50 is a haptic device that transmits information to the operator O through touch. In the example shown in FIG. 9, the visual guidance device 50 is glasses that the operator O wears on his head. For example, when guiding the line of sight to the left, the left frame of the glasses vibrates. In the example shown in FIG. 10, the visual guidance device 50 is a seat on which the operator O sits. For example, when guiding the line of sight to the left, the left side of the seat vibrates. The vibration intensity may increase as the degree of divergence D increases. Guidance of the line of sight through the sense of touch is intuitive and easy for the operator O to understand.

In the example shown in FIG. 11, the visual guidance device 50 is a speaker that transmits information to the operator O through hearing. For example, a sound is generated from a speaker positioned in the target line-of-sight direction LSt. This allows the operator to accurately gaze at the target line-of-sight direction LSt.

In the example shown in FIG. 12, the visual guidance device 50 is a display that conveys information to the operator O through vision. For example, an arrow indicating the guidance direction from the operator line-of-sight direction LSo to the target line-of-sight direction LSt is displayed on the display.

Since the operator O has a limited field of view, the operator O may miss the information displayed on the display. From the viewpoint of preventing overlooking, visual guidance processing through touch or sound is preferable.

5. Vehicle Control System FIG. 13 is a block diagram showing a configuration example of a vehicle control system 100 according to the present embodiment. A vehicle control system 100 is mounted on a vehicle V and controls the vehicle V. FIG. The vehicle control system 100 includes an operator monitor 10 , a visual guidance device 50 , a communication device 110 , a sensor group 120 , a travel device 130 and a control device 150 .

The communication device 110 communicates with the outside of the vehicle V. FIG. For example, the communication device 110 communicates with a remote driving device that remotely drives the vehicle V. FIG.

The sensor group 120 includes recognition sensors, vehicle state sensors, position sensors, and the like. The recognition sensor recognizes (detects) the situation around the vehicle V. FIG. Examples of recognition sensors include cameras, LIDAR, radar, and the like. A vehicle state sensor detects the state of the vehicle V. FIG. Vehicle state sensors include speed sensors, acceleration sensors, yaw rate sensors, steering angle sensors, and the like. A position sensor detects the position of the vehicle V. FIG.

Traveling device 130 includes a steering device, a driving device, and a braking device. The steering device steers the wheels. For example, the steering system includes a power steering (EPS: Electric Power Steering) system. A driving device is a power source that generates a driving force. An engine, an electric motor, an in-wheel motor, etc. are illustrated as a drive device. A braking device generates a braking force.

Control device 150 controls vehicle V. FIG. The controller 150 includes one or more processors 160 (hereinafter simply processors 160) and one or more storage devices 170 (hereinafter simply storage devices 170). Processor 160 executes various processes. For example, processor 160 includes a CPU (Central Processing Unit). The storage device 170 stores various information necessary for processing by the processor 160 . Examples of the storage device 170 include volatile memory, nonvolatile memory, HDD (Hard Disk Drive), SSD (Solid State Drive), and the like. The control device 150 may include one or more ECUs (Electronic Control Units).

Vehicle control program 180 is a computer program executed by processor 160 . The functions of control device 150 are implemented by processor 160 executing vehicle control program 180 . Vehicle control program 180 is stored in storage device 170 . Alternatively, vehicle control program 180 may be recorded on a computer-readable recording medium.

The control device 150 acquires the driving environment information 200 (see FIG. 4) indicating the driving environment of the vehicle V using the sensor group 120 . Driving environment information 200 is stored in storage device 170 .

Further, the control device 150 executes vehicle travel control for controlling travel of the vehicle V. FIG. Vehicle travel control includes steering control, drive control, and braking control. Control device 150 executes vehicle travel control by controlling travel device 130 (steering device, drive device, and braking device).

The control device 150 may perform automatic driving control based on the driving environment information 200 . More specifically, control device 150 generates a travel plan for vehicle V based on driving environment information 200 . The driving plan is exemplified by maintaining the current driving lane, changing lanes, turning left or right, avoiding obstacles, and the like. Furthermore, based on the driving environment information 200, the control device 150 generates a target trajectory required for the vehicle V to travel according to the travel plan. A target trajectory includes a target position and a target velocity. Then, the control device 150 performs vehicle travel control so that the vehicle V follows the target trajectory.

Furthermore, the control device 150 uses the operator monitor 10 to acquire the operator line-of-sight direction LSo. The control device 150 sets the target line-of-sight direction LSt based on the driving environment information 200 . The control device 150 determines whether or not the operating conditions for the line-of-sight guidance process are satisfied. When the operating condition is satisfied, the control device 150 performs the visual guidance processing by controlling the visual guidance device 50 . That is, the target line-of-sight direction setting unit 20, the operating condition determination unit 30, and the line-of-sight guidance processing unit 40 shown in FIG. A vehicle control system 100 corresponds to the operator support system 1 shown in FIG.

6. Remote Driving System Next, consider a case where the operator O remotely drives the vehicle V. FIG. In this case, the vehicle control system 100 does not have to include the operator monitor 10 and the visual guidance device 50 .

FIG. 14 is a conceptual diagram for explaining the remote operation system according to this embodiment. A remote driving system includes a vehicle V and a remote driving device 300 . The remote driving device 300 is a device operated by an operator O (remote operator) when the vehicle V is remotely driven. Vehicle V (vehicle control system 100) and remote operation device 300 can communicate via a communication network.

The vehicle control system 100 transmits vehicle information VCL including at least part of the driving environment information 200 to the remote driving device 300 . The vehicle information VCL includes at least image information captured by the camera and object information 215 . The vehicle information VCL may include surrounding situation information 210 , vehicle state information 220 , vehicle position information 240 and target route information 250 . The remote driving device 300 receives the vehicle information VCL. The remote driving device 300 displays the vehicle information VCL and map information on the display device. The operator O remotely drives the vehicle V while referring to the information displayed on the display device. The remote operation device 300 transmits operation information OPE indicating the amount of operation by the operator O to the vehicle control system 100 . The vehicle control system 100 (control device 150) performs the above-described vehicle travel control according to the received operation information OPE.

FIG. 15 is a block diagram showing a configuration example of the remote operation device 300 according to this embodiment. The remote operation device 300 includes an operator monitor 10 , a visual guidance device 50 , a communication device 310 , a display device 320 , a remote control member 330 and a control device 350 .

Communication device 310 communicates with vehicle V (vehicle control system 100).

The display device 320 presents various information to the operator O by displaying various information.

The remote control member 330 is a member operated by the operator O when the vehicle V is remotely operated (remotely controlled). The remote control member 330 includes a steering wheel, an accelerator pedal, a brake pedal, direction indicators, and the like.

Control device 350 controls remote operation device 300 . Controller 350 includes one or more processors 360 (hereinafter simply processors 360) and one or more storage devices 370 (hereinafter simply storage devices 370). Processor 360 executes various processes. For example, processor 360 includes a CPU. Storage device 370 stores various information necessary for processing by processor 360 . Examples of the storage device 370 include volatile memory, nonvolatile memory, HDD, SSD, and the like.

Remote operation program 380 is a computer program executed by processor 360 . The functions of the control device 350 are implemented by the processor 360 executing the remote operation program 380 . Remote operation program 380 is stored in storage device 370 . Alternatively, remote operation program 380 may be recorded on a computer-readable recording medium. Remote operation program 380 may be provided via a network.

Control device 350 communicates with vehicle V (vehicle control system 100 ) via communication device 310 . Control device 350 receives vehicle information VCL transmitted from vehicle control system 100 . Vehicle information VCL is stored in storage device 370 . Control device 350 presents vehicle information VCL to operator O by displaying vehicle information VCL on display device 320 . Control device 350 also displays map information on display device 320 . The operator O can grasp the situation of the vehicle V based on the map information displayed on the display device 320 and the vehicle information VCL.

An operator O operates the remote control member 330 . The control device 350 acquires the operation amount of the remote control member 330 by the operator O. FIG. The amount of operation is detected by a sensor installed on the remote control member 330 . Control device 350 generates operation information OPE reflecting the amount of operation, and transmits the operation information OPE to vehicle control system 100 .

Further, the control device 350 uses the operator monitor 10 to acquire the operator line-of-sight direction LSo. The control device 350 sets the target line-of-sight direction LSt based on the vehicle information VCL corresponding to the driving environment information 200 . The control device 350 determines whether or not the operating conditions for the line-of-sight guidance process are satisfied. When the operating condition is satisfied, the control device 350 performs the visual guidance processing by controlling the visual guidance device 50 . That is, the target line-of-sight direction setting unit 20, the operating condition determination unit 30, and the line-of-sight guidance processing unit 40 shown in FIG. The remote operation device 300 corresponds to the operator assistance system 1 shown in FIG.

1 operator support system 10 operator monitor 20 target line-of-sight direction setting unit 30 operating condition determination unit 40 line-of-sight guidance processing unit 50 line-of-sight guidance device 100 vehicle control system 150 control device 160 processor 170 storage device 200 driving environment information 215 object information 300 remote driving device 350 control device 360 processor 370 storage device M moving body O operator V vehicle LSo operator line-of-sight direction LSt target line-of-sight direction OPE operation information VCL vehicle information

Claims (1)

An operator support method for supporting an operator who operates or remotely operates a mobile object, comprising:
a process of acquiring the line-of-sight direction of the operator;
A process of setting a target line-of-sight direction that requires the operator's gaze according to the scene in which the moving object is placed;
a process of determining whether the divergence between the line-of-sight direction of the operator and the target line-of-sight direction exceeds a threshold;
a process of notifying the operator of a guidance direction from the line-of-sight direction to the target line-of-sight direction through at least one of tactile, auditory, and visual senses of the operator when the degree of deviation exceeds the threshold; including operator assistance methods.

Images