JP2022173769A - Vehicle control method, automatic driving system, and automatic driving program - Google Patents

Abstract

To prevent timing of remote assistance to an automatic driving vehicle from being delayed in a situation that a traffic guide is present.SOLUTION: A vehicle control method controls an automatic driving vehicle comprising at least one camera and at least one microphone. The vehicle control method includes image acquisition processing for acquiring an image around the automatic driving vehicle using the at least one camera. The vehicle control method further includes horn sound detection processing for detecting sounds around the automatic driving vehicle by means of the at least one microphone and detecting a horn sound based on a sound detection result. Moreover, the vehicle control method includes auxiliary notification processing for transmitting the image to a remote assistance system which remotely assists the automatic driving vehicle, before transmitting a remote assistance request to the remote assistance system in response to detecting the horn sound.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to technology for controlling an autonomous vehicle that is a target of remote assistance.

Patent Literature 1 discloses vehicle braking force control in the presence of a traffic director. When the collision avoidance target is detected, the braking force control device automatically decelerates the vehicle and stops the vehicle at the target stop position in front of the collision avoidance target. When the collision avoidance target is a traffic director, the braking force control device makes the stop margin between the collision avoidance target and the target stop position smaller than usual.

Patent Literature 2 and Patent Literature 3 disclose techniques for identifying traffic guides based on image information acquired by an on-board camera of an autonomous vehicle.

Patent Literature 4 discloses a vehicle remote operation support system. A vehicle remote operation support system includes an automatically driving vehicle and a control center having an operator. When autonomous driving is difficult, an autonomous vehicle transmits peripheral information to a control center and calls an operator. The operator refers to the peripheral information and remotely controls the autonomous vehicle.

Japanese Patent Application Laid-Open No. 2020-23249 JP 2019-156192 A JP 2018-045397 A JP 2019-191982 A

2. Description of the Related Art A remote support technology is known in which an operator remotely supports traveling of an automatically driven vehicle in response to a request from the automatically driven vehicle. Now, consider a situation where there is a traffic director on the road. In order to follow the guidance instructions of the traffic director, the autonomous vehicle must first recognize the presence of the traffic director. After recognizing the existence of the traffic director, it is necessary to recognize the guidance instruction by the traffic director based on the motion of the traffic director. A self-driving vehicle may request remote assistance from an operator if it is unable to perceive or understand the guidance instructions of the traffic attendant.

The inventors of the present application have found the following problems with respect to situations in which an automatically driven vehicle needs to follow guidance instructions from a traffic guide. In general, it is relatively easy to recognize a "person" by image analysis, but it is difficult to determine whether or not the person is a traffic guide. In other words, it is actually not easy for a self-driving vehicle to quickly recognize the presence of a traffic director. It takes time to identify the traffic guide in the autonomous vehicle, and when remote assistance is requested after that, the timing of remote assistance by the operator is also delayed. This can have a significant impact on traffic flow around self-driving vehicles.

One object of the present disclosure is to provide a technique capable of suppressing a delay in the timing of remote support for an autonomous vehicle in a situation where a traffic guide is present.

A first aspect relates to a vehicle control method for controlling a self-driving vehicle comprising at least one camera and at least one microphone.
The vehicle control method is
an image acquisition process for acquiring an image of the surroundings of the automated driving vehicle using at least one camera;
A horn sound detection process that detects sounds around the automated driving vehicle using at least one microphone and detects a horn sound based on the sound detection result;
and preliminary notification processing of transmitting an image to a remote assistance system in response to detecting the horn sound before transmitting a remote assistance request to the remote assistance system that remotely assists the autonomous vehicle.

The second aspect further has the following features in addition to the first aspect.
At least one camera is a plurality of cameras.
The at least one microphone is a plurality of microphones.
The vehicle control method further includes a blowing direction estimation process of estimating the blowing direction of the horn based on the sound detection results from the plurality of microphones.
The preliminary notification process includes a process of selecting an image corresponding to the blowing direction from among the plurality of images captured by each of the plurality of cameras and transmitting the selected image to the remote support system.

The third aspect further has the following features in addition to the second aspect.
The vehicle control method further includes person recognition processing for analyzing the image corresponding to the sounding direction and determining whether or not a person is shown in the image corresponding to the sounding direction.
When a horn is detected and a person appears in the image corresponding to the direction of the horn, the preliminary notification process described above is performed.

The fourth aspect further has the following features in addition to the second or third aspect.
The vehicle control method further includes a guide member recognition process of analyzing the image corresponding to the direction of the sound and determining whether or not the traffic guide is shown in the image corresponding to the direction of the sound.

The fifth aspect further has the following features in addition to any one of the first to fourth aspects.
The vehicle control method further comprises:
Analyzing the image and determining whether or not the traffic guide is visible in the image;
If the image shows a traffic guide, a process of determining whether or not the automated driving vehicle can be driven automatically;
Remote support request processing for transmitting a remote support request to a remote support system when automatic driving is not possible.

A sixth aspect relates to an autonomous driving system for controlling an autonomous vehicle comprising at least one camera and at least one microphone.
The automated driving system
one or more processors;
One or a plurality of storage devices for storing image information representing an image of the surroundings of the automated driving vehicle captured by at least one camera, and audio information representing the sound of the surroundings of the automated driving vehicle detected by at least one microphone. and .
The one or more processors are
a horn sound detection process for detecting a horn sound based on the audio information;
In response to the detection of the horn sound, before sending a remote support request to the remote support system that provides remote support for the automated driving vehicle, a preliminary notification process of transmitting image information to the remote support system is executed.

A seventh aspect relates to a self-driving program for controlling a self-driving vehicle comprising at least one camera and at least one microphone.
The automated driving program is executed by a computer,
an image acquisition process for acquiring an image of the surroundings of the automated driving vehicle using at least one camera;
A horn sound detection process that detects sounds around the automated driving vehicle using at least one microphone and detects a horn sound based on the sound detection result;
and a preliminary notification process of sending an image to the remote support system before sending a remote support request to the remote support system for remotely supporting the automated driving vehicle in response to the detection of the horn sound.

According to the present disclosure, the horn sound is detected based on the voice detection result by the microphone mounted on the autonomous vehicle. Pre-notification processing is performed prior to transmission of the remote assistance request in response to detection of the horn. The preliminary notification process transmits an image obtained by the camera mounted on the autonomous vehicle to the remote support system. By this preliminary notification processing, the operator can anticipate the occurrence of a remote support request and perform standby for remote support. In particular, by looking at the image sent from the automatically driving vehicle, the operator can grasp the situation around the automatically driving vehicle in advance. In other words, the operator can grasp the situation around the autonomous vehicle at an early stage before actually receiving the remote assistance request. Therefore, when the autonomous vehicle actually issues a remote assistance request, the operator can promptly provide remote assistance to the autonomous vehicle. That is, the delay in the timing of remote support for the autonomous vehicle is suppressed. As a result, the impact on the traffic flow around the autonomous vehicle is also reduced.

1 is a conceptual diagram for explaining an overview of remote support according to an embodiment of the present disclosure; FIG. 1 is a conceptual diagram for explaining an overview of remote support according to an embodiment of the present disclosure; FIG. It is a conceptual diagram which shows the situation where a traffic director exists on a road. FIG. 4 is a conceptual diagram for explaining preliminary notification processing according to an embodiment of the present disclosure; FIG. 4 is a conceptual diagram for explaining preliminary notification processing according to an embodiment of the present disclosure; 1 is a block diagram showing a configuration example of an automatic driving system installed in an automatic driving vehicle according to an embodiment of the present disclosure; FIG. 4 is a block diagram showing an example of driving environment information according to the embodiment of the present disclosure; FIG. 7 is a flowchart illustrating a first example of processing related to preliminary notification processing according to the embodiment of the present disclosure; 6 is a flowchart showing an example of horn detection processing (step S11) according to the embodiment of the present disclosure; FIG. 11 is a flow chart showing a second example of processing related to preliminary notification processing according to the embodiment of the present disclosure; FIG. 4 is a timing chart for explaining sounding direction estimation processing (step S12) according to the embodiment of the present disclosure; FIG. 13 is a flowchart illustrating a third example of processing related to preliminary notification processing according to the embodiment of the present disclosure; FIG. 6 is a flowchart showing an example of processing related to remote assistance request processing according to the embodiment of the present disclosure; FIG. 4 is a conceptual diagram for explaining traffic guide recognition processing (step S30) according to the embodiment of the present disclosure; 1 is a block diagram showing a configuration example of a remote support system according to an embodiment of the present disclosure; FIG. 4 is a flow chart showing processing by the remote support system according to the embodiment of the present disclosure;

Embodiments of the present disclosure will be described with reference to the accompanying drawings.

1. Overview of Remote Support FIGS. 1 and 2 are conceptual diagrams for explaining an overview of remote support according to the present embodiment. The automatically driven vehicle 1 is a vehicle capable of automatically driving. The automatic driving here assumes that the driver does not necessarily have to concentrate on driving 100% (so-called level 3 or higher automatic driving). The automatic driving vehicle 1 is the target of remote assistance. The remote support system 100 communicates with the automatically driven vehicle 1 and executes remote support processing for remotely supporting the running of the automatically driven vehicle 1 .

Typically, a situation in which remote assistance is required for the autonomous driving vehicle 1 is a situation in which automatic driving is difficult. For example, when a traffic light installed at an intersection is exposed to sunlight, there is a possibility that the recognition accuracy of the traffic light indication will be degraded. If the signal indication cannot be recognized accurately, the autonomous vehicle 1 needs remote assistance for signal recognition. In addition, when the signal display cannot be recognized, it is also difficult for the automatic driving vehicle 1 to determine what action should be taken at what timing. Therefore, the automatic driving vehicle 1 requires remote support also for determination of actions and timing.

When remote assistance is required, the autonomous vehicle 1 communicates with the remote assistance system 100 and transmits a “remote assistance request REQ” to the remote assistance system 100 . For example, as shown in FIG. 2, the automatic driving vehicle 1 is equipped with a camera 22 . The camera 22 captures the surroundings of the automatically driven vehicle 1 and acquires an image IMG of the surroundings of the automatically driven vehicle 1 . The image IMG is typically a moving image, but may be a still image. In this embodiment, moving images and still images are collectively referred to as "image IMG". The remote assistance request REQ contains the image IMG. Alternatively, the image IMG is sent to the remote assistance system 100 together with the remote assistance request REQ.

The remote assistance system 100 receives a remote assistance request REQ and an image IMG from the autonomous vehicle 1 . The remote support system 100 has a display device 120 and displays the received image IMG on the display device 120 . In response to the remote support request REQ, the human operator sees the image IMG displayed on the display device 120, grasps the surroundings of the automatically driven vehicle 1, and remotely supports the operation of the automatically driven vehicle 1. .

Remote support by an operator includes recognition support, judgment support, remote operation, and the like. In the case of remote driving, the operator remotely drives the autonomous vehicle 1 instead of the autonomous driving system that controls the autonomous vehicle 1 . The remote support system 100 receives an "operator instruction INS" indicating the details of remote support from the operator. Then, the remote support system 100 communicates with the automatically driven vehicle 1 and notifies the automatically driven vehicle 1 of the operator instruction INS. The automatic driving vehicle 1 performs vehicle travel control according to the operator instruction INS.

In this way, the remote support system 100 receives information necessary for remote support such as the image IMG from the automatic driving vehicle 1 and provides the information to the operator (information provision processing). Further, the remote support system 100 receives an operator instruction INS from an operator and notifies the automatic driving vehicle 1 of the operator instruction INS (operator instruction notification process). That is, the remote support processing by the remote support system 100 includes information provision processing and operator instruction notification processing. It can also be said that the remote support system 100 is a system that assists remote support of the automated driving vehicle 1 by the operator.

2. Early Standby of Remote Assistance in a Situation Where Traffic Guides Are Present FIG. 3 shows a situation where a traffic guide TG is present on a road on which an autonomous vehicle 1 is traveling. Traffic guides TG include police officers, traffic patrols, security guards, and the like. Guidance instructions (signals) by the traffic guide TG include a stop instruction, an advance instruction, a march instruction, a slowdown instruction, a danger instruction, and the like.

The automatic driving vehicle 1 needs to follow guidance instructions from the traffic guide TG. For this purpose, first, the automatically driven vehicle 1 needs to recognize the presence of the traffic guide TG. After recognizing the existence of the traffic director TG, it is necessary to recognize the guidance instruction by the traffic director TG based on the motion of the traffic director TG. Here, when the guidance instruction by the traffic guide TG cannot be recognized or understood, the automatically driven vehicle 1 may request remote support by the operator.

The following problems exist regarding situations in which the automatically driven vehicle 1 needs to follow guidance instructions from the traffic guide TG. In general, it is relatively easy to recognize a "person" by image analysis, but it is difficult to determine whether or not the person is a traffic director TG. This is because the clothes and behavior of traffic guides TG are not uniform. Actually, it is not easy for the automatic driving vehicle 1 to quickly recognize the presence of the traffic director TG. It takes time to identify the traffic guide TG in the automatic driving vehicle 1, and when remote assistance is requested after that, the timing of the remote assistance by the operator is also delayed. This may greatly affect the traffic flow around the autonomous vehicle 1 .

Therefore, according to the present embodiment, the “preliminary notification process” is performed in order to suppress delays in the timing of remote assistance for the autonomous vehicle 1 . The preliminary notification process is a process for notifying the remote support system 100 in advance that a remote support request REQ may occur, and is performed before the remote support request REQ is actually sent to the remote support system 100. will be

FIG. 4 is a conceptual diagram for explaining preliminary notification processing according to this embodiment. According to the present embodiment, the "horn sound" of the horn WH used by the traffic director TG is used as a trigger for the preliminary notification process. A traffic guide TG must carry a horn WH, and blowing the horn WH when directing traffic is also stipulated in the traffic guidance security level 2.

The automatic driving vehicle 1 has a camera 22 and a microphone 24. - 特許庁The self-driving vehicle 1 detects surrounding sounds with the microphone 24 and detects a horn sound based on the sound detection result. In response to detecting the horn sound, the automatically driven vehicle 1 performs preliminary notification processing. Specifically, the autonomous vehicle 1 communicates with the remote support system 100 and transmits the preliminary notification PRE to the remote support system 100 . The preliminary notification PRE contains the image IMG captured by the camera 22 . That is, the automatic driving vehicle 1 transmits (uploads) the image IMG to the remote support system 100 before issuing the remote support request REQ by using the detection of the horn sound as a trigger.

The remote support system 100 displays the preliminary notification PRE sent from the automatic driving vehicle 1 on the display device 120 . As a result, the operator can anticipate the occurrence of the remote support request REQ and perform standby for remote support.

In particular, the image IMG is sent to the remote assistance system 100 along with the preliminary notification PRE, and the remote assistance system 100 displays the received image IMG on the display device 120 . By viewing the image IMG, the operator can grasp the situation around the automatic driving vehicle 1 . In other words, the operator can grasp the situation around the automatically driven vehicle 1 in advance at an early stage before actually receiving the remote assistance request REQ. The operator does not see the image IMG for the first time after receiving the remote assistance request REQ. Therefore, when the remote assistance request REQ is actually issued from the automatically driven vehicle 1, the operator can quickly input the operator instruction INS to the automatically driven vehicle 1. FIG. That is, delay in the timing of remote support for the autonomous vehicle 1 is suppressed. As a result, the influence on the traffic flow around the automatic driving vehicle 1 is also suppressed.

As a comparative example, when the image IMG is constantly uploaded from the automatically driven vehicle 1, it is necessary to allocate an operator to each automatically driven vehicle 1 to constantly monitor it. This is not realistic in terms of personnel and communication costs.

As shown in FIG. 5 , the autonomous vehicle 1 may have multiple cameras 22 and multiple microphones 24 . For example, multiple cameras 22 and multiple microphones 24 are mounted on the roof of the self-driving vehicle 1 . The pointing direction of each of the multiple microphones 24 may match the viewing direction of each of the multiple cameras 22 . In the example shown in FIG. 5, six cameras 22-1 to 22-6 and six microphones 24-1 to 24-6 are arranged so as to cover the entire circumference of the self-driving vehicle 1. In the example shown in FIG.

The autonomous vehicle 1 may estimate the direction in which the horn sounds based on the sound detection results from the multiple microphones 24 . In that case, the automatic driving vehicle 1 can select an image IMG corresponding to the sounding direction from among the plurality of images IMG captured by the plurality of cameras 22, and upload only the selected image IMG to the remote support system 100. . In the example shown in FIG. 5, an image IMG captured by camera 22-2 corresponding to microphone 24-2 is uploaded to remote assistance system 100. In the example shown in FIG.

By selectively uploading the image IMG in the blowing direction, it is possible to reduce the amount of communication, the communication time, and the communication cost compared to uploading all the images IMG in a plurality of directions. Since the amount of communication is reduced, image quality degradation due to congestion control is less likely to occur. In addition, reduction in the number of images IMG to be uploaded at the same time means that image transmission with low delay becomes possible. That is, the high-quality image IMG is displayed on the display device 120 of the remote support system 100 with low delay. As a result, the visibility and real-time performance of the image IMG are sufficiently ensured, and the operator can correctly grasp the situation around the automatic driving vehicle 1 . Furthermore, since there is a high possibility that the traffic director TG is shown in the image IMG in the direction of the sound, the operator can efficiently grasp the state of the traffic director TG.

3. Automatic driving system 3-1. Configuration Example FIG. 6 is a block diagram showing a configuration example of the automatic driving system 10 installed in the automatic driving vehicle 1 according to the present embodiment. The automatic driving system 10 includes a sensor group 20 , a communication device 30 , a traveling device 40 and a control device 50 . Sensor group 20 includes recognition sensor 21 , vehicle state sensor 23 , and at least one microphone 24 .

The recognition sensor 21 recognizes (detects) the circumstances around the automatically driven vehicle 1 . Recognition sensor 21 includes at least one camera 22 . The camera 22 captures the surroundings of the automatically driven vehicle 1 and acquires an image IMG of the surroundings of the automatically driven vehicle 1 . The image IMG is typically a moving image, but may be a still image. The recognition sensor 21 may also include LIDAR (Laser Imaging Detection and Ranging), radar, and the like.

A vehicle state sensor 23 detects the position and state of the automatically driven vehicle 1 . For example, the vehicle state sensor 23 includes a GPS (Global Positioning System) sensor, vehicle speed sensor, acceleration sensor, yaw rate sensor, steering angle sensor, and the like.

The microphone 24 detects sounds around the automatic driving vehicle 1 .

The communication device 30 communicates with the outside of the automatic driving vehicle 1 . For example, the communication device 30 communicates with the remote support system 100 .

The traveling device 40 includes a steering device, a driving device, and a braking device. The steering device steers the wheels of the automatic driving vehicle 1 . For example, the steering system includes a power steering (EPS: Electric Power Steering) system. A driving device is a power source that generates a driving force. An engine, an electric motor, an in-wheel motor, etc. are illustrated as a drive device. A braking device generates a braking force.

The control device 50 controls the automatic driving vehicle 1 . The control device 50 includes one or more processors 51 (hereinafter simply referred to as processors 51) and one or more storage devices 52 (hereinafter simply referred to as storage devices 52). The processor 51 executes various processes. For example, the processor 51 includes a CPU (Central Processing Unit). The storage device 52 stores various information. Examples of the storage device 52 include a volatile memory, a nonvolatile memory, a HDD (Hard Disk Drive), an SSD (Solid State Drive), and the like. Various processes by the control device 50 are realized by the processor 51 executing the automatic driving program 60, which is a computer program. The automatic driving program 60 is stored in the storage device 52 or recorded in a computer-readable recording medium. The control device 50 may include one or more ECUs (Electronic Control Units). A part of the control device 50 may be an information processing device outside the automatic driving vehicle 1 . In that case, part of the control device 50 communicates with the automatically driven vehicle 1 and remotely controls the automatically driven vehicle 1 .

3-2. Information Acquisition Processing The processor 51 acquires the driving environment information 70 indicating the driving environment of the automatically driven vehicle 1 . The driving environment information 70 is stored in the storage device 52 .

FIG. 7 is a block diagram showing an example of the driving environment information 70. As shown in FIG. The driving environment information 70 includes surrounding situation information 71 , vehicle state information 73 , voice information 74 and map information 75 .

The surrounding situation information 71 is information indicating the surrounding situation of the automatic driving vehicle 1 . The surrounding situation information 71 includes information obtained by the recognition sensor 21 . For example, the surrounding situation information 71 includes image information 72 indicating an image IMG captured by the camera 22 . As another example, the surrounding situation information 71 includes measurement information measured by LIDAR or radar.

In addition, the peripheral situation information 71 includes object information related to objects around the automatic driving vehicle 1 . Examples of objects around the automatic driving vehicle 1 include people (pedestrians, etc.), other vehicles (preceding vehicles, parked vehicles, etc.), signs, white lines, obstacles, and the like. The object information indicates the relative position and relative speed of the object with respect to the automatic driving vehicle 1 . For example, by analyzing the image information 72 obtained by the camera 22, an object can be identified and the relative position of the object can be calculated. It is also possible to identify an object and calculate the relative position and relative velocity of the object based on the information measured by LIDAR or radar.

The vehicle state information 73 is information indicating the position and state of the automatically driven vehicle 1 . The processor 51 acquires vehicle state information 73 from the detection result of the vehicle state sensor 23 .

Audio information 74 indicates the audio detected by microphone 24 .

The map information 75 indicates lane layout, road shape, and the like. The processor 51 acquires the map information of the required area from the map database. The map database may be stored in a predetermined storage device mounted on the automatic driving vehicle 1 or may be stored in a management server outside the automatic driving vehicle 1 . In the latter case, the processor 51 communicates with the management server to acquire the necessary map information.

3-3. Communication Processing The processor 51 communicates with the remote support system 100 via the communication device 30 . For example, the processor 51 transmits at least part of the driving environment information 70 to the remote assistance system 100 as required. In particular, the processor 51 transmits the image information 72 and the vehicle state information 73 to the remote support system 100 when remote support by the operator is required. Processor 51 also receives operator instructions INS from remote assistance system 100 .

3-4. Vehicle Driving Control The processor 51 executes “vehicle driving control” for controlling driving of the automatically driven vehicle 1 . Vehicle running control includes steering control, acceleration control, and deceleration control. Processor 51 executes vehicle travel control by controlling travel device 40 . Specifically, the processor 51 performs steering control by controlling the steering device. Processor 51 also performs acceleration control by controlling the drive device. The processor 51 also executes deceleration control by controlling the braking device.

The processor 51 also executes “automatic driving control” for controlling automatic driving of the autonomous vehicle 1 . Specifically, the processor 51 generates a driving plan for the automatically driven vehicle 1 based on the driving environment information 70, and generates a target trajectory for driving according to the driving plan. A target trajectory includes a target position and a target velocity. Then, the processor 51 executes vehicle travel control so that the automatically driven vehicle 1 follows the target trajectory.

When receiving an operator instruction INS from the remote support system 100, the processor 51 executes vehicle travel control according to the operator instruction INS.

3-5. Processing Related to Preliminary Notification Processing Processing related to preliminary notification processing by the automatic driving system 10 according to the present embodiment will be described below.

3-5-1. First Example FIG. 8 is a flow chart showing a first example of processing related to preliminary notification processing.

In step S10, the processor 51 performs a "preliminary notification condition determination process" for determining whether or not the preliminary notification condition is satisfied. The preliminary notification condition is a condition for executing preliminary notification processing. In a first example, the preliminary notification condition is that a horn is detected. Therefore, the processor 51 performs a "horn sound detection process" in step S11. The horn detection process detects a horn based on the voice information 74 obtained by the microphone 24, and determines whether or not the horn has been detected.

FIG. 9 is a flow chart showing an example of the horn detection process (step S11). In step S11a, processor 51 obtains audio information 74 indicative of the audio detected by microphone 24. FIG. In step S11b, the processor 51 performs filtering using a band-pass filter to extract sounds in specific frequency bands including frequencies of general horn sounds. The processor 51 may remove stationary noise such as pre-sampled vehicle running noise.

In step S11c, the processor 51 determines whether or not the sound pressure level of the sound in the specific frequency band is equal to or higher than the threshold. If the sound pressure level of the sound in the specific frequency band is less than the threshold (step S11c; No), the processor 51 determines that no alarm has been detected (step S11; No). On the other hand, if the sound pressure level of the sound in the specific frequency band is equal to or higher than the threshold (step S11c; Yes), the process proceeds to step S11d.

In step S11d, the processor 51 performs pattern analysis on the sound in the specific frequency band. Specifically, the processor 51 extracts a waveform similar to the sounding pattern of a general traffic whistle, and increases the amplitude and shapes the waveform. Furthermore, the processor 51 calculates the degree of similarity with the sounding pattern of a general traffic guidance horn. If the degree of similarity is greater than or equal to the threshold (step S11e; Yes), the processor 51 determines that the alarm has been detected (step S11; Yes). On the other hand, if the degree of similarity is less than the threshold (step S11e; No), the processor 51 determines that no horn has been detected (step S11; No).

If the preliminary notification condition is satisfied (step S10; Yes), the process proceeds to step S20. Otherwise (step S10; No), the process in this cycle ends.

In step S20, the processor 51 performs a "preliminary notification process". Specifically, the processor 51 communicates with the remote support system 100 via the communication device 30 and transmits the preliminary notification PRE to the remote support system 100 . In particular, the processor 51 transmits (uploads) the image information 72 obtained by the camera 22 to the remote support system 100 (step S21). Additionally, processor 51 may transmit vehicle status information 73 to remote assistance system 100 . This pre-notification processing enables the operator to perform remote support standby early.

3-5-2. Second Example FIG. 10 is a flow chart showing a second example of processing related to preliminary notification processing. Explanations that overlap with the first example will be omitted as appropriate. In a second example, the self-driving vehicle 1 comprises multiple cameras 22 and multiple microphones 24 as shown in FIG.

If the horn is detected (step S11; Yes), the process proceeds to step S12. In step S12, the processor 51 performs a "sounding direction estimation process" for estimating the sounding direction of the horn. The sounding direction estimation process is performed based on the sound detection results from the multiple microphones 24 , that is, the sound information 74 obtained from each of the multiple microphones 24 .

FIG. 11 is a timing chart for explaining the blowing direction estimation process (step S12). The vertical axis represents sound pressure and the horizontal axis represents time. FIG. 11 shows signal waveforms of voices detected by each of the plurality of microphones 24-1 to 24-6. This signal waveform is obtained after the processing of steps S11b and S11c in FIG. 9 has been performed. Based on the magnitude of the sound pressure and the rising time of the audio signal, it is possible to accurately estimate the blowing direction of the horn sound. In the example shown in FIG. 11, the direction of the microphone 24-2 from which an audio signal with the highest sound pressure and the fastest rising time is obtained is determined to be closest to the blowing direction of the horn.

In subsequent preliminary notification processing (step S20), the processor 51 selects an image IMG corresponding to the blowing direction from among the plurality of images IMG captured by the plurality of cameras 22, and sends only the selected image IMG to the remote support system 100. (step S22). In the example of FIG. 11, an image IMG captured by camera 22-2 corresponding to microphone 24-2 is selectively uploaded.

By selectively uploading the image IMG in the blowing direction, it is possible to reduce the amount of communication, the communication time, and the communication cost compared to uploading all the images IMG in a plurality of directions. Since the amount of communication is reduced, image quality degradation due to congestion control is less likely to occur. In addition, reduction in the number of images IMG to be uploaded at the same time means that image transmission with low delay becomes possible. That is, the high-quality image IMG is displayed on the display device 120 of the remote support system 100 with low delay. As a result, the visibility and real-time performance of the image IMG are sufficiently ensured, and the operator can correctly grasp the situation around the automatic driving vehicle 1 . Furthermore, since there is a high possibility that the traffic director TG is shown in the image IMG in the direction of the sound, the operator can efficiently grasp the state of the traffic director TG.

3-5-3. Third Example FIG. 12 is a flow chart showing a third example of processing related to preliminary notification processing. Explanations that overlap with the above first and second examples will be omitted as appropriate.

In a third example, the preliminary notification condition is "a horn is detected and a person is recognized in the direction of the horn". For this purpose, the processor 51 performs person recognition processing (step S13) following sounding direction estimation processing (step S12). More specifically, the processor 51 analyzes the image IMG corresponding to the blowing direction and determines whether or not the image IMG corresponding to the blowing direction shows a "person". At this stage, it is not necessary to ascertain whether or not the traffic director is shown. Techniques for person recognition (image analysis) are known. It should be noted that simple person recognition can be performed much more easily and accurately than recognition of a traffic director.

If a person appears in the image IMG corresponding to the sound direction (step S13; Yes), the preliminary notification condition is established (step S10; Yes). Then, the process proceeds to the preliminary notification process (step S20). On the other hand, if no person is shown in the image IMG corresponding to the sounding direction (step S13; No), the preliminary notification condition is not satisfied (step S10; No), and preliminary notification processing (step S20) is not performed.

As described above, according to the third example, preliminary notification processing is performed when a horn is detected and a person is recognized in the blowing direction of the horn. This is equivalent to improving the accuracy of the horn detection process (step S11). Since the detection accuracy of the warning whistle is improved, unnecessary execution of the preliminary notification process is suppressed. Moreover, the possibility that the traffic director TG is shown in the image IMG uploaded to the remote support system 100 is further increased. These things mean an improvement in the accuracy of preliminary notification processing.

3-6. Remote Assistance Request Processing Processing related to remote assistance request processing by the automatic driving system 10 according to the present embodiment will be described below. FIG. 13 is a flowchart illustrating an example of processing related to remote support request processing.

In step S<b>30 , the processor 51 performs “traffic guide recognition processing” for determining whether or not a traffic guide TG exists around the automatically driven vehicle 1 . More specifically, the processor 51 analyzes the image IMG indicated by the image information 72 and determines whether or not the traffic director TG is shown in the image IMG. If the blast direction estimation process (step S12) has already been performed, the processor 51 analyzes the image IMG corresponding to the blast direction and determines whether the traffic director TG is shown in the image IMG corresponding to the blast direction. It may be determined whether

FIG. 14 is a conceptual diagram for explaining the traffic guide recognition process (step S30). A traffic guide TG has a guide rod GS. The guide rod GS is considered as one of the appearance features of the traffic director TG. Additionally, the attire of the traffic director TG (safety vest, etc.) may be considered as an appearance feature. Guidance using the guide rod GS may also be considered as an operational feature. Based on whether or not such characteristics of the traffic director TG are detected, it is determined whether or not the traffic director TG is shown in the image IMG.

When the sound direction estimation process (step S12) has already been performed, the processor 51 may determine whether or not the traffic director TG is shown in the image IMG corresponding to the sound direction. As a result, the traffic guide recognition process can be performed efficiently. In other words, the recognition efficiency and recognition accuracy of the traffic guide TG are improved by considering the sound direction. This also reduces the processing load and processing time required for traffic guide recognition processing.

If it is determined that the traffic guide TG exists in the image IMG, that is, if the traffic guide TG is recognized (step S30; Yes), the process proceeds to step S40. Otherwise, the process proceeds to step S80.

In step S40, the processor 51 determines whether or not the automatic driving vehicle 1 can be automatically driven. That is, the processor 51 determines whether or not it is possible to respond to the guidance instruction from the traffic guide TG without remote assistance. For example, in order to follow a guidance instruction by a traffic guide TG, it is necessary to recognize the guidance instruction by analyzing the image IMG and understand its contents. If the guidance instruction by the traffic guide TG cannot be recognized or understood, the processor 51 determines that automatic driving is not possible (continuation of automatic driving is difficult).

If it is determined that automatic driving is not possible (step S40; No), the process proceeds to step S50. On the other hand, if it is determined that automatic driving is possible (step S40; Yes), the process proceeds to step S80.

In step S50, the processor 51 performs remote support request processing. Specifically, the processor 51 communicates with the remote support system 100 via the communication device 30 and transmits a remote support request REQ to the remote support system 100 . The processor 51 sends the operating environment information 70 required for remote assistance to the remote assistance system 100 together with the remote assistance request REQ. Driving environment information 70 required for remote assistance includes at least image information 72 obtained by camera 22 . Processor 51 may also send image information 72 for a period of time before the remote assistance request REQ is generated. In addition, the processor 51 may send necessary vehicle state information 73 and the like to the remote support system 100 .

At step S60 after step S50, the processor 51 receives an operator instruction INS from the remote support system 100 via the communication device 30. FIG. The operator instruction INS indicates the content of remote support by the operator.

In step S70, the processor 51 executes vehicle travel control according to the operator instruction INS.

In step S80, the processor 51 continues automatic operation control.

3-7. Processing after Passing The automatically driven vehicle 1 passes the position of the traffic director TG after responding to the guidance instruction from the traffic director TG. After that, the traffic director TG may sound the horn WH to guide the following vehicles. In order not to erroneously react to the sound of the horn to the following vehicle, the processor 51 may record the handling history information in the storage device 52 while handling the guidance instruction by the traffic guide TG. The coping history information includes positions where guidance instructions are coping with automatic driving or remote assistance. The countermeasure history information may include the sounding direction of the horn and/or the recognized position of the traffic director TG.

When a new horn sound is detected after handling the guidance instruction by the traffic guide TG, the processor 51 generates the new horn sound based on the handling history information, the vehicle position, and the direction of the new horn sound. is for the following vehicle. If the new horn is for a following vehicle, processor 51 does not perform preliminary notification processing. This prevents unnecessary preliminary notification processing.

4. Remote Support System FIG. 15 is a block diagram showing a configuration example of a remote support system 100 according to this embodiment. The remote support system 100 includes a communication device 110 , a display device 120 , an input device 130 and an information processing device 150 .

The communication device 110 communicates with the outside. For example, the communication device 110 communicates with the autonomous vehicle 1 .

The display device 120 displays various information. Examples of the display device 120 include a liquid crystal display, an organic EL display, a head-mounted display, a touch panel, and the like.
The input device 130 is an interface for receiving input from an operator. Examples of the input device 130 include a touch panel, keyboard, mouse, and the like. Further, when the remote support is remote driving, the input device 130 includes driving operation members for the operator to perform driving operations (steering, acceleration, and deceleration).

The information processing device 150 performs various types of information processing. The information processing device 150 includes one or more processors 151 (hereinafter simply referred to as processors 151) and one or more storage devices 152 (hereinafter simply referred to as storage devices 152). The processor 151 executes various processes. For example, processor 151 includes a CPU. The storage device 152 stores various information. Examples of the storage device 152 include volatile memory, nonvolatile memory, HDD, SSD, and the like. The functions of the information processing apparatus 150 are implemented by the processor 151 executing the remote support program 160, which is a computer program. Remote support program 160 is stored in storage device 152 . The remote assistance program may be recorded on a computer-readable recording medium. A remote assistance program may be provided via a network.

The vehicle information 170 is information sent from the automatic driving vehicle 1 and is information necessary for remote support. Vehicle information 170 includes a portion of driving environment information 70 . More specifically, vehicle information 170 includes at least image information 72 obtained by camera 22 . The vehicle information 170 may include not only the latest image information 72 but also the image information 72 for a certain period of time before the remote assistance request REQ is generated. Vehicle information 170 may include vehicle state information 73 . The processor 151 receives vehicle information 170 from the automatic driving vehicle 1 via the communication device 110 . Vehicle information 170 is stored in storage device 152 .

FIG. 16 is a flow chart showing processing by remote support system 100 (processor 151) according to the present embodiment.

In step S<b>100 , the processor 151 determines whether or not the preliminary notification PRE has been received from the autonomous vehicle 1 . If the preliminary notification PRE has not been received (step S100; No), the processing in this cycle ends. On the other hand, if the preliminary notification PRE is received (step S100; Yes), the process proceeds to step S110.

At step 110, the processor 151 performs standby processing. Specifically, the processor 151 displays the preliminary notification PRE received from the automatic driving vehicle 1 on the display device 120 . As a result, the operator can anticipate the occurrence of the remote support request REQ and perform standby for remote support. In particular, the processor 151 displays the image IMG indicated by the image information 72 received from the autonomous vehicle 1 on the display device 120 . By viewing the image IMG, the operator can grasp the situation around the automatic driving vehicle 1 .

In step S<b>120 , the processor 151 determines whether or not a remote assistance request REQ has been received from the autonomous vehicle 1 . If the remote support request REQ has not been received (step S120; No), the process proceeds to step S130. If a certain period of time has passed without receiving a remote support request REQ after receiving the preliminary notification PRE (step S130; Yes), the processor 151 ends the standby process (step S140). On the other hand, if the remote support request REQ is received within a certain period of time after receiving the preliminary notification PRE (step S120; Yes), the process proceeds to step S150.

In step S150, the processor 151 performs "information providing processing". Specifically, the processor 151 presents the vehicle information 170 to the operator by displaying the vehicle information 170 on the display device 120 .

The operator refers to the vehicle information 170 displayed on the display device 120 and remotely supports the autonomous vehicle 1 . Remote support by an operator includes recognition support, judgment support, remote operation, and the like. The operator uses the input device 130 to input an operator instruction INS indicating the content of remote support. Since the operator is already on standby by the preliminary notification process and the standby process described above, the operator can quickly input the operator instruction INS. Processor 151 receives an operator instruction INS via input device 130 (step S160).

If the traffic guide TG is not shown in the image IMG displayed on the display device 120, the operator may input an operator instruction INS instructing switching of the image IMG to be uploaded.

In step S170, the processor 151 performs "operator instruction notification processing". Specifically, the processor 151 communicates with the automatically driven vehicle 1 via the communication device 110 and notifies the automatically driven vehicle 1 of the operator instruction INS. The automatic driving system 10 performs vehicle travel control according to operator instructions INS.

In step S180, the processor 151 determines whether or not the remote support end condition is satisfied. Typically, the operator decides to end remote assistance. If the remote support end condition is satisfied (step S180; Yes), the process proceeds to step S190. Otherwise (step S180; No), the process returns to step S150.

In step S190, the processor 151 terminates the remote support processing (information provision processing and operator instruction notification processing). The automatic driving system 10 also finishes uploading the driving environment information 70 required for remote assistance.

1 Autonomous Driving Vehicle 10 Autonomous Driving System 20 Sensor Group 22 Camera 24 Microphone 30 Communication Device 40 Running Device 50 Control Device 51 Processor 52 Storage Device 60 Autonomous Driving Program 70 Driving Environment Information 72 Image Information 74 Audio Information 100 Remote Support System 110 Communication Device 120 display device 130 input device 150 information processing device 160 remote support program 170 vehicle information INS operator instruction PRE preliminary notification REQ remote support request TG traffic guide WH horn

Claims (1)

A vehicle control method for controlling an autonomous vehicle comprising at least one camera and at least one microphone, comprising:
an image acquisition process for acquiring an image of the surroundings of the automated driving vehicle using the at least one camera;
A horn sound detection process for detecting sounds around the automated driving vehicle with the at least one microphone and detecting a horn sound based on the sound detection result;
and a preliminary notification process of transmitting the image to the remote support system before transmitting a remote support request to the remote support system that provides remote support for the autonomous vehicle in response to detecting the horn sound. Vehicle control method.

Images