JP2019031264A - Information processing method, information processing system, and program - Google Patents

Abstract

To provide an information processing method that allows a person to grasp whether or not recognition processing on driving a vehicle is properly executed.SOLUTION: An information processing method: acquires event-related information for detecting a predetermined event on a vehicle by using a processor; determines whether or not the predetermined event is detected by using the event-related information; acquires direction information indicating a first direction when it is determined that the predetermined event is detected; acquires first recognition result information obtained by recognition processing by the vehicle on an object located outside the vehicle; determines whether or not the first recognition result information includes an object recognition result in the first direction; determines whether or not a specific object is present in the first direction by using the first recognition result information; and creates and outputs processed information indicating that the vehicle has already executed recognition processing for the first direction when it is determined that the specific object is not present in the first direction, and it is also determined that the first recognition result information includes the object recognition result in the first direction.SELECTED DRAWING: Figure 4

Description

  The present disclosure relates to an information processing method, an information processing system, and a program that communicate with a mobile object such as a vehicle that is automatically driven.

  In recent years, a technique related to a mobile body such as an automatically driving vehicle (hereinafter referred to as an automatically driving vehicle) has been developed. In addition, there is a technique for notifying information related to driving control of an autonomous driving vehicle to a passenger boarding the autonomous driving vehicle (see, for example, Patent Document 1 and Patent Document 2).

  In Patent Literature 1, the passenger is associated with setting information related to travel control predetermined for the autonomous driving vehicle based on an instruction from the passenger of the autonomous driving vehicle, the content of the traveling control of the autonomous driving vehicle, and the like. The technology presented in is disclosed.

  Patent Document 2 discloses a technique for presenting the contents of travel control that can be executed by an autonomous driving vehicle to a passenger of the autonomous driving vehicle.

JP 2017-144110 A Japanese Patent Laid-Open No. 2016-216029

  However, in the prior art, a person can grasp the content of the travel control, but it is difficult to grasp the content of the recognition process by the vehicle up to the travel control. For example, there is a problem that an evaluator who evaluates automatic driving of an autonomous driving vehicle cannot grasp whether or not the autonomous driving vehicle actually confirms a predetermined item to be confirmed.

  Therefore, the present disclosure provides an information processing method that enables a person to grasp whether or not recognition processing related to driving of a vehicle is appropriately performed.

  An information processing method according to an aspect of the present disclosure uses a processor to acquire event-related information for detecting a predetermined event for a vehicle, and whether the predetermined event is detected using the event-related information. And determining that the predetermined event has been detected, obtaining direction information indicating a first direction associated with the predetermined event, and performing recognition processing by the vehicle of an object located outside the vehicle. Obtained first recognition result information is obtained, it is determined whether or not the first recognition result information includes an object recognition result in the first direction, and the first direction is determined using the first recognition result information. It is determined whether or not there is a specific object that may affect the traveling of the vehicle, and it is determined that the specific object does not exist in the first direction, and the first When it is determined that the recognition result information includes the object recognition result in the first direction, the processed information indicating that the vehicle has recognized the first direction is generated, and the generated processed information is generated. Output information.

  Note that these comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and the system, method, integrated circuit, and computer program. And any combination of recording media.

  In the information processing method according to an aspect of the present disclosure, a person can grasp whether or not recognition processing related to driving of a vehicle is appropriately performed.

FIG. 1 is a diagram for explaining a configuration of a system including an information processing system according to an embodiment. FIG. 2 is a block diagram illustrating a functional configuration of the information processing system according to the embodiment. FIG. 3 is a block diagram illustrating a functional configuration of the autonomous driving vehicle that communicates with the information processing system according to the embodiment. FIG. 4 is a flowchart illustrating a processing procedure executed by the information processing system according to the embodiment. FIG. 5 is a flowchart illustrating a processing procedure executed by the autonomous driving vehicle communicating with the information processing system in the embodiment. FIG. 6A is a diagram illustrating an example of an image including an autonomous driving vehicle that communicates with the information processing system in the embodiment. FIG. 6B is a diagram for describing an example of an automatic driving vehicle that communicates with the information processing system in the embodiment and a distance from the automatic driving vehicle to an object. FIG. 7 is a diagram illustrating an example of event-related information. FIG. 8A is a diagram illustrating an example of map information and route information. FIG. 8B is a diagram for explaining each point of the map information shown in FIG. 8A. FIG. 9A is a diagram illustrating a first example of an image displayed on the information processing system in the embodiment. FIG. 9B is a diagram illustrating a second example of an image displayed on the information processing system in the embodiment. FIG. 10 is a diagram illustrating a third example of an image displayed on the information processing system in the embodiment. FIG. 11A is a diagram illustrating a first example of display that highlights an obstacle displayed on the information processing system in the embodiment. FIG. 11B is a diagram illustrating a second example of display that highlights an obstacle displayed on the information processing system in the embodiment. FIG. 12 is a diagram illustrating a fourth example of an image displayed on the information processing system in the embodiment. FIG. 13 is a diagram illustrating a fifth example of an image displayed on the information processing system in the embodiment.

(Knowledge that became the basis of this disclosure)
As described above, the present disclosure has found that the following problems occur in the prior art.

  In recent years, there is a technology related to automatic driving in a moving body such as a vehicle. Hereinafter, the mobile body that performs automatic driving is referred to as an automatic driving vehicle.

  Autonomous vehicles, such as safety checks performed by drivers who drive non-automatically driven vehicles when performing start, stop, left turn, right turn, etc., control the surrounding environment of an autonomously driven vehicle, etc. And using the acquired information, traveling control is performed.

  Here, before the autonomous driving vehicle travels on the public road, it is necessary to evaluate whether the autonomous driving vehicle can safely travel on the public road. For example, an evaluator evaluates in advance whether or not an automatic driving vehicle can be appropriately automatically driven.

  Therefore, in order to evaluate whether or not an autonomous driving vehicle can travel safely, it is considered that an evaluator evaluates the automatic driving of an autonomous driving vehicle by automatically driving a test course such as a driving school.

  Here, when the evaluator evaluates the automatic driving of the autonomous driving vehicle, it is desirable that the evaluation can be performed in the same manner as when the driving of the manual driving is evaluated.

  At this time, even if the autonomous driving vehicle is able to automatically drive appropriately, it is determined whether or not all predetermined items to be confirmed such as the environment around the autonomous driving vehicle are confirmed. I can't. Therefore, there is a problem that automatic driving may be performed without checking all predetermined items to be checked, such as the environment around the autonomous driving vehicle.

  Therefore, it is conceivable that the evaluator displays the result of the automatic driving vehicle recognizing an object around the automatic driving vehicle on a display or the like. However, when all the recognition results of objects around the autonomous driving vehicle are displayed, for example, objects far away, static objects such as guardrails that do not require safety confirmation, and the like are displayed. In this case, the evaluator has a problem that it is difficult to evaluate whether or not the autonomous driving vehicle appropriately confirms safety, that is, whether or not all items to be confirmed that are appropriately determined in advance are confirmed.

  In order to solve such a problem, an information processing method according to one aspect of the present disclosure uses a processor to acquire event-related information for detecting a predetermined event about a vehicle, and uses the event-related information. Determining whether or not the predetermined event has been detected. If it is determined that the predetermined event has been detected, direction information indicating a first direction associated with the predetermined event is acquired and positioned outside the vehicle. First recognition result information obtained by recognition processing by the vehicle of the object to be obtained is determined, whether or not the first recognition result information includes the object recognition result in the first direction, and the first recognition Using the result information, it is determined whether or not there is a specific object that may affect the travel of the vehicle in the first direction, and it is determined that the specific object does not exist in the first direction. And when it is determined that the first recognition result information includes the object recognition result in the first direction, the vehicle has been processed to indicate that the first direction has been recognized. Information is generated, and the generated processed information is output.

  In order to solve such a problem, a program according to an aspect of the present disclosure causes a computer to execute the information processing method.

  According to such a method, it is possible for a person to grasp whether or not recognition processing relating to driving of the vehicle is appropriately performed. For example, an evaluator who evaluates whether or not automatic driving of an autonomous driving vehicle is appropriately performed can determine and output whether or not recognition processing related to driving of the autonomous driving vehicle is appropriately performed. As a result, the evaluator can check whether or not the automatic driving vehicle has been properly confirmed for safety at a point where safety confirmation is required in addition to the normal automatic driving control for each predetermined event.

  In addition, according to such a method, since the direction for confirming safety is set in advance for each predetermined event, the safety is confirmed by the autonomous driving vehicle in the direction in which safety confirmation is particularly necessary. It is possible to evaluate appropriately.

  In addition, even if a specific object including an obstacle that is an obstacle to automatic driving is not recognized by the automatic driving vehicle, the evaluator should use the automatic driving vehicle to display information indicating that. It is possible to grasp whether or not safety confirmation has been made. That is, the evaluator can easily grasp whether or not recognition processing for automatic driving is appropriately performed by the automatic driving vehicle.

  Moreover, according to such a method, when the autonomous driving vehicle performs the driving control of the automatic driving, only the information used for the determination with respect to the predetermined event is output. Can be reduced. Therefore, the output process can be efficiently speeded up.

  For example, in the information processing method according to an aspect of the present disclosure, when it is determined that the specific object exists in the first direction, the first information indicating that the specific object is recognized in the first direction Including the second information indicating that the specific object is not recognized in the first direction when it is determined that the specific object does not exist in the first direction. Processed information may be generated.

  According to such a method, it is possible to more appropriately evaluate whether or not recognition processing for automatic driving is appropriately performed by the automatic driving vehicle.

  Further, for example, the first recognition result information may be information acquired by performing the recognition process on an image obtained by photographing a predetermined range including the first direction from the vehicle.

  In other words, such a method is not applied to an automatic driving system that acquires information from a camera installed on a public road or the like, but is applied to an automatic driving vehicle that performs traveling control using a camera mounted on the automatic driving vehicle. Can be done. Therefore, such a method is particularly effective for an autonomous driving vehicle controlled using an apparatus such as a camera provided in the autonomous driving vehicle, not an apparatus arranged on a public road or the like.

  In addition, for example, the information processing method according to an aspect of the present disclosure further acquires travel information of the vehicle including at least one of a steering angle and a vehicle speed of the vehicle, and the predetermined range is the travel of the vehicle It may be determined according to information.

  According to such a method, the evaluator can more easily grasp whether or not recognition processing for automatic driving is appropriately performed by the automatic driving vehicle according to the traveling state of the automatic driving vehicle. .

  Further, for example, the information processing method according to one aspect of the present disclosure may further output the acquired travel information of the vehicle together with the processed information.

  According to such a method, it is possible for the evaluator to easily understand whether or not recognition processing for automatic driving is appropriately performed by the automatic driving vehicle.

  Further, for example, when the predetermined event is detected, the processed information may be repeatedly output until the detected predetermined event is not detected or a predetermined time elapses.

  According to such a method, the evaluator always recognizes the automatic driving vehicle for automatic driving while maintaining the changed driving control state after the automatic driving vehicle changes the driving control. It is possible for the evaluator to easily grasp whether or not he / she is doing.

  In addition, for example, in the information processing method according to one aspect of the present disclosure, when it is determined that the specific object exists in the first direction, the vehicle determines or changes control of traveling due to the specific object. The first information including a determination result of whether or not the vehicle has determined or changed control of traveling may be generated.

  According to such a method, it is possible for the evaluator to easily understand whether or not recognition processing for automatic driving is appropriately performed by the automatic driving vehicle.

  In addition, for example, the information processing method according to one aspect of the present disclosure further uses the first recognition result information to determine whether or not the specific object exists in a second direction different from the first direction. When the determination is made and it is determined that the specific object exists in the second direction, the processed information including information indicating the second direction may be generated.

  The driving control of automatic driving is processed from highly urgent information such as a specific object closest to the autonomous driving vehicle that needs to stop driving immediately. In addition, processing of information that affects travel control is executed even for information with low urgency, such as an object away from an autonomous driving vehicle, which does not need to stop traveling immediately. Therefore, it is necessary for the processor to determine whether or not a specific object such as an obstacle that affects the traveling of the autonomous driving vehicle exists in a direction other than the direction of high urgency, and to change the control content of the traveling. Also outputs information that has sex. By doing so, the evaluator can more appropriately grasp whether or not the automatic driving recognition process is appropriately performed by the automatic driving vehicle.

  In addition, for example, in the information processing method according to one aspect of the present disclosure, an image generated by being photographed in the first direction by a photographing device mounted on the vehicle is acquired, and the acquired image is acquired. You may output with the said processed information in the aspect according to the said predetermined event detected.

  According to such a method, it is possible to make the evaluator grasp the information processed by the autonomous driving vehicle more appropriately.

  For example, in the information processing method according to one aspect of the present disclosure, when it is determined that the first recognition result information does not include the object recognition result in the first direction, the vehicle is Unprocessed information indicating that the direction is not recognized may be generated, and the generated unprocessed information may be output.

  According to such a method, the evaluator can appropriately recognize that the vehicle is not properly automatically driven by the autonomous driving vehicle.

  Moreover, the information processing system which concerns on 1 aspect of this indication acquires the event relevant information for detecting the predetermined event about a vehicle, and determines whether the said predetermined event was detected using the said event relevant information An event determination unit, a direction information acquisition unit that acquires direction information indicating a first direction associated with the predetermined event when the event determination unit determines that the predetermined event has been detected, and an outside of the vehicle A vehicle processing information acquisition unit that acquires first recognition result information obtained by recognition processing by the vehicle of a positioned object, and determines whether or not the first recognition result information includes an object recognition result in the first direction And using the first recognition result information to determine whether or not there is a specific object that may affect the traveling of the vehicle in the first direction. When it is determined that the specific object does not exist by the determination unit and the vehicle recognition determination unit, and when it is determined that the first recognition result includes the object recognition result in the first direction, A display information generation unit that generates processed information indicating that the vehicle has recognized the first direction, and a display control unit that displays the processed information generated by the display information generation unit on a display device And comprising.

  According to such a configuration, it is possible for a person to grasp whether or not recognition processing relating to driving of the vehicle is appropriately performed. For example, an evaluator who evaluates whether or not automatic driving of an autonomous driving vehicle is appropriately performed can determine and output whether or not recognition processing related to driving of the autonomous driving vehicle is appropriately performed. As a result, the evaluator can check whether or not the automatic driving vehicle can be appropriately recognized at a point where safety confirmation is required in addition to the normal automatic driving control for each predetermined event.

  In addition, according to such a configuration, the direction for safety confirmation (that is, the confirmation direction) is set in advance for each predetermined event. It is possible to appropriately evaluate whether the recognition process has been performed.

  In addition, even when a specific object including an obstacle that is an obstacle to automatic driving is not recognized by the automatic driving vehicle, the evaluator should appropriately use the automatic driving vehicle to display information indicating that. It is possible to grasp whether or not the recognition process has been performed. That is, the evaluator can easily grasp whether or not recognition processing for automatic driving is appropriately performed by the automatic driving vehicle.

  Further, according to such a configuration, when the autonomous driving vehicle performs the driving control of the automatic driving, only the information used for the determination with respect to the predetermined event is output. Therefore, the processing amount for performing the output process Can be reduced. Therefore, the output process can be efficiently speeded up.

  Note that these comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and the system, method, integrated circuit, and computer program. Alternatively, it may be realized by any combination of recording media.

  Hereinafter, embodiments of the present disclosure will be specifically described with reference to the drawings.

  It should be noted that each of the embodiments described below shows a comprehensive or specific example of the present disclosure. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements.

(Embodiment)
[Overview]
First, an overview of an information processing system according to an embodiment of the present disclosure will be described.

  FIG. 1 is a diagram for explaining a configuration of a system including an information processing system according to an embodiment.

  FIG. 1 illustrates an autonomous driving vehicle 2, a terminal 1 that wirelessly communicates with the autonomous driving vehicle 2, and a user U who operates the terminal 1.

  The terminal 1 is an example of an information processing system according to an embodiment of the present disclosure, and is a communication device that communicates with the autonomous driving vehicle 2 and receives information related to traveling control of the autonomous driving vehicle 2 from the autonomous driving vehicle 2. is there.

  The autonomous driving vehicle 2 is a moving body that autonomously drives based on a control program that is arbitrarily determined in advance, that is, an autonomous driving. FIG. 1 illustrates an automobile (that is, a vehicle) as an example of the autonomous driving vehicle 2. Note that the autonomous driving vehicle 2 may be any mobile body that is automatically driven, and is not particularly limited to a motorcycle, a three-wheeled vehicle, or the like.

  The user U is an evaluator who uses the terminal 1 to evaluate whether or not the autonomous driving vehicle 2 can be appropriately automatically driven. For example, the user U gets on the autonomous driving vehicle 2 and evaluates the operation of the autonomous driving vehicle 2.

  In FIG. 1, the autonomous driving vehicle 2 and the terminal 1 are illustrated as separate bodies. However, when the user U gets on the autonomous driving vehicle 2 and evaluates the operation of the autonomous driving vehicle 2, the autonomous driving vehicle 2 and the terminal 1 are automatically driven. The vehicle 2 and the terminal 1 may be integrally formed. For example, the terminal 1 may be connected to the autonomous driving vehicle 2 via a communication line so as to perform wired communication, and may be disposed in the autonomous driving vehicle 2. Further, the user U may not board the autonomous driving vehicle 2.

  In the following description, an information processing system according to an embodiment of the present disclosure will be described as a terminal 1. Moreover, in the following description, the case where the user U uses the terminal 1 and uses it in the autonomous driving vehicle 2 will be described.

[Constitution]
Next, the configuration of the terminal 1 according to the embodiment of the present disclosure will be described.

  FIG. 2 is a block diagram illustrating a functional configuration of the terminal 1 which is an example of the information processing system according to the embodiment.

  The terminal 1 is a communication device used for evaluating whether or not the autonomous driving vehicle 2 is appropriately automatically driven. The terminal 1 is a communication device that can communicate with the autonomous driving vehicle 2 such as a smartphone or a tablet terminal. Note that the terminal 1 may be a dedicated computer or a server device for realizing the information processing system according to the embodiment of the present disclosure. In that case, the display unit 12 and the position measurement unit 13 of the terminal 1 are provided in a terminal possessed by the user U, and the terminal 1 and the terminal possessed by the user U exchange data with each other via communication.

  The terminal 1 includes a communication unit 11, a display unit 12, a position measurement unit 13, a control unit 14, and a storage unit 15.

  The communication unit 11 is a communication interface such as a communication adapter that communicates with the autonomous driving vehicle 2.

  The display unit 12 displays information received from the autonomous driving vehicle 2, an image generated by the control unit 14, and the like. The display unit 12 is a display, for example, and is a touch panel display in the present embodiment.

  The position measuring unit 13 measures the position of the terminal 1. The position measuring unit 13 is realized by, for example, a GPS (Global Positioning System) receiver and acquires the position information of the terminal 1.

  Based on the information received from the autonomous driving vehicle 2, the control unit 14 generates information for allowing the user U to evaluate whether or not the autonomous driving vehicle 2 is appropriately automatically driven and outputs the information to the display unit 12. . The control unit 14 is realized by a CPU (Central Processing Unit), for example. Functionally, the control unit 14 includes a communication control unit 141, a display information generation unit 142, a display control unit 143, a vehicle processing information acquisition unit 144, a vehicle recognition determination unit 145, an event determination unit 146, A direction information acquisition unit 147.

  The communication control unit 141 controls communication between the terminal 1 and the autonomous driving vehicle 2 by controlling the communication unit 11.

  The event determination unit 146 acquires event related information 152 for detecting a predetermined event for the autonomous driving vehicle 2, and determines whether a predetermined event is detected using the acquired event related information 152. Specifically, the event determination unit 146 uses the event related information 152 to determine whether or not a predetermined event has occurred in the autonomous driving vehicle 2. The event determination unit 146 determines, for example, whether or not a predetermined event has occurred in the autonomous driving vehicle 2 from the position measured by the position measurement unit 13, the event related information 152, the map information 153, or the route information. Here, the predetermined event is an event related to the operation of the autonomous driving vehicle 2 or the position of the autonomous driving vehicle 2. Specifically, the predetermined event is that the automatic driving vehicle 2 changes traveling control or the automatic driving vehicle 2 moves to a predetermined location as shown in FIG. 7 described later. For example, changing the driving control of the autonomous driving vehicle 2 includes starting, stopping, turning right, turning left, etc., and moving the autonomous driving vehicle 2 to a predetermined location includes entering an intersection.

  The direction information acquisition unit 147 uses the event related information 152 to detect the predetermined event when the event determination unit 146 determines that the predetermined event has occurred in the autonomous driving vehicle 2, that is, when the predetermined event is detected. Direction information indicating the confirmation direction associated with is acquired. The confirmation direction is an example of the first direction.

  The vehicle processing information acquisition unit 144 acquires information received from the autonomous driving vehicle 2 by the communication unit 11. Specifically, the vehicle processing information acquisition unit 144 acquires recognition result information (first recognition result information) obtained by recognition processing by the automatic driving vehicle 2 for an object located outside the automatic driving vehicle 2. The recognition result information is information that can determine whether or not an object recognition process has been performed. Specifically, the recognition result information includes whether or not the object recognition process is executed, whether or not the object recognition process is successful, or whether or not there is information on the recognized object. The information of the recognized object includes the type of the object, the shape of the object or a rectangle surrounding the object, or the direction or distance from the photographing unit 22 (see FIG. 3) or the autonomous driving vehicle 2 to the object. The recognition result information is, for example, information acquired by the automatic driving vehicle 2 performing recognition processing on shooting information (that is, an image) obtained by shooting a predetermined range including the confirmation direction from the automatic driving vehicle 2. is there. According to such a configuration, for example, not an automatic driving system that acquires information from a camera installed on a public road or the like, but an automatic driving vehicle that performs traveling control using a camera mounted on an automatic driving vehicle. Can be done.

  For example, the predetermined range may be determined according to the travel information acquired by acquiring travel information of the automatic drive vehicle 2 including at least one of the steering angle and the vehicle speed of the automatic drive vehicle 2. According to such a method, according to the traveling state of the autonomous driving vehicle 2, the user U can more easily grasp whether or not recognition processing for automatic driving is appropriately performed by the autonomous driving vehicle 2. Can do.

  For example, when a predetermined event is detected, the event determination unit 146 transmits a signal requesting travel information including vehicle speed to the autonomous driving vehicle 2 via the communication unit 11. In addition, the event determination unit 146 determines a predetermined range based on the vehicle speed included in the travel information received from the autonomous driving vehicle 2, and outputs a signal requesting shooting information in the confirmation direction including the determined predetermined range. To the autonomous driving vehicle 2.

  The vehicle recognition determination unit 145 determines the recognition process performed by the autonomous driving vehicle 2 in the confirmation direction using the recognition result information acquired by the vehicle processing information acquisition unit 144. Specifically, the vehicle recognition determination unit 145 determines whether or not the recognition result information acquired by the vehicle processing information acquisition unit 144 includes an object recognition result in the confirmation direction. For example, the vehicle recognition determination unit 145 recognizes whether the object recognition process has been executed in the confirmation direction, whether the object recognition process has been normally executed in the confirmation direction, or the recognized object information corresponding to the confirmation direction. It is determined whether or not it exists in the processing range. For example, when the recognition process is an image recognition process, the vehicle recognition determination unit 145 may determine whether or not to execute the process for the range on the image corresponding to the confirmation direction from the operation log of the object recognition process. Further, the vehicle recognition determination unit 145 may determine whether or not there is a region where the recognition has failed within the range on the image corresponding to the confirmation direction. Further, the vehicle recognition determination unit 145 may determine whether or not there is a rectangle surrounding the recognized object located within the range on the image corresponding to the confirmation direction.

  Further, the vehicle recognition determination unit 145 determines whether or not there is a specific object that may affect the traveling of the autonomous driving vehicle 2 in the confirmation direction using the recognition result information. Specifically, the vehicle recognition determination unit 145 determines whether or not the specific object exists in the confirmation direction by using the machine learning model learned about the recognition of the specific object. For example, the machine learning model may be a learning model using a neural network such as deep learning, or may be a learning model using rule information such as a dictionary. Further, the vehicle recognition determination unit 145 determines whether or not an object exists in the confirmation direction using a machine learning model learned about recognition of a general object including an object other than the specific object, and the specific object described later The information 154 may be used to determine whether or not the object determined to exist is a specific object. Below, the example in which the process of the vehicle recognition determination part 145 is performed using this system is demonstrated. Here, the specific object may be arbitrarily determined. For example, the specific object may be an obstacle such as a pedestrian or an oncoming vehicle, which may be an obstacle to driving, and is confirmed for safely traveling such as a traffic light. It may be a target to be.

  The display information generation unit 142 generates information to be displayed on the display unit 12. For example, the display information generation unit 142 determines that the specific object does not exist in the confirmation direction by the vehicle recognition determination unit 145 and determines that the recognition result information includes the object recognition result in the confirmation direction. In the case of the automatic driving vehicle 2, processed information indicating that the autonomous driving vehicle 2 has already recognized the confirmation direction is generated.

  The display control unit 143 outputs the processed information (specifically, an image) generated by the display information generation unit 142 to the display unit 12 and causes the display unit 12 to display the information. For example, the display control unit 143 outputs the processed information generated by the display information generation unit 142 to the display unit 12. The user U evaluates whether or not the autonomous driving vehicle 2 is appropriately performing recognition processing for automatic driving by checking the processed information displayed on the display unit 12.

  Further, for example, when the event determination unit 146 acquires travel information of the autonomous driving vehicle 2, the travel information may be output to the display unit 12 and displayed on the display unit 12. According to such a method, the user U can easily understand whether or not the recognition process for the automatic driving is appropriately performed by the automatic driving vehicle 2.

  Further, for example, when a predetermined event is detected by the event determination unit 146, the display control unit 143 continues to output the processed information repeatedly to the display unit 12 until the detected predetermined event is not detected. The processed information may be continuously displayed on the unit 12. For example, when the autonomous driving vehicle 2 makes a left turn, the display control unit 143 starts turning left, that is, after the event determination unit 146 detects a predetermined event until the left turn ends, that is, the event determination unit 146 The processed information is continuously output to the display unit 12 until the predetermined event is not detected. For example, the event determination unit 146 may determine that the predetermined event has ended based on the position information measured by the position measurement unit 13, or the travel distance after the autonomous driving vehicle 2 starts traveling, Alternatively, the determination may be made based on whether or not the traveling time satisfies a predetermined threshold. Thus, the user U always performs recognition processing for automatic driving while the automatic driving vehicle 2 has changed the driving control and continues to maintain the changed driving control state. The user U can easily grasp whether or not there is.

  The display information generation unit 142 further includes processed information including first information indicating that the specific object is recognized in the confirmation direction when the vehicle recognition determination unit 145 determines that the specific object exists in the confirmation direction. May be generated. On the other hand, if it is determined that the specific object does not exist in the confirmation direction, the display information generation unit 142 may generate processed information including the second information indicating that the specific object has not been recognized in the confirmation direction. Good. As described above, the display information generation unit 142 generates different information depending on the presence or absence of a specific object, and causes the display unit 12 to display the generated information using the display control unit 143. By doing so, it is possible to more appropriately evaluate whether or not recognition processing for automatic driving is appropriately performed by the automatic driving vehicle 2.

  In addition, when the vehicle recognition determination unit 145 determines that the specific object exists in the confirmation direction, the event determination unit 146 determines whether or not the autonomous driving vehicle 2 has determined or changed the driving control due to the specific object. May be. For example, when a predetermined event is detected, the event determination unit 146 transmits a signal requesting traveling information to the autonomous driving vehicle 2 via the communication unit 11. Further, the event determination unit 146 determines whether or not the automatic driving vehicle 2 is executing a driving based on the event related information 152 based on the driving information received from the automatic driving vehicle 2 and the event related information 152. In this case, the display information generation unit 142 generates information including a determination result as to whether or not the autonomous driving vehicle 2 has changed the driving control from the driving control content based on the event related information 152, and displays the information on the display unit 12. Display. Thereby, it can be made to grasp | ascertain easily by the user U whether the recognition process for automatic driving | running | working is appropriately performed by the self-driving vehicle 2. FIG.

  Note that the automatic driving vehicle 2 may be shooting not only the confirmation direction based on the event-related information 152 but also a plurality of directions around the automatic driving vehicle 2. In some cases, an object recognition process is performed on an image generated by shooting in each of a plurality of directions. In this case, the terminal 1 may receive an image taken in a direction different from the confirmation direction. In this case, for example, the vehicle processing information acquisition unit 144 further recognizes by recognition processing of an object by the autonomous driving vehicle 2 located outside the autonomous driving vehicle in a direction (second direction) different from the confirmation direction. Result information (second recognition result information) may be acquired. Moreover, the vehicle recognition determination part 145 may determine the recognition process performed by the automatic driving vehicle 2 about a 2nd direction using 2nd recognition result information. Further, when the vehicle recognition determination unit 145 determines that an obstacle exists in the second direction, the display information generation unit 142 generates processed information including information indicating the second direction in which the obstacle exists. Also good. The display control unit 143 may cause the display unit 12 to display the processed information generated by the display information generation unit 142. Note that the plurality of directions described above may be included in one image. For example, the confirmation direction and the second direction may be different directions on the same image. In addition, the image that becomes the input of the recognition process may be a panoramic image obtained by shooting with a camera having a wide shooting field of view such as an omnidirectional camera.

  The driving control for the automatic driving of the autonomous driving vehicle 2 is processed from highly urgent information such as a specific object closest to the autonomous driving vehicle 2 that needs to be stopped immediately. In addition, processing of information that affects traveling control is executed even for information with low urgency, such as an object away from the autonomous driving vehicle 2 that does not need to stop traveling immediately. Therefore, it is necessary to determine whether there is a specific object such as an obstacle that affects the traveling of the autonomous driving vehicle 2 in a direction other than the direction of high urgency, and to change the control content of the traveling. Some information is also output. In this way, the user U can more appropriately grasp whether or not the automatic driving recognition process is appropriately performed by the automatic driving vehicle 2.

  Moreover, the autonomous driving vehicle 2 may photograph the same direction (for example, a confirmation direction) from different positions. For example, the automatic driving vehicle 2 may have a camera attached to each of the traveling direction side of the autonomous driving vehicle 2 and the rear side opposite to the traveling direction. In such a case, for example, when photographing the left direction of the autonomous driving vehicle 2, the autonomous driving vehicle 2 may photograph the left direction with cameras attached to the traveling direction side and the rear side, respectively. In such a case, the display information generation unit 142 may provide at least one image selected from images obtained from a plurality of cameras to the display control unit 143 together with the processed information based on the recognition result information. That is, the vehicle processing information acquisition unit 144 may acquire a plurality of images generated by photographing the confirmation directions from a plurality of positions by the autonomous driving vehicle 2. In this case, the display information generation unit 142 displays at least one image selected from the plurality of images acquired by the vehicle processing information acquisition unit 144 based on the recognition result information, together with the generated processed information, the display control unit 143. May be output to the display unit 12. For example, based on the recognition result information, the display information generation unit 142 may output an image including a recognized object among a plurality of images to the display control unit 143 together with the processed information. The display control unit 143 may cause the display unit 12 to display the processed information and the image generated by the display information generation unit 142.

  The display control unit 143 may display the processed information and the image obtained from the display information generation unit 142 on the display unit 12 in a manner corresponding to a predetermined event. For example, when a left turn is detected, the display control unit 143 may cause the display unit 12 to display the image and the processed information in such a manner as to look down on the autonomous driving vehicle 2. The display control unit 143 may display the image and the processed information on the display unit 12 in a manner as viewed from the autonomous driving vehicle 2 when the intersection approach is detected. Thereby, the user U can be made to grasp the information processed by the autonomous driving vehicle 2 more appropriately.

  The storage unit 15 is a storage medium that stores a control program executed by the control unit 14, information received from the autonomous driving vehicle 2, and the like. The storage unit 15 is a storage medium such as an HDD (Hard Disk Drive), a ROM (Read Only Memory) realized by a flash memory, or a RAM (Random Access Memory).

  The storage unit 15 stores an automatic driving evaluation program 151, event-related information 152, map information 153, and specific object information 154.

  The automatic driving evaluation program 151 is a control program that is executed by the control unit 14 to determine whether or not the automatic driving vehicle 2 can be appropriately automatically driven.

  The event related information 152 is information in which the autonomous driving vehicle 2 is associated with the type of the predetermined event and the confirmation direction for the predetermined event. Details of the event related information 152 will be described later.

  The map information 153 is information indicating a map including information such as roads, traffic lights, and signs on which the autonomous driving vehicle 2 travels.

  The specific object information 154 is information indicating an object that is a specific object. The specific object information 154 includes, for example, a pedestrian and a bicycle. The specific object information 154 may include information indicating an object that is not a specific object. For example, the specific object information 154 may include a guardrail or the like as an object that is not a specific object.

  The specific object information 154 may be accompanied by an object mode. For example, even if it is a “pedestrian”, for example, whether or not it is determined as a specific object may be different between when walking toward the autonomous driving vehicle 2 and when stopping. In this case, the specific object information 154 includes “moving” + “pedestrian”, “face orientation is front” + “pedestrian”, or “moving” + “face orientation is front. Or “pedestrian”. Moreover, even if it is the same object, the determination whether it is a specific object may differ based on the distance of the autonomous driving vehicle 2 and an object. For example, an object may be determined to be a specific object when it is closer than an arbitrarily determined reference distance, and may be determined to be not a specific object when it is farther than the reference distance. In this case, the specific object information 154 may be “closer to the reference distance” + “pedestrian” or the like. Thus, the specific object information 154 may be arbitrarily determined.

  FIG. 3 is a block diagram illustrating a functional configuration of the autonomous driving vehicle 2 that communicates with the terminal 1 that is an example of the information processing system according to the embodiment.

  The autonomous driving vehicle 2 is a moving body that autonomously operates (that is, travel control) based on a control program. The automatic driving vehicle 2 is, for example, a vehicle. Further, the automatic driving performed by the automatic driving vehicle 2 may be a fully automatic driving that is not driven by a person, or may be an automatic driving in which a part of driving is performed by a person.

  The automatic driving vehicle 2 includes a communication unit 21, a photographing unit 22, a position measurement unit 23, other sensors 24, a drive unit 25, a control unit 26, and a storage unit 27.

  The communication unit 21 is a communication interface such as a communication adapter that communicates with the terminal 1.

  The photographing unit 22 is a camera for photographing the outside of the autonomous driving vehicle 2. The photographing unit 22 may be a single camera that is movable so as to be able to photograph all directions outside the autonomous driving vehicle 2, or may be a plurality of cameras that respectively capture the front, rear, left, and right directions of the autonomous driving vehicle 2.

  The position measuring unit 23 measures the position of the autonomous driving vehicle 2. The position measurement unit 23 is realized by, for example, a distance sensor such as GPS, LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), or a millimeter wave radar, and the distance from the autonomous driving vehicle 2 and the autonomous driving vehicle 2 Acquires position information of an object located within a predetermined range. Further, the position measuring unit 23 may measure the distance between the autonomous driving vehicle 2 and an object located around the autonomous driving vehicle 2.

  The other sensors 24 are various sensors that measure the driving state of the autonomous driving vehicle 2 such as the steering angle and the vehicle speed. The other sensors 24 are realized by, for example, a steering angle sensor, a speed sensor, or the like.

  The drive unit 25 is various devices such as an engine, a battery, and a drive train related to driving of the autonomous driving vehicle 2.

  The control unit 26 controls the drive unit 25 based on the information acquired by shooting by the shooting unit 22, the information acquired by the position measurement unit 23, the map information 271 and the route information 273 stored in the storage unit 27. Thus, the control device controls the traveling of the autonomous driving vehicle 2. The control unit 26 is realized by a CPU, for example. Functionally, the control unit 26 includes a communication control unit 261, a travel control unit 262, an object recognition unit 263, a shooting information acquisition unit 264, and a recognition object information acquisition unit 265.

  The communication control unit 261 controls communication between the terminal 1 and the autonomous driving vehicle 2 by controlling the communication unit 21.

  The travel control unit 262 controls the travel of the autonomous driving vehicle 2 by controlling the drive unit 25 based on the map information 271, the route information 273, and the recognition result information obtained from the object recognition unit 263.

  The imaging information acquisition unit 264 acquires an image outside the autonomous driving vehicle 2 as imaging information by controlling the imaging unit 22.

  The object recognition unit 263 recognizes an object included in the image by analyzing the image, which is the shooting information acquired by the shooting information acquisition unit 264, based on the object recognition program 274. The object recognition unit 263 generates recognition result information such as information on the recognized object. The generated recognition result information is stored in the storage unit 27.

  The recognized object information acquisition unit 265 transmits the recognition result information generated by the object recognition unit 263 to the terminal 1 via the communication unit 21. For example, when the recognition object information acquisition unit 265 receives a request for recognition result information from the terminal 1, the recognition object information acquisition unit 265 acquires the recognition result information from the storage unit 27 and provides it to the communication control unit 261. The recognition result information is transmitted to the terminal 1 via the communication control unit 261 and the communication unit 21.

  The storage unit 27 is a storage medium that stores a control program executed by the control unit 26, information received from the terminal 1, and the like. The storage unit 27 is a storage medium such as a ROM or a RAM realized by, for example, an HDD or a flash memory.

  The storage unit 27 stores map information 271, an automatic driving program 272, route information 273, and an object recognition program 274.

  The map information 271 is map information used for traveling control of the autonomous driving vehicle 2. The map information 271 may include information similar to the map information 153 shown in FIG.

  The automatic driving program 272 is a program executed by the travel control unit 262 and is a control program for controlling the automatic driving of the automatic driving vehicle 2.

  The route information 273 is information indicating a travel route on which the autonomous driving vehicle 2 travels.

  The object recognition program 274 is a program executed by the object recognition unit 263 and is a control program for recognizing an object.

[Processing procedure]
Subsequently, a processing procedure of the terminal 1 and the autonomous driving vehicle 2 according to the embodiment of the present disclosure will be described.

  Hereinafter, a processing procedure of the terminal 1 according to the embodiment of the present disclosure will be described.

  FIG. 4 is a flowchart illustrating a processing procedure executed by the terminal 1 according to the embodiment of the present disclosure.

  First, the user U carries the terminal 1 and gets on the autonomous driving vehicle 2. The automatic driving vehicle 2 starts automatic driving.

  Here, the terminal 1 detects a predetermined event based on, for example, the position information of the terminal 1 acquired by the position measurement unit 13, the map information 153 or route information, and the event related information 152 (step S1). Specifically, in step S <b> 1, the terminal 1 acquires the position type indicated by the position information of the terminal 1 acquired by the position measurement unit 13 from the map information 153, and the acquired position type is the event related information 152. It is determined whether or not it matches the type of position indicated. In addition, the terminal 1 determines a change in the travel control of the autonomous driving vehicle 2 from the route after the position indicated by the position information of the terminal 1 acquired by the position measurement unit 13 in the route indicated by the route information, and the determined changed change It is determined whether or not the traveling control matches the traveling control indicated in the event related information 152.

  In addition, since the timing of the predetermined event can be determined from the travel distance when the route on which the autonomous driving vehicle 2 travels is a fixed route determined in advance, the terminal 1 is acquired by the position measurement unit 13, for example. Based on the position information of the terminal 1, the travel distance of the autonomous driving vehicle 2 may be calculated, and it may be determined from the calculated travel distance whether a predetermined event has occurred.

  Next, the terminal 1 acquires the confirmation direction associated with the predetermined event detected in step S1 from the event related information 152 (step S2).

  Next, the terminal 1 transmits a signal requesting photographing information including a confirmation direction and recognition result information to the autonomous driving vehicle 2 (step S3). If the shooting information including the confirmation direction can be acquired, the request to the autonomous driving vehicle 2 may not be a request for specifying the shooting information as the shooting information including the confirmation direction, but may be a simple request for shooting information.

  Next, the terminal 1 receives the shooting information and the recognition result information from the autonomous driving vehicle 2, and determines whether or not the received recognition result information includes an object recognition result in the confirmation direction (step S4). .

  When it is determined that the recognition result information received from the autonomous driving vehicle 2 includes the object recognition result in the confirmation direction (Yes in step S4), the terminal 1 uses the received recognition result information in the confirmation direction. It is determined whether or not a specific object exists (step S5).

  If it is determined that the specific object exists (Yes in step S5), the terminal 1 adds enhancement information that emphasizes the specific object to the shooting information (step S6).

  Next, the terminal 1 generates display information including shooting information to which the emphasis information is added and information (first information) indicating that there is a specific object in the confirmation direction, that is, processed information (step S7).

  On the other hand, if the terminal 1 determines that the information received from the autonomous driving vehicle 2 does not include the recognition direction recognition result information (No in step S4), the terminal 1 sets the shooting information and the confirmation direction in the autonomous driving vehicle 2. Display information including unprocessed information indicating that recognition processing has not been performed is generated (step S8). According to such a method, it is possible to make the user U properly grasp that the automatic driving vehicle 2 is not appropriately automatically driven.

  Further, when the terminal 1 determines that the specific object does not exist (No in step S5), the display including the imaging information and information indicating that there is no obstacle in the confirmation direction (second information), that is, processed information. Information is generated (step S9).

  After step S7, step S8, or step S9, the terminal 1 gives the display information a check box for the user U to input an evaluation result as to whether or not the autonomous driving vehicle 2 is appropriately performing the recognition process. (Step S10).

  Next, the terminal 1 outputs the display information generated in step S10 (step S11).

  According to the method as described above, the user U who evaluates whether or not the recognition process for automatically driving the autonomous driving vehicle 2 is appropriately performed has the recognition process for automatically driving the autonomous driving vehicle 2. Can output whether it is done properly. Therefore, according to the method as described above, the user can grasp whether or not the recognition process related to the driving of the autonomous driving vehicle 2 is appropriately performed.

  FIG. 5 is a flowchart illustrating a processing procedure executed by the autonomous driving vehicle 2 communicating with the terminal 1 which is an example of the information processing system in the embodiment.

  First, the user U carries the terminal 1 and gets on the autonomous driving vehicle 2. The automatic driving vehicle 2 starts automatic driving.

  It is assumed that the autonomous driving vehicle 2 receives a signal indicating a request for the shooting information in the confirmation direction and the recognition result information from the terminal 1 (step S21). That is, step S21 is a step performed after step S3 shown in FIG.

  Next, the autonomous driving vehicle 2 acquires shooting information in the confirmation direction (step S22). Specifically, the autonomous driving vehicle 2 operates the photographing unit 22 that is photographing the confirmation direction, and acquires an image as photographing information.

  Next, the autonomous driving vehicle 2 recognizes information indicating an object included in the shooting information using the shooting information acquired in step S22 (specifically, by performing recognition processing using the shooting information). Result information is acquired (step S23).

  Next, the autonomous driving vehicle 2 transmits the imaging information acquired in step S22 and the recognition result information acquired in step S23 to the terminal 1 (step S24). Note that since the shooting information and the recognition result information are acquired in the automatic driving of the autonomous driving vehicle 2 regardless of whether there is a request or not, the already acquired shooting information and the recognition result information may be transmitted in response to the request. .

  After step S24, the terminal 1 executes step S4 shown in FIG.

  FIG. 6A is a diagram illustrating an example of an image that is photographing information including the autonomous driving vehicle 2 that communicates with the terminal 1 that is an example of the information processing system in the embodiment. FIG. 6B is a diagram for describing an example of the automatic driving vehicle 2 that communicates with the terminal 1 that is an example of the information processing system in the embodiment, and the distance from the automatic driving vehicle 2 to the object. That is, FIG. 6A is a diagram illustrating an example of shooting information, and FIG. 6B is a diagram illustrating an example of recognition result information.

  The automatic driving vehicle 2 transmits, as shooting information, for example, a captured image 31 that is an example of shooting information obtained by shooting around the autonomous driving vehicle 2 to the terminal 1.

  In addition, the autonomous driving vehicle 2 recognizes the distance between the autonomous driving vehicle 2 and the object and the type of the object by the recognition process, and transmits the recognized distance and type to the terminal 1 as recognition result information. In FIG. 6B, as shown in FIG. 6A, the center of the autonomous driving vehicle 2 when the autonomous driving vehicle 2 is viewed from above is defined as the origin, and the traveling direction of the autonomous driving vehicle 2 passing through the origin is defined as the Y-axis direction. The position (that is, the coordinates) of the object when the direction orthogonal to the Y-axis direction and passing through the origin is taken as the X-axis direction is shown.

  FIG. 7 is a diagram illustrating an example of the event related information 152.

  The storage unit 15 of the terminal 1 stores event related information 152 as shown in FIG.

  The event related information 152 includes an event type of a predetermined event that occurs in the autonomous driving vehicle 2 such as a left turn and a right turn, and a confirmation direction corresponding to the predetermined event. When the terminal 1 detects a predetermined event using the event-related information 152 as illustrated in FIG. 7, the terminal 1 acquires direction information indicating a confirmation direction corresponding to the detected type of the predetermined event.

  FIG. 8A is a diagram illustrating an example of map information 271 and route information 273. FIG. 8B is a diagram for explaining each point of the map information 271 shown in FIG. 8A.

  The autonomous driving vehicle 2 controls traveling based on map information 271 and route information 273 as shown in FIG. 8A. In addition, for example, when the autonomous driving vehicle 2 enters the intersection, which is a case where the vehicle arrives at the point A shown in FIG. 8A, it is assumed that the predetermined event “intersection entry” of the event types shown in FIG. 8B occurs. As shown in FIG. 7, at least the left direction and the right direction of the autonomous driving vehicle 2 are photographed, and automatic driving is performed based on the result of recognition processing using an image obtained by photographing. At that time, shooting information and recognition result information are generated.

  Based on the event related information 152, the terminal 1 is acquired by shooting information acquired by shooting a direction including a confirmation direction related to a predetermined event (event type shown in FIG. 7), and recognition processing using the shooting information. Recognition result information is obtained from the autonomous driving vehicle 2, and it is determined whether or not the autonomous driving vehicle 2 is executing the recognition process to be executed in the detected predetermined event without any problem. Display an image accordingly. The user U evaluates whether or not there is a problem with the automatic driving of the autonomous driving vehicle 2 by checking the image.

[Display Information]
Subsequently, display information output by the terminal 1 according to the embodiment of the present disclosure will be described. Hereinafter, as a specific example of the display information output from the terminal 1 (that is, the display information output from the terminal 1 in step S11 in FIG. 4), a GUI (Graphical) that is an image displayed on the display unit 12 of the terminal 1 is used. User Interface) will be described.

<First example>
FIG. 9A is a diagram illustrating a first example of an image displayed on the terminal 1 according to the embodiment. The image 3 is an example of display information.

  The image 3 includes a captured image 31 captured by the autonomous driving vehicle 2, a safety confirmation image 33, and a check box 34. The safety confirmation image 33 is an example of an image indicating first information indicating that the autonomous driving vehicle 2 has recognized a specific object in the confirmation direction.

  The captured image 31 is an image including shooting information acquired by the terminal 1 from the autonomous driving vehicle 2. Specifically, the captured image 31 is a composite image of each image captured by the plurality of imaging units 22, and an overhead image of the autonomous driving vehicle 2 is superimposed on the composite image. FIG. 9A illustrates a case where the captured image 31 includes the autonomous driving vehicle 2 and the object 32.

  The safety confirmation image 33 indicates whether or not the recognition result information acquired by the terminal 1 from the autonomous driving vehicle 2 includes the recognition direction recognition result information, and whether or not a specific object using the recognition result information exists. It is an image including the determination result. FIG. 9A shows a safety confirmation image in which the confirmation direction and whether or not the recognition result information of the confirmation direction is included and the information indicating whether or not the object is recognized are “left rear OK: there is a moving object”. An example displayed as 33 is shown.

  The user U confirms the safety confirmation image 33 to grasp whether or not the autonomous driving vehicle 2 has performed the recognition process for appropriately performing the automatic driving, and operates the check box 34 to operate the automatic driving vehicle 2. However, it is evaluated whether or not the object 32 is correctly recognized.

  For example, if the recognition result information in the confirmation direction is included in the recognition result information acquired from the autonomous driving vehicle 2, the terminal 1 displays “safety confirmation image 33” as shown in FIG. If the body is present and the confirmation direction recognition result information is not included, an image indicating unprocessed information such as “NG” is displayed as the safety confirmation image 33. Thereby, the user U can grasp | ascertain whether the automatic driving vehicle 2 is performing the recognition process appropriately by confirming the safety confirmation image 33. FIG. That is, the user U can easily grasp whether or not the autonomous driving vehicle 2 is appropriately automatically driven.

  The check box 34 includes, for example, a display such as Good or Bad. The user U confirms that the object 32 is displayed, and the safety confirmation image 33 confirms the display such as “left rear OK: there is a moving object” and the automatic driving vehicle 2 should confirm the predetermined event. Whether or not the direction recognition processing is correctly executed is evaluated. For example, if it is evaluated as correct, “Good” in the check box 34 is selected, and if it is evaluated as incorrect, “Bad” is selected.

  By repeating such an operation for each predetermined event, the user U can easily grasp whether or not the automatic driving vehicle 2 is performing recognition processing for appropriately performing automatic driving.

<Second example>
FIG. 9B is a diagram illustrating a second example of an image displayed on the terminal 1 in the embodiment. The image 30 is an example of display information.

  The image 30 includes a captured image 31 captured by the autonomous driving vehicle 2, a safety confirmation image 331, and a check box 34. The safety confirmation image 331 is an example of an image indicating second information indicating that the autonomous driving vehicle 2 has not recognized the specific object in the confirmation direction.

  Unlike the image 3 illustrated in FIG. 9A, the captured image 31 does not include the object 32. In this case, when the terminal 1 acquires the recognition result information from the autonomous driving vehicle 2 and determines that there is no specific object using the recognition result information, the image 30 does not include the specific object. The safety confirmation image 331 indicating the second information indicating that is displayed. For example, the safety confirmation image 331 displays “left rear OK: no moving object”.

  Even when the object 32 is not displayed in the captured image 31, the user U confirms the safety confirmation image 331 that is displayed as “left rear OK: no moving object” or the like, so that the autonomous driving vehicle 2 It is possible to grasp that recognition processing is performed to appropriately perform automatic driving.

<Third example>
FIG. 10 is a diagram illustrating a third example of an image displayed on the terminal 1 according to the embodiment. The image 300 is an example of display information.

  An image 300 shown in FIG. 10 further includes a confirmation area 35, emphasis information 36, traveling direction information 37, and a travel control image 332 in addition to the image 3 shown in FIG. 9A. The travel control image 332 is an example of an image indicating that the autonomous driving vehicle 2 has changed travel control from predetermined control content.

  The confirmation area 35 is an image showing a range on the captured image 31 corresponding to the confirmation direction, and is superimposed on the captured image 31. By doing so, the user U can easily grasp which direction and range the autonomous driving vehicle 2 has executed the object recognition processing.

  The enhancement information 36 is an image superimposed on the captured image 31 so that the position of the object 32 in the captured image 31 can be easily understood. In FIG. 10, an image surrounding the periphery of the object 32 in the captured image 31 is shown as the enhancement information 36. By doing so, it becomes easy for the user U to check whether or not the object 32 is included in the captured image 31.

  FIG. 11A is a diagram illustrating a first example of the enhancement information 36 of the object 32 displayed on the terminal 1 in the embodiment. FIG. 11B is a diagram illustrating a second example of the enhancement information 361 of the object 32 displayed on the terminal 1 according to the embodiment.

  The emphasis information is displayed in a different manner depending on whether or not the travel control of the autonomous driving vehicle 2 is affected. For example, the first example of the enhancement information 36 illustrated in FIG. 11A is an example of a display when the object 32 is moving, and the second example of the enhancement information 361 illustrated in FIG. 11B does not move the object 32. It is an example of the display of a case.

  In automatic driving, for example, whether or not the content of the traveling control is changed may be different between the case where the object 32 is moving toward the automatic driving vehicle 2 and the case where the object 32 is stopped. Therefore, when the autonomous driving vehicle 2 changes the control of traveling, that is, when the object 32 is a specific object (for example, FIG. 11A), when it is not changed, that is, when the object 32 is not a specific object (for example, 11B), by changing the display method like the emphasis information 36 or the emphasis information 361, the user U can grasp at a glance whether or not the object 32 is a specific object.

  In FIG. 11A, the emphasis information 36 is indicated by a solid line, and in FIG. 11B, the emphasis information 361 is indicated by a broken line. However, it is sufficient that the difference between the enhancement information 36 and the enhancement information 361 can be identified to the user U. The emphasis information 36 and the emphasis information 361 may have different colors.

  Further, for example, when the object 32 is moving as in the traveling direction information 37 illustrated in FIG. 10, the terminal 1 displays the moving direction of the object 32 as indicated by an arrow, for example. Also good.

  Moreover, the terminal 1 may display the travel information which shows the content of the travel control of the automatic driving vehicle 2 like the travel control image 332 shown in FIG. For example, the traveling performed by the autonomous driving vehicle 2 from a predetermined event is determined in advance, such as a left turn. Here, when the autonomous driving vehicle 2 determines that the object 32 is a specific object, the autonomous driving vehicle 2 may change the content of the traveling control, such as stopping. Therefore, the terminal 1 acquires travel information indicating the content of the travel control of the autonomous driving vehicle 2, determines whether the acquired travel information matches the content of the travel control executed in a predetermined event, and determines the determination result. You may display like the driving control image 332.

  With these displays, the user U can more easily grasp the content of the travel control of the autonomous driving vehicle 2 and the change in the travel control.

<Fourth example>
FIG. 12 is a diagram illustrating a fourth example of an image displayed on the terminal 1 according to the embodiment. The image 4 is an example of display information. In FIG. 12, for example, recognition result information may be displayed in the same manner as the safety confirmation image 33 shown in FIG. 9A, but the illustration is omitted.

  An image 4 shown in FIG. 12 includes a captured image 311 obtained by capturing a direction (second direction) different from the confirmation direction (first direction) with respect to the image 3.

  The photographed image 311 is an image obtained by photographing a direction (second direction) not included in the photographed image 31 including the confirmation direction (first direction), for example. Thus, the terminal 1 acquires not only the image obtained by photographing the confirmation direction but also the image obtained by photographing the second direction having a lower priority than the confirmation direction from the autonomous driving vehicle 2. That is, the automatic driving vehicle 2 transmits to the terminal 1 shooting information and recognition result information including an image obtained by shooting the confirmation direction and an image obtained by shooting the second direction. The terminal 1 determines whether the automatic driving vehicle 2 has recognized the specific object in the second direction using the recognition result information acquired from the automatic driving vehicle 2. When it is determined that the specific object is recognized in the second direction, the terminal 1 determines whether the traveling control of the autonomous driving vehicle 2 has changed. When it is determined that the traveling control has changed, the terminal 1 includes the shooting information including the second direction and the information indicating that the specific object is recognized in the second direction in the display information, for example, the image illustrated in FIG. 4 is displayed. Note that the change in the travel control may be determined using information indicating the content of the travel control acquired from the autonomous driving vehicle 2, and is determined based on shooting information or position information acquired from the autonomous driving vehicle 2. Also good.

  As described above, when a change occurs in the travel control due to the specific object detected in a direction different from the confirmation direction, the terminal 1 may display an image in the different direction. For example, when the predetermined event is a left turn, the terminal 1 displays an image of the autonomous driving vehicle 2 in the left direction. Furthermore, the terminal 1 acquires, for example, recognition result information in which the autonomous driving vehicle 2 recognizes a specific object in the right direction of the autonomous driving vehicle 2 and determines that the traveling control of the autonomous driving vehicle 2 has changed. Also displays an image of the autonomous driving vehicle 2 in the right direction. By doing so, the user U can more appropriately grasp whether or not the automatic driving recognition process is appropriately performed by the automatic driving vehicle 2 in directions other than the confirmation direction. Further, the user U can know the reason why the traveling control different from the traveling control according to the predetermined event is performed.

  Similar to the confirmation area 35, the confirmation area 351 is an image indicating a range on the captured image 311 corresponding to the second direction in the captured image 311, and may be superimposed on the captured image 311. Further, an image in the second direction in which the specific object is recognized may be displayed regardless of whether or not the traveling control has changed.

<Fifth example>
FIG. 13 is a diagram illustrating a fifth example of an image displayed on the terminal 1 according to the embodiment. The image 40 is an example of display information.

  Although FIG. 9A-FIG. 12 showed the example which the terminal 1 displays the imaging | photography image of the perimeter of the autonomous driving vehicle 2 produced | generated based on the image image | photographed by the some imaging | photography part 22, the terminal 1 is a picked-up image. As in 312, only an image indicating the confirmation direction may be displayed. For example, when the terminal 1 receives a captured image of the entire circumference of the autonomous driving vehicle 2 from the autonomous driving vehicle 2, the terminal 1 selects and displays a range of the image including the confirmation direction based on the recognition result information. Also good. Specifically, the terminal 1 displays an image acquired from the autonomous driving vehicle 2 in a mode corresponding to a predetermined event. For example, when the event type of the event that has occurred is a left turn, the terminal 1 displays a captured image of the entire circumference of the autonomous driving vehicle 2, and when the event type of the event that has occurred is an intersection approach, Display the captured image in front.

  In this way, the user U can more easily grasp whether or not the recognition direction recognition processing has been appropriately executed by the autonomous driving vehicle 2.

  Further, for example, like the confirmation area 35 shown in FIG. 10, the confirmation area 352 is an image showing a range on the photographed image 312 corresponding to the confirmation direction in the photographed image 312, and may be superimposed on the photographed image 312. Good.

  Here, when the terminal 1 displays an image such as the captured image 312, the user U has difficulty in grasping the depth from the confirmation image 312. That is, the user U can confirm whether or not the object 32 exists by confirming the captured image 312, but the user U estimates it from the size of the distance object 32 between the object 32 and the autonomous driving vehicle 2. Difficult to grasp. Therefore, the terminal 1 may change the display mode of the confirmation area 352 according to the distance between the object 32 and the autonomous driving vehicle 2. For example, the terminal 1 displays the confirmation area 352 in black when the distance between the object 32 and the autonomous driving vehicle 2 is not less than a predetermined distance that is arbitrarily determined in advance, and the confirmation area 352 if the distance is less than the predetermined distance. Is displayed in red. For example, if the distance between the object 32 and the autonomous driving vehicle 2 is less than a predetermined distance that is arbitrarily determined in advance, the terminal 1 blinks and displays the object 32 in the captured image 312. The display mode may be changed.

  According to such a configuration, the user U can easily understand how far the position of the object 32 is from the autonomous driving vehicle 2. Therefore, the user U can grasp whether or not the traveling control of the autonomous driving vehicle 2 is appropriately performed.

  Further, for example, similarly to the safety confirmation image 33 illustrated in FIG. 9A, the terminal 1 uses the recognition result information acquired from the autonomous driving vehicle 2 such as “forward OK: there is a moving object” illustrated in the safety confirmation image 333. An image including the determination result as to whether or not the specific object is present is displayed.

(Other embodiments)
As described above, the information processing method or the information processing system according to one or more aspects of the present disclosure has been described based on the embodiment, but the present disclosure is not limited to the embodiment. Unless it deviates from the gist of the present disclosure, one or more of the present disclosure may be applied to various modifications conceived by those skilled in the art in the present embodiment, or forms configured by combining components in different embodiments. It may be included within the scope of the embodiments.

  For example, in the above-described embodiment, the example in which the user U evaluates the recognition result of the autonomous driving vehicle 2 and the traveling control based on the recognition result has been described, but the terminal 1 may perform the evaluation. For example, when it is determined that the recognition result information acquired from the autonomous driving vehicle 2 includes the object recognition result in the confirmation direction, the terminal 1 adds points, adds up the points after the end of travel, and scores as an evaluation. May be calculated. In addition, when it is determined that the recognition result information acquired from the autonomous driving vehicle 2 does not include the object recognition result in the confirmation direction, the points may be subtracted. Thereby, driving | running | working of the autonomous driving vehicle 2 can be evaluated automatically.

  Further, for example, in the above embodiment, all or part of the components such as the terminal 1 may be configured by dedicated hardware, or realized by executing a software program suitable for each component. May be. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as an HDD or a semiconductor memory. Here, the software that realizes the information processing system of the above-described embodiment is a program as follows.

  That is, this program acquires event related information 152 for detecting a predetermined event for the autonomous driving vehicle 2 in a computer, determines whether or not the predetermined event is detected using the event related information 152, When it is determined that the event has been detected, the direction information indicating the first direction associated with the predetermined event is acquired, and the object located outside the autonomous driving vehicle 2 is obtained by the recognition process by the autonomous driving vehicle 2. 1 recognition result information is acquired, it is determined whether or not the first recognition result information includes an object recognition result in the first direction, and the automatic driving vehicle 2 travels in the first direction using the first recognition result information. It is determined whether or not there is a specific object that may affect the first direction, and it is determined that there is no specific object in the first direction, and the first recognition result information includes When it is determined that the object recognition result of the direction is included, the processed information indicating that the autonomous driving vehicle 2 has already recognized the first direction is generated, and the generated processed information is output. Let

  Further, the constituent elements such as the terminal 1 may be configured by one or a plurality of electronic circuits. Each of the one or more electronic circuits may be a general-purpose circuit or a dedicated circuit.

  The one or more electronic circuits may include, for example, a semiconductor device, an IC (Integrated Circuit), an LSI (Large Scale Integration), or the like. The IC or LSI may be integrated on one chip or may be integrated on a plurality of chips. Here, it is called IC or LSI, but the name changes depending on the degree of integration, and may be called system LSI, VLSI (Very Large Scale Integration), or ULSI (Ultra Large Scale Integration). An FPGA (Field Programmable Gate Array) programmed after manufacturing the LSI can be used for the same purpose.

  In addition, general or specific aspects of the present disclosure may be realized by a system, an apparatus, a method, an integrated circuit, or a computer program. Alternatively, it may be realized by a computer-readable non-transitory recording medium such as an optical disk, HDD, or semiconductor memory in which the computer program is stored. Further, the present invention may be realized by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

  In addition, it is realized by arbitrarily combining the components and functions in each embodiment without departing from the scope of the present invention, or a form obtained by subjecting each embodiment to various modifications conceived by those skilled in the art. Forms are also included in the present disclosure.

  The information processing method and the information processing system according to the embodiment of the present disclosure can be used for an apparatus or a system that appropriately leaves a user whether or not automatic driving of a vehicle is appropriately performed.

DESCRIPTION OF SYMBOLS 1 Terminal 11, 21 Communication part 12 Display part 13, 23 Position measurement part 14, 26 Control part 141,261 Communication control part 142 Display information generation part 143 Display control part 144 Vehicle process information acquisition part 145 Vehicle recognition determination part 146 Event determination Unit 147 direction information acquisition unit 15, 27 storage unit 151 automatic driving evaluation program 152 event-related information 153, 271 map information 154 specific object information 2 automatic driving vehicle 22 photographing unit 24 other sensor 25 driving unit 262 travel control unit 263 object recognition Unit 264 imaging information acquisition unit 265 recognized object information acquisition unit 272 automatic operation program 273 route information 274 object recognition program 3, 30, 300, 4, 40 image 31, 311, 312 captured image 32 object 33, 331, 333 safety confirmation image 332 Travel control screen 34 checkbox 35,351,352 check area 36,361 enhancement information 37 traveling direction information U user

Claims (12)

Using a processor
Get event related information to detect a predetermined event about the vehicle,
It is determined whether the predetermined event is detected using the event related information,
If it is determined that the predetermined event has been detected, direction information indicating a first direction associated with the predetermined event is acquired,
Obtaining first recognition result information obtained by recognition processing by the vehicle of an object located outside the vehicle;
Determining whether the first recognition result information includes an object recognition result in the first direction;
Using the first recognition result information, it is determined whether or not there is a specific object that may affect traveling of the vehicle in the first direction,
When it is determined that the specific object does not exist in the first direction, and when it is determined that the object recognition result in the first direction is included in the first recognition result information, the vehicle is Generate processed information indicating that the first direction has been recognized,
Outputting the generated processed information;
Information processing method. further,
If it is determined that the specific object exists in the first direction, the processed information including first information indicating that the specific object has been recognized in the first direction is generated.
If it is determined that the specific object does not exist in the first direction, the processed information including second information indicating that the specific object has not been recognized in the first direction is generated.
The information processing method according to claim 1. The first recognition result information is information obtained by performing the recognition process on an image obtained by photographing a predetermined range including the first direction from the vehicle.
The information processing method according to claim 1 or 2. further,
Obtaining travel information of the vehicle including at least one of a steering angle and a vehicle speed of the vehicle;
The predetermined range is determined according to the travel information of the vehicle.
The information processing method according to claim 3. Furthermore, the travel information of the acquired vehicle is output together with the processed information.
The information processing method according to claim 4. When the predetermined event is detected, the processed information is repeatedly output until the detected predetermined event is not detected or a predetermined time elapses.
The information processing method according to claim 1. Further, when it is determined that the specific object exists in the first direction, it is determined whether or not the vehicle has determined or changed control of traveling due to the specific object,
Generating the first information including a determination result of whether or not the vehicle has determined or changed control of traveling;
The information processing method according to claim 2. further,
Using the first recognition result information, determine whether or not the specific object exists in a second direction different from the first direction;
When it is determined that the specific object is present in the second direction, the processed information including information indicating the second direction is generated.
The information processing method according to claim 1. further,
Obtaining an image generated by being photographed in the first direction by a photographing device mounted on the vehicle;
Outputting the acquired image together with the processed information in a manner corresponding to the detected predetermined event;
The information processing method according to claim 1. further,
If it is determined that the object recognition result in the first direction is not included in the first recognition result information, unprocessed information indicating that the vehicle cannot recognize the first direction is generated.
Outputting the generated raw information,
The information processing method according to claim 1. In order for a computer to execute the information processing method according to any one of claims 1 to 10,
program. An event determination unit that acquires event-related information for detecting a predetermined event about a vehicle and determines whether the predetermined event is detected using the event-related information;
A direction information acquisition unit that acquires direction information indicating a first direction associated with the predetermined event when the event determination unit determines that the predetermined event is detected;
A vehicle processing information acquisition unit for acquiring first recognition result information obtained by recognition processing of an object located outside the vehicle by the vehicle;
It is determined whether the object recognition result in the first direction is included in the first recognition result information, and using the first recognition result information, there is a possibility of affecting the traveling of the vehicle in the first direction. A vehicle recognition determination unit that determines whether or not a specific object exists;
When it is determined by the vehicle recognition determination unit that the specific object does not exist, and when it is determined that the first recognition result includes the object recognition result in the first direction, the vehicle is A display information generation unit that generates processed information indicating that the first direction has been recognized;
A display control unit that causes the display device to display the processed information generated by the display information generation unit,
Information processing system.

Images