JP7213704B2 - AUTOMATED DRIVING PROGRAM EVALUATION SYSTEM AND AUTOMATED DRIVING PROGRAM EVALUATION METHOD - Google Patents

Description

The present invention relates to an automatic driving program evaluation system and an automatic driving program evaluation method.

In the development of self-driving cars (self-driving cars), the improvement of the self-driving program, which is the core of self-driving technology, is progressing. Since the performance of an automated driving program is directly linked to the safety of automated driving, improving its performance is extremely important in terms of the spread of automated driving vehicles. Therefore, tests are widely conducted to verify the performance of autonomous driving programs. However, when testing an automatic driving program by actually driving a car on a road, etc., it is necessary to acquire a huge amount of driving images and spatial data. In addition, it is difficult for such test methods to take into account the effects of uncertainties such as weather.

Therefore, as shown in Non-Patent Document 1, a virtual environment (so-called simulator) that models the behavior of an automatic driving program for a vehicle and various environments around the vehicle is provided.

https://ipg-automotive.com/products-services/simulation-software/carmaker/ (accessed January 21, 2019)

However, while it is possible to execute automated driving in such a virtual environment, almost no method has been developed to objectively evaluate whether such virtual automated driving was appropriate. Therefore, it can be said that the advantage of the virtual environment is not fully utilized.

The present invention has been made in view of such a situation, and its purpose is to provide an automated driving program evaluation system capable of objectively and quantitatively evaluating virtually executed automated driving, and an automated driving system. An object of the present invention is to provide a driving program evaluation method.

One of the present inventions for solving the above problems is an automatic driving program evaluation system, comprising a processor and a memory, for evaluating the content of driving a predetermined automobile under predetermined conditions with predetermined numerical values. an evaluation criteria information storage unit that stores evaluation criteria information that is information of evaluation criteria; and driving of the vehicle that is output as a result of execution of a predetermined program that determines details of driving of the predetermined vehicle under the predetermined conditions. an execution log receiving unit that receives an execution log, which is information including the history of the above, from a predetermined information processing system; and an evaluation result information display unit for displaying the generated evaluation result information, wherein the evaluation result information display unit displays the evaluation result information as , the item of the evaluation criteria related to the driving of the automobile, the numerical value indicating the extent to which the driving of the automobile violates the evaluation criteria, the number of times the driving of the automobile violates the evaluation criteria, and the driving of the automobile The location where the evaluation criteria are violated is displayed .
Further, one of the present inventions for solving the above problems is an evaluation system which includes a processor and a memory, and which is information of evaluation criteria for evaluating the content of driving a predetermined automobile under predetermined conditions with predetermined numerical values. An evaluation reference information storage unit that stores a plurality of pieces of reference information, and information including the driving history of the automobile output as a result of execution of a first program that determines the content of driving of the predetermined automobile under the predetermined conditions. and a second program that is different from the first program and that determines the details of driving of the predetermined vehicle under the predetermined conditions. An execution log receiving unit that receives a second execution log, which is information including a history of driving, from a predetermined information processing system, the received first execution log and second execution log, and the evaluation criteria first evaluation result information indicating the evaluation of the driving of the predetermined vehicle under the predetermined conditions by a predetermined numerical value, based on the first execution log, and the second execution log. an evaluation result information generation unit that generates, for each of the plurality of evaluation criteria, second evaluation result information indicating the evaluation of the driving of the predetermined vehicle under the predetermined conditions by a predetermined numerical value; and an evaluation result information display unit that displays the first evaluation result information and the second evaluation result information, the evaluation result information display unit displaying information about the plurality of evaluation criteria based on the first evaluation result information. , comparing and displaying the average numerical value of the driving evaluation of the automobile and the average numerical value of the driving evaluation of the automobile for the plurality of evaluation criteria based on the second evaluation result information; It is an automatic driving program evaluation method.

According to the present invention, it is possible to objectively and quantitatively evaluate virtually executed automatic driving.

FIG. 1 is a diagram showing an example of the configuration of an automatic driving program development system according to the first embodiment. FIG. 2 is a diagram showing an example of the data configuration of the execution log 300. As shown in FIG. FIG. 3 is a diagram showing another example of the data configuration of the execution log 300. As shown in FIG. FIG. 4 is a diagram showing an example of the evaluation criteria information 210. As shown in FIG. FIG. 5 is a diagram illustrating an example of hardware included in each information processing device that constitutes the automatic driving program development system 1. As shown in FIG. FIG. 6 is a flowchart showing a typical flow of each process in the program development support process. FIG. 7 is a flowchart illustrating an example of scenario information generation processing. FIG. 8 is a diagram showing an example in which the parameter information 213 is converted into an information processing language (XML) processing unit. FIG. 9 is a diagram showing an example of the detailed evaluation result information 250. As shown in FIG. FIG. 10 is a diagram showing another example of the detailed evaluation result information 250. As shown in FIG. FIG. 11 is a diagram showing an example of data items of the evaluation result summary information 270 generated by the evaluation result information generation process. FIG. 12 is a diagram for explaining an example of a screen 1000 for comparing the scenario evaluation result information 210 related to the scenario information 150 . FIG. 13 is a diagram illustrating an example of a screen 1000 on which comparison is made between scenario evaluation result information 210 related to scenario information 150 . FIG. 14 is a diagram illustrating an example of a screen on which each evaluation result information 210 based on the same automatic driving program 170 is compared. FIG. 15 is an example of a screen that compares evaluation result information 210 based on different types of automatic driving programs 170 . FIG. 16 is a flowchart showing an example of program development support processing according to the second embodiment. FIG. 17 is a diagram showing an example of display of the supplementary image 195. As shown in FIG.

Hereinafter, the automatic driving program evaluation system of each embodiment will be described with reference to the drawings.
[[First embodiment]]

<<Configuration of Autonomous Driving Program Development System>>
FIG. 1 is a diagram showing an example of the configuration of an automatic driving program development system according to the first embodiment. The automatic driving program development system 1 evaluates the automatic driving program execution system 10, which is a virtual automatic driving system (automatic driving simulator) using the automatic driving program of the automobile, and the content of the automatic driving by the automatic driving program execution system 10. It is configured including an automatic driving program evaluation system 20.

The automatic driving program execution system 10 is composed of one or more information processing devices. The automatic driving program execution system 10 executes a designated automatic driving program under virtual predetermined conditions (for example, conditions related to virtual roads, signs, pedestrians, weather, etc.). automatic operation is executed on the information processing device.

The automatic driving program evaluation system 20 is composed of one or more information processing devices. The automatic driving program evaluation system 20 evaluates the automatic driving executed by the automatic driving program execution system 10 according to predetermined evaluation criteria stored in advance, and the results are evaluated by users such as developers of the automatic driving program and automobile (hereinafter , simply called the user).

The automatic driving program execution system 10 and the automatic driving program evaluation system 20 are communicably connected by a communication network 5 . The communication network 5 may be, for example, a wired or wireless communication network such as a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or a dedicated line, or directly connect devices in both systems. A predetermined communication line may be used. Further, the information processing devices in the automatic driving program execution system 10 and the automatic driving program evaluation system 20 are also connected by a similar communication network.

Next, the configuration of the automatic driving program execution system 10 will be described.
<Automatic driving program execution system>
The automatic driving program execution system 10 stores map information 110, external world recognition information 130, scenario information 150, and one or more automatic driving programs 170, as shown in FIG.

The map information 110 is information that specifies the shape and position of static objects such as roads, facilities, traffic lights, road signs, and terrain.

The external world recognition information 130 is information of an object that can be recognized by a vehicle that automatically drives (specifically, the automatic driving program 170). In other words, the external world recognition information 130 corresponds to data or image information acquired by various sensors and cameras mounted on automobiles that automatically drive. The external world recognition information 130 includes, for example, images of static objects stored in the map information 110 as well as images or moving images of moving objects such as people, animals, running objects, and objects on the road. . The external world recognition information 130 may use data or images actually acquired by various sensors or cameras of the automobile, or may be data or images arbitrarily created by the user.

The scenario information 150 is information that defines the operating conditions of the automobile. Scenario information 150 includes, for example, motion information of people, animals, or traveling objects (automobiles, bicycles, etc.), existence of objects on the road, existence of traffic lights and their behavior, existence of road signs and their contents, and so on. contains information on various objects that define the operating conditions of the

The scenario information 150 includes, for example, weather (weather, temperature, humidity, wind direction, wind speed, etc.), time zone (morning, noon, twilight, twilight, night, etc.), road surface conditions (humidity, dryness, wetness, puddles, etc.). Driving conditions such as driving route (slope of the road, direction of the road (straight/curve/continuous curve, etc.)), type of road (general road, expressway, sea bridge, etc.) It contains information about the external environment (conditions) that defines the

In addition, the scenario information 150 indicates what kind of control the automobile will perform on its own controlled devices (for example, accelerator, brake, winker, lights, etc.) at what timing and what route it will travel. including information about the operation of the vehicle, which defines the conditions of operation of the vehicle, such as;

In this embodiment, the scenario information 150 is generated by the automatic driving program evaluation system 20, as will be described later. The scenario information 150 is written in a predetermined information processing language. In this embodiment, the scenario information 150 is written in XML (eXtensible Markup Language), but other writing methods may be used.

The automatic driving program 170 drives the vehicle according to the conditions defined by the scenario information 150 . Specifically, the automatic driving program 170 generates a control signal for each control target device of the vehicle so that the vehicle travels along a route defined by the scenario information 150, and virtualizes the control target devices of the vehicle. to operate. In this case, the automatic driving program 170 uses artificial intelligence (AI: artificial intelligence), generates a control signal based on the prediction result, and controls the control target device with the generated control signal. In this way, the automatic driving program 170 realizes automatic driving of the automobile by generating control signals for the control target device.

In this embodiment, as the automatic driving program 170, it is assumed that a plurality of different programs created according to each vehicle are stored in advance. Also, as the automatic driving program 170, a plurality of programs different for each developer (manufacturer, etc.) are stored in advance.

Next, the automatic driving program execution system 10 has functions of an automatic driving program execution unit 101 , an execution log generation unit 103 , and a driving situation display unit 105 .

The automatic driving program execution unit 101 executes the automatic driving program 170 selected by the user.

The execution log generation unit 103 outputs data indicating the details of the operation executed by the automatic operation program execution unit 101 to the execution log 300 . The execution log 300 is data recorded in association with each time (or time) of driving and the details of the driving of the automobile at that time.

A specific example of the execution log 300 will now be described.
<Execution log>
FIG. 2 is a diagram showing an example of the data configuration of the execution log 300. As shown in FIG. This execution log 300 (hereinafter referred to as execution log A) sets a no-stop area in the map information 110 and sets a scenario in which the automobile travels or stops along a predetermined route including this no-stop area in the scenario information 150. This is the log data for the case. This execution log 300 has an elapsed time 301 from when the vehicle started running, and a stop position 303 when the vehicle was stopped in a stop-prohibited area at the timing indicated by the elapsed time 301 as data items. I have.

FIG. 3 is a diagram showing another example of the execution log 300. As shown in FIG. This execution log 300 (hereinafter referred to as execution log B) is log data obtained when a predetermined road with curves is set in the map information 110 and a scenario in which a car travels on this road at high speed is set in the scenario information 130. is. This execution log 300 includes, as data items, an elapsed time 321 from when the automobile started running, and an acceleration 323 in the lateral direction (lateral direction with respect to the traveling direction) of the automobile at the timing indicated by the elapsed time 321. there is

Returning to FIG. 1 , the driving situation display unit 105 visually reproduces automatic driving by the automatic driving program 170 . That is, the driving situation display unit 105 displays the execution process of the predetermined program (automatic driving program 170) as an image (hereinafter referred to as a driving simulation image 190) showing the driving situation of the predetermined automobile under the predetermined conditions. Display by

Specifically, based on the data calculated by the automatic driving program execution unit 101, the driving situation display unit 105 displays an image or video showing the driving situation of the automobile indicated by the map information 110, the external world recognition information 130, and the scenario information 150. Generate and display information.

Further, the driving situation display unit 105 displays an additional image 195 as described in the second embodiment. The accompanying image 195 is an image displayed superimposed on the driving simulation image 190 . Note that the accompanying image 195 is generated by the automatic driving program evaluation system 20 .

<Autonomous driving program evaluation system>
Next, the configuration of the automatic driving program evaluation system 20 will be described.

First, the automatic driving program evaluation system 20 stores evaluation standard information 210, which is information on evaluation standards for evaluating the content of driving a predetermined automobile under predetermined conditions with predetermined numerical values. This evaluation criteria includes criteria for multiple items. The content of the driving indicated by the execution log 300 generated by the automatic driving program execution system 10 corresponds to one of the evaluation criteria items in the evaluation criteria information 210 .

A specific example of the evaluation criteria information 210 will now be described.
<Evaluation Criteria Information>
FIG. 4 is a diagram showing an example of the evaluation criteria information 210. As shown in FIG. The evaluation criteria information 210 includes regulation information 211 that records the content of a predetermined regulation that defines criteria for evaluating the content of driving a vehicle, and parameter information 213 that is information obtained by parameterizing the regulation information 211. ing.

In the present embodiment, the regulation information 211 is a scoring standard (“driver's license skill test implementation standard”) defined based on the Road Traffic Law and the like for a driver's license skill test. The driver's license proficiency test implementation standards prescribe predetermined demerit points for each of a plurality of items (demerit point details) relating to driving skills. If the driver violates the demerit details, the points corresponding to the demerit points will be deducted. It should be noted that the regulation information 211 may be information based on any laws and regulations, or may be based on a guideline by a predetermined industry. That is, the regulation information 211 may be of any format as long as it defines objective evaluation criteria.

The regulation information 211 defines an item 215 that is an object of driving evaluation (demerit points), a stipulated content 217 of the item 215, and a demerit point 219 for driving in violation of the stipulated content 217. there is On the other hand, the parameter information 213 numerically expresses the identifier 221, which is the parameter value set corresponding to the item 215 of the regulation information 211, and the stipulated content 217 of the regulation information 211 by numerical values, formulas, conditional expressions, or the like. It includes a parameter group 223 which is data and a deduction parameter 225 set corresponding to the deduction number 219 of the regulation information 211 . In this embodiment, the number of demerit points 219 (demerit parameter 225) is one of three types of "-5", "-10", and "-20", and an objective and quantitative automatic driving program evaluation is performed. It can be done.

In the example of FIG. 4, in the regulation information 211, regarding the demerit item of "too fast (large)", it is stipulated that "it is a violation when a lateral acceleration of about 0.4 G or more occurs on a curve", and this is violated. If you do, you will be deducted 20 points. Correspondingly, in the parameter information 213, the identifier "1" indicates that "the vehicle's initial speed when entering a curve is 20 km/h to 120 km/h (step 10 km/h), and the curve radius is 15 m to 1000 m ( Step 10m)" is set to "-20" if it is not satisfied.

Returning to FIG. 1, the automatic driving program evaluation system 20 includes a scenario information generation unit 201, a scenario information transmission unit 203, an execution log reception unit 205, an evaluation result information generation unit 207, an accompanying image information transmission unit 208, and an evaluation result information display. Each function of the unit 209 is provided.

Scenario information generation unit 201 generates scenario information 150, which is information indicating conditions for driving a vehicle, based on the portion that defines the details of driving the vehicle extracted from evaluation criteria information 210. FIG.

The scenario information transmission unit 203 transmits the generated scenario information 150 to the automatic driving program execution system 10 .

The execution log reception unit 205 receives the execution log 300 from the automatic driving program execution system 10 . That is, the execution log receiving unit 205 uses information including the driving history of the automobile that is output as the execution result of a predetermined program (automatic driving program 170) that determines the details of driving of the predetermined automobile under predetermined conditions. A certain execution log 300 is received from a predetermined information processing system (automatic driving program execution system 10).

For example, the execution log receiving unit 205 outputs a first execution log 300 under predetermined conditions, which is output as an execution result of a predetermined program (automatic driving program 170), and a predetermined program under the predetermined conditions (automatic driving program 170) to which a predetermined correction has been applied (upgraded automatic driving program 170).

Further, for example, the execution log receiving unit 205 outputs the first execution log 300 under a predetermined condition, which is output as the execution result of the first predetermined program (the first automatic driving program 170), and the predetermined condition A second execution log 300 output as an execution result of a second predetermined program (second automatic operation program 170) different from the first program in .

Further, for example, the execution log reception unit 205 receives the execution log 300 output as the execution result of the predetermined program (automatic driving program 170) executed based on the scenario information 150 generated by the scenario information generation unit 201, and outputs the execution log 300 to a predetermined It is received from the information processing system (automatic driving program execution system 10).

Based on the execution log 300 and the evaluation criteria information 210 received by the execution log receiving unit 205, the evaluation result information generating unit 207 generates evaluation result information 230 indicating the evaluation of driving of a predetermined vehicle under predetermined conditions with a predetermined numerical value. do.

For example, the evaluation result information generation unit 207 generates first evaluation result information 230 based on the first execution log 300 and second evaluation result information 230 based on the second execution log 300 .

Also, the evaluation result information generation unit 207 generates evaluation result information 230 based on the scenario information 150 generated by the scenario information generation unit 201 , the execution log 300 received by the execution log reception unit 205 , and the evaluation criteria information 210 .

The evaluation result information 230 includes evaluation result detailed information 250, which is information about evaluation results for each item of the evaluation criteria, and evaluation result summary information 270, which is information summarizing the evaluation results of all evaluation criteria items. Details of these will be described later.

The accompanying image information transmitting unit 208 transmits image information (accompanying image 195) to the information processing system (automatic driving program execution system 10).

The evaluation result information display section 209 displays the evaluation result information 250 generated by the evaluation result information generation section 207 .

For example, the evaluation result information display unit 209 compares and displays the first evaluation result information 230 and the second evaluation result information 230 using a predetermined numerical value of the evaluation criteria.

Here, FIG. 5 is a diagram illustrating an example of hardware included in each information processing device that constitutes the automatic driving program development system 1. As shown in FIG. Each information processing device includes a processor 51 such as a CPU (Central Processing Unit), a main storage device 52 such as a RAM (Random Access Memory) and a ROM (Read Only Memory), a HDD (Hard Disk Drive), an SSD (Solid State drive), etc., an input device 54 consisting of a keyboard, a mouse, a touch panel, etc., an output device 55 consisting of a monitor (display), etc. for screen display, and a communication device 56 for communicating with each device Prepare.

Each function of the automatic driving program execution system 10 and the automatic driving program evaluation system 20 is implemented by the hardware of each information processing device, or by the processor 51, each program stored in the main storage device 52 or the auxiliary storage device 53 is implemented by reading and executing

In addition, these programs are, for example, secondary storage devices, non-volatile semiconductor memories, hard disk drives, storage devices such as SSD, or IC cards, SD cards, DVDs, etc. Stored on a data storage medium.

<<Processing by Autonomous Driving Program Development System>>
Next, processing (program development support processing) performed by the automatic driving program development system 1 will be described.

<Program development support processing>
FIG. 6 is a flowchart showing a typical flow of each process in the program development support process. First, the automated driving program evaluation system 20 executes scenario information generation processing for generating scenario information 150 (s51), stores the generated scenario information 150, and sends the generated scenario information 150 to the automated driving program execution system 10. Send (s52).

The automatic driving program execution system 10 stores the received scenario information 150 (s11). Based on the scenario information 150, the automatic driving program 170, etc., the automatic driving program execution system 10 executes the automatic driving execution process for executing virtual automatic driving (s13-s17).

That is, first, the automatic driving program execution system 10 selects the automatic driving program 170 to execute processing (s13). Specifically, for example, the automatic driving program execution system 10 receives selection of one of the multiple automatic driving programs 170 from the user.

The automatic driving program execution system 10 reads the map information 110, the external world recognition information 130, and the scenario information 150 (s15). Note that when there is a plurality of pieces of scenario information 150, for example, the automatic driving program execution system 10 accepts selection of one of the pieces of scenario information 150 from the user.

The automatic driving program execution system 10 performs the virtual driving of the automobile specified in the scenario information 150 by processing by the automatic driving program 170 to which the map information 110 and the external world recognition information 130 are input, and the scenario related to the scenario information 150 ends. (s17). Specifically, when the automatic driving program execution system 10 executes the driving of the automobile specified in the scenario information 150, the automatic driving program 170 selected in s13 controls the automobile (controls each component device). .

At this time, the automatic driving program execution system 10 displays the driving simulation image 190 . Also, the automatic driving program execution system 10 accumulates an execution log 300 .

After completing the automatic driving execution process, the automatic driving program execution system 10 transmits the execution log 300, which is the execution result, to the automatic driving program evaluation system 20 (s19).

Upon receiving the execution log 300, the automatic driving program evaluation system 20 executes evaluation result information generation processing for generating evaluation result information 230 based on the received execution log 300, evaluation criteria information 210, and the like (s53).

Specifically, for example, the automatic driving program evaluation system 20 acquires, from the execution log 300, the contents of the driving at each time point (for example, the stop position 303 of the execution log A, the acceleration 323 of the execution log B), and acquires The item 215 of the evaluation criteria information 210 corresponding to the content of driving is specified. Then, the automatic driving program evaluation system 20 acquires the parameter group 223 of the parameter information 213 corresponding to the item 215, and determines whether the operation content satisfies the conditions defined by the acquired parameter group 223 ( OK/NG) is determined. If the driving content does not satisfy the conditions defined by the parameter group 223, the automatic driving program evaluation system 20 subtracts the evaluation score related to the item 215 by the demerit score 219 corresponding to the item 215 specified above. do. The automatic driving program evaluation system 20 makes such determinations at all times in the execution log 300 and generates the results (e.g., evaluation scores for each item of evaluation criteria) as evaluation result information 230 .

Based on the generated evaluation result information 230, the automatic driving program evaluation system 20 executes evaluation result screen display processing for displaying a screen showing the evaluation of the automatic driving program 170 (s55). Note that the automatic driving program execution system 10 may display a screen showing the evaluation of the automatic driving program 170 . With this, the program development support processing ends.

Details of each process in the program development support process will now be described.
<Scenario information generation processing>
First, FIG. 7 is a flowchart for explaining an example of scenario information generation processing. First, the automatic driving program evaluation system 20 selects items of evaluation criteria necessary for generating scenario information 150 (s101). Specifically, for example, the automatic driving program evaluation system 20 receives an input of the item 215 of the regulation information 211 in the evaluation criteria information 210 from the user.

Then, the automatic driving program evaluation system 20 reads the parameter group 223 of the parameter information 213 corresponding to the item 215 selected in s101 (s103). Then, the automatic driving program evaluation system 20 converts the read parameter information 213 into a processing unit of an information processing language (s105).

Here, FIG. 8 is a diagram showing an example in which the parameter information 213 is converted into a processing part of an information processing language (XML). In this example, a processing unit 153 for a driving scenario of a vehicle that sets an initial speed 151 that satisfies the evaluation criteria for entering a curve, and a driving scenario that the vehicle travels a curve having a curve radius 155 that satisfies the evaluation criteria. and the processing unit 157 are generated by XML.

Returning to FIG. 7, the automatic driving program evaluation system 20 determines whether or not there are other evaluation criteria items 215 necessary for generating the scenario information 150 (s107). Specifically, for example, the automatic driving program evaluation system 20 receives input of another item 215 of the evaluation criteria from the user.

If there are other evaluation criteria items 215 necessary for generating scenario information 150 (s107: YES), the automated driving program evaluation system 20 repeats the processes from s101 onwards, and determines the evaluation criteria necessary for generating scenario information 150. If there is no other item 215 (s107: NO), the automatic driving program evaluation system 20 completes generation of the scenario information 150 by combining each processing unit generated so far (s109).

Next, details of the evaluation result information generation process will be described.
<Evaluation result information generation processing>
The evaluation result information generation process displays the evaluation result detailed information 250 and the evaluation result summary information 270 as the evaluation result information 230 as described above.

FIG. 9 is a diagram showing an example of the detailed evaluation result information 250. As shown in FIG. This evaluation result detailed information 250 is information generated corresponding to the execution log A, and the elapsed time 251 and the stop position 253 similar to those of the execution log A, and the car stopped at the position indicated by the stop position 253. and a judgment result 255 in which information indicating whether or not the condition satisfies the judgment criteria is set as an item.

FIG. 10 is a diagram showing another example of the detailed evaluation result information 250. As shown in FIG. This evaluation result detailed information 250 is information generated corresponding to the execution log B. The elapsed time 251 and the acceleration 257 similar to those of the execution log B, and the first degree of risk is that the curve is run at the acceleration 257. (whether or not there is a small danger), and whether traveling on a curve with acceleration 257 exceeds the second degree of danger. and a second determination result 259 in which a determination result as to whether or not there is a great danger is set as an item.

Next, FIG. 11 is a diagram showing an example of data items of the evaluation result summary information 270 generated by the evaluation result information generating process. This evaluation result overview information 270 is generated for each scenario 271 related to the scenario information 150, and includes right/left turns, traveling directions, lanes, collisions with other objects, use of lamps or lights, stop time, speed, and so on in the driving of the scenario 271. It contains information 272 of ratings (expressed in numbers) for items such as exceeding limits.

Next, details of the evaluation result screen display process will be described.
<Evaluation result screen display processing>
The evaluation result screen display process compares and displays the contents of each generated evaluation result information 230 based on the results of the automatic driving execution process executed multiple times. For example, the evaluation result screen display processing includes display of evaluation result information 230 of scenarios related to each scenario information 150, and comparison of each evaluation result information 230 based on the same automatic driving program 170 (comparison for each version of the same program). A screen displaying a comparison of each evaluation result information 230 based on different types of automatic driving programs 170 (comparison by program developers) is displayed. These screens are described below.

(Comparison screen between scenarios)
12 and 13 are diagrams for explaining an example of a screen 1000 for comparing the scenario evaluation result information 230 related to the scenario information 150. FIG. This screen 1000 includes a scenario display column 1001 that displays scenario information 150 used in executing the automatic driving program 170, and items of evaluation criteria that the vehicle has violated (does not satisfy the criteria) in the scenario related to the scenario display column 1001. Violation item display column 1003 for displaying the contents of the violation item display column 1003, deduction point display column 1005 for displaying the total number of points deducted in items related to the violation item display column 1003, and frequency display for displaying the number of violations related to the violation item display column 1003. It has display columns of a column 1007 and a place of occurrence display column 1009 that displays the place where the violation related to the violation item display column 1003 has occurred. The screen 1000 also includes an evaluation point display field 1011 that displays the total number of points deducted for all items (evaluation points).

Specifically, FIG. 12 shows that there were 11 violations against the evaluation criteria of 20 demerit points, ``Do not stop at a place that interferes with the passage of vehicles, etc. on the cross road.'' ing.

In addition, Fig. 13 shows that there were 7 violations of the evaluation criteria of 20 demerit points, ``no lateral acceleration of 0.3G or more and less than 0.4G,'' and ``no lateral acceleration of 0.4G or more.'' This indicates that there was no violation of the evaluation criteria of 20 demerit points.

(Confirmation screen of performance by version upgrade)
FIG. 14 is a diagram illustrating an example of a screen on which each evaluation result information 230 based on the same automatic driving program 170 is compared. This screen 1200 displays a version display field 1201 that displays the version of the automatic driving program 170, and the average value of deduction points for all evaluation criteria when the automatic driving execution process is performed with the version related to the version display field 1201. An all-item average score display column 1203 is provided. That is, this screen 1200 is a screen for confirming an improvement in performance of a new automatic operation program 170 that is a new automatic operation program 170 that has undergone a predetermined modification, that is, has been upgraded to a predetermined automatic operation program 170 .

(Comparison screen by program developer)
FIG. 15 is an example of a screen that compares evaluation result information 230 based on different types of automatic driving programs 170 . This screen 1300 includes a program type display field 1301 that displays the type of the automatic driving program 170, and the number of points deducted in all evaluation criteria when the automatic driving execution process is performed with the automatic driving program 170 related to the program type display field 1301. An all-item average score display column 1303 for displaying an average value is provided. That is, this screen 1300 is a screen for comparing the performance of each automatic driving program 170 that differs for each manufacturer.

As described above, the automatic driving program evaluation system 20 of the present embodiment evaluates the driving history of the vehicle, which is output as the execution result of the automatic driving program 170 that determines the driving details of the vehicle under predetermined conditions. Receive the execution log 300, which is information including, from the automatic driving program execution system 10, and the received execution log 300 and the content of driving a predetermined automobile under predetermined conditions Information of evaluation criteria for evaluating with a predetermined numerical value Based on the evaluation criteria information 210, the evaluation result information 230 indicating the evaluation of the driving of the predetermined vehicle under the predetermined conditions with a predetermined numerical value is generated. It can be determined by reference information 210 .
Thus, according to the automatic driving program evaluation system 20 of the present embodiment, it is possible to objectively and quantitatively evaluate the virtually executed automatic driving.

[Second embodiment]
In the first embodiment, in the program development support process, the automatic driving program execution system 10 executes the vehicle driving process by the automatic driving program 170 defined in the scenario information 150 until the scenario related to the scenario information 150 ends. , the execution log 300 was sent to the automatic driving program evaluation system 20 after the execution.

On the other hand, in the second embodiment, while the automatic driving program execution system 10 is performing the vehicle driving process specified in the scenario information 150, the execution log 300 is sent to the automatic driving program evaluation system 20 at any time. Send. Program development support processing in this embodiment will be described below.

<Program development support processing>
FIG. 16 is a flowchart showing an example of program development support processing according to the second embodiment. First, the scenario information generation process (s51) performed by the automatic driving program evaluation system 20, the storage of the scenario information 150 performed by the automatic driving program execution system 10, the selection of the automatic driving program 170, and the reading of the map information 110 (s11-s15 ) are the same as in the first embodiment.

After that, the automatic driving program execution system 1 executes the driving of the automobile by the automatic driving program 170 based on the map information 110 and the external world recognition information 130 until a predetermined condition is satisfied (s21). Specifically, for example, the automatic driving program execution system 10 executes the automatic driving program 170 for a preset length of time. Also, for example, the automatic driving program execution system 10 executes the automatic driving program 170 until the vehicle reaches a predetermined location. At this time, the automatic driving program execution system 10 accumulates the execution log 300 as in the first embodiment. Also, at this time, the automatic driving program execution system 10 displays the driving simulation image 190 (for example, the driving state of the automobile and the map information 110) as a moving image.

Then, the automatic driving program execution system 10 transmits the execution log 300 accumulated in s21 to the automatic driving program evaluation system 20 (s23).

When receiving the execution log 300, the automatic driving program evaluation system 20 executes evaluation result information generation processing for generating evaluation result information 230 corresponding to the time length of the received execution log 300, as in the first embodiment. (s61).

The automatic driving program evaluation system 20 generates the additional image 195 based on the generated evaluation result information 230, and transmits the generated additional image (s55).

The automatic driving program execution system 10 displays the received accompanying image 195 overlaid on the currently displayed driving simulation image 190 (s25).

The automatic driving program execution system 10 determines whether the driving of the automobile by the automatic driving program 170 has ended (whether the scenario related to the scenario information 150 has ended) (s27). If the driving has ended (s27: YES), the program development support process ends (s29), and if the driving has not ended (s27: NO), the automatic driving program execution system 10 performs the steps after s21. Execute the process again.

A specific example of the additional image 195 will now be described.
<Accompanying image>
FIG. 17 is a diagram showing an example of display of the supplementary image 195. As shown in FIG. A supplementary image 195 is displayed on the driving simulation image 190 displayed in real time by the automatic driving program execution system 10 . The supplementary image 195 is a so-called pop-up image, and includes a demerit point detail display section 1951 that displays the violated evaluation criteria item 215 (demerit point detail), and a demerit point display section that displays the demerit point number 219 corresponding to the item 215. 1952;

As described above, the automatic driving program execution system 10 of the present embodiment displays the execution process of the automatic driving program 170 with the driving simulation image 190 showing the driving situation of a predetermined automobile under predetermined conditions, The automatic driving program evaluation system 20 transmits to the automatic driving program execution system 10 an accompanying image 195 indicating the content of the evaluation result information 230, which is displayed accompanying the image of the automatic driving program execution system 10, so that the user can , during execution of the automatic driving program 170 displaying the driving simulation image 190 (for example, in real time), the information of the evaluation result of the automatic driving program 170 can be easily confirmed. Thereby, the user can know the evaluation result of the automatic driving program 170 at an appropriate timing.

The above description of each embodiment is for facilitating understanding of the present invention, and does not limit the present invention. The present invention may be modified and improved without departing from its spirit, and the present invention includes equivalents thereof.

For example, in each embodiment, the automatic driving program evaluation system 20 generates the scenario information 150 in advance and transmits it to the automatic driving program execution system 10, but the automatic driving program execution system 10 prepares the scenario information 150 in advance. You may make it generate|occur|produce and transmit this to the automatic driving program evaluation system 20. FIG.

Further, part of the functions of the automatic driving program execution system 10 may be provided in the automatic driving program evaluation system 20, and conversely, part of the functions of the automatic driving program evaluation system 20 may be provided in the automatic driving program execution system 10. may

Further, the program development support processing of the first embodiment and the program development support processing of the second embodiment may be combined, and part of the program development support processing of each embodiment may be omitted.

At least the following will be clarified by the above description of the present specification. That is, the automatic driving program evaluation system of each embodiment may include an evaluation result information display unit that displays the generated evaluation result information.

By displaying the generated evaluation result information 230 in this way, the user can objectively and easily evaluate the automatic driving program 170 .

Further, in the automatic driving program evaluation system of each embodiment, the execution log receiving unit outputs a first execution log under the predetermined condition, which is output as an execution result of the predetermined program, and and a second execution log output as an execution result of the program in which the predetermined program is modified in a predetermined manner, and the evaluation result information generation unit generates a first second evaluation result information based on the evaluation result information and the second execution log, and the evaluation result information display unit displays the first evaluation result information and the second evaluation result information according to the evaluation It is also possible to compare and display using a predetermined numerical value as a reference.

In this way, each evaluation result information 230 for each of the first execution log 300, which is the execution result of the predetermined program, and the second execution log 300, which is the execution result of the program to which the predetermined correction has been performed, is provided in a predetermined manner. By comparing and displaying the numerical values of , the user can easily check whether or not the performance of the program has been improved when the version of the predetermined automatic driving program 170 is upgraded.

Further, in the automatic driving program evaluation system of each embodiment, the execution log receiving unit includes the first execution log under the predetermined condition, which is output as the execution result of the first predetermined program, and the predetermined and a second execution log output as an execution result of the second predetermined program different from the first program under the conditions of the evaluation result information generating unit, based on the first execution log second evaluation result information is generated based on the first evaluation result information and the second execution log, and the evaluation result information display unit displays the first evaluation result information and the second evaluation result information, A predetermined numerical value of the evaluation criteria may be used for comparison and display.

In this way, by comparing and displaying each evaluation result information 230 with a predetermined numerical value for each execution log 300, which is the execution result of different automatic driving programs 170 under the same conditions, the user can It is possible to easily confirm superiority or inferiority of performance by comparing different types of automatic driving programs 170 (for example, automatic driving programs 170 from different developers).

Further, in the automatic driving program evaluation system of each embodiment, scenario information, which is information indicating conditions for driving the vehicle, is generated based on the part that defines the details of driving the vehicle extracted from the evaluation criteria information. The execution log receiving unit receives from the predetermined information processing system the execution log output as an execution result of the predetermined program executed based on the generated scenario information. The evaluation result information generating unit may generate the evaluation result information based on the generated scenario information, the received execution log, and the evaluation criterion information.

In this way, the scenario information 150 is created from the evaluation criteria information 210, and the evaluation result information 230 is generated based on the execution log 300, which is the execution result of the automatic driving program 170 executed based on this scenario information 150. A driving scenario that conforms to the reference information 210 can be generated. As a result, it is possible to appropriately evaluate virtually executed automatic driving based on objective criteria.

1 automatic operation program development system 10 automatic operation program execution system 20 automatic operation program evaluation system 170 automatic operation program 300 execution log 210 evaluation criteria information 205 execution log reception unit 207 evaluation result information generation unit 230 evaluation Result information

Claims (7)

with a processor and memory,
an evaluation criterion information storage unit for storing evaluation criterion information, which is evaluation criterion information for evaluating the content of driving of a predetermined automobile under predetermined conditions with a predetermined numerical value;
receiving from a predetermined information processing system an execution log, which is information including a history of driving the vehicle, output as a result of execution of a predetermined program for determining details of driving of the predetermined vehicle under the predetermined conditions; an execution log receiver;
an evaluation result information generating unit that generates evaluation result information indicating an evaluation of the driving of the predetermined vehicle under the predetermined conditions using a predetermined numerical value based on the received execution log and the evaluation criteria information;
an evaluation result information display unit for displaying the generated evaluation result information,
The evaluation result information display unit includes, as the evaluation result information, the items of the evaluation criteria related to the driving of the automobile, the numerical value indicating the extent to which the driving of the automobile violates the evaluation criteria, and the driving of the automobile. displaying the number of times a criterion was violated and where driving of the vehicle violated the criterion;
Automated driving program evaluation system. The execution log receiving unit outputs a first execution log under the predetermined condition and the predetermined program under the predetermined condition, which are output as execution results of the predetermined program, and the predetermined correction is performed. receiving a second execution log output as a program execution result;
The evaluation result information generating unit generates first evaluation result information based on the first execution log and second evaluation result information based on the second execution log,
The evaluation result information display unit compares and displays the first evaluation result information and the second evaluation result information according to a predetermined numerical value of the evaluation criteria.
The automatic driving program evaluation system according to claim 1 . with a processor and memory,
an evaluation criterion information storage unit for storing a plurality of evaluation criterion information, which is evaluation criterion information for evaluating the content of driving of a predetermined automobile under predetermined conditions with a predetermined numerical value;
a first execution log, which is information including a driving history of the vehicle, output as a result of execution of a first program for determining details of driving of the vehicle under the predetermined conditions; A second execution that is information including the driving history of the vehicle output as a result of execution of a second program that is a program different from the program and that determines the details of driving of the vehicle under the predetermined conditions. an execution log receiving unit that receives a log from a predetermined information processing system;
Based on the received first execution log and second execution log, and the evaluation criteria information, a predetermined evaluation of the driving of the predetermined automobile under the predetermined condition is performed based on the first execution log. First evaluation result information indicated by a numerical value, and second evaluation result information indicating an evaluation of driving of the predetermined automobile under the predetermined condition by a predetermined numerical value based on the second execution log, an evaluation result information generation unit that generates for each of a plurality of evaluation criteria;
an evaluation result information display unit that displays the generated first evaluation result information and second evaluation result information;
The evaluation result information display unit is configured to display an average value of evaluation values of the driving of the vehicle for the plurality of evaluation criteria based on the first evaluation result information, and based on the second evaluation result information. , comparing and displaying the average value of the numerical values of the driving evaluation of the automobile for the plurality of evaluation criteria;
Automated driving program evaluation system. The information processing system includes a driving situation display unit that displays the execution process of the predetermined program by an image showing the driving situation of the predetermined vehicle under the predetermined conditions,
The automatic driving program evaluation system includes an additional image information transmission unit that transmits image information indicating the content of the evaluation result information, which is displayed accompanying the image displayed by the information processing system, to the information processing system. prepare
The automatic driving program evaluation system according to claim 1 or 3 . a scenario information generation unit that generates scenario information, which is information indicating driving conditions of the vehicle, based on the part that defines the details of driving the vehicle, extracted from the evaluation criteria information;
The execution log receiving unit receives, from the predetermined information processing system, the execution log output as an execution result of the predetermined program executed based on the generated scenario information,
The evaluation result information generation unit generates the evaluation result information based on the generated scenario information, the received execution log, and the evaluation criteria information.
The automatic driving program evaluation system according to claim 1 or 3 . An automated driving program evaluation system comprising a processor and memory,
Evaluation standard information storage processing for storing evaluation standard information, which is evaluation standard information for evaluating the content of driving of a predetermined automobile under predetermined conditions with a predetermined numerical value;
receiving from a predetermined information processing system an execution log, which is information including a history of driving the vehicle, output as a result of execution of a predetermined program for determining details of driving of the predetermined vehicle under the predetermined conditions; execution log reception processing;
an evaluation result information generating process for generating evaluation result information indicating an evaluation of driving of the predetermined vehicle under the predetermined condition by a predetermined numerical value based on the received execution log and the evaluation criteria information;
an evaluation result information display process for displaying the generated evaluation result information;
and run
In the evaluation result information display process, as the evaluation result information, the item of the evaluation criteria related to the driving of the vehicle, a numerical value indicating the extent to which the driving of the vehicle violates the evaluation criteria, and displaying the number of times a criterion was violated and where driving of the vehicle violated the criterion;
Automated driving program evaluation method. An automated driving program evaluation system comprising a processor and memory,
Evaluation standard information storage processing for storing a plurality of evaluation standard information, which is evaluation standard information for evaluating the content of driving a predetermined automobile under predetermined conditions with a predetermined numerical value;
a first execution log, which is information including a driving history of the vehicle, output as a result of execution of a first program for determining details of driving of the vehicle under the predetermined conditions; A second execution that is information including the driving history of the vehicle output as a result of execution of a second program that is a program different from the program and that determines the content of driving of the predetermined vehicle under the predetermined conditions. an execution log reception process for receiving a log from a predetermined information processing system;
Based on the received first execution log and second execution log, and the evaluation criteria information, a predetermined evaluation of the driving of the predetermined automobile under the predetermined condition is performed based on the first execution log. First evaluation result information indicated by a numerical value, and second evaluation result information indicating an evaluation of driving of the predetermined automobile under the predetermined condition by a predetermined numerical value based on the second execution log, Evaluation result information generation processing for generating each of a plurality of evaluation criteria;
an evaluation result information display process for displaying the generated first evaluation result information and second evaluation result information;
In the evaluation result information display process, for the plurality of evaluation criteria based on the first evaluation result information, an average value of the numerical values of the driving evaluation of the automobile, and based on the second evaluation result information , comparing and displaying the average value of the numerical values of the driving evaluation of the automobile for the plurality of evaluation criteria;
Automated driving program evaluation method.

Images