JP2022122773A - Evaluation device and evaluation method - Google Patents

Abstract

To provide an evaluation device and an evaluation method that can efficiently perform a regression test.SOLUTION: An evaluation device according to an embodiment comprises: a control unit; and a storage unit. The control unit is configured to output input data on a simulation to a simulation device causing a simulation of control to implement between an Electronic Control Unit and a simulation device simulating a control object of the Electronic Control Unit; and collect the output data serving as a result of the simulation. The storge unit is configured to store the input data and a data set serving a set of output data. The control unit is configured to, when changing a program of at least one of the Electronic Control Unit and the simulation device, implement a simulation using the input data stored in the storage unit; and implement a regression test comparing the data sets before and after changing the program thereof.SELECTED DRAWING: Figure 2

Description

The present invention relates to an evaluation device and an evaluation method.

Conventionally, there is known a technique for evaluating control programs of various control devices installed in a vehicle by operating (simulating) a simulation device that simulates the operation of the vehicle in an environment where the actual vehicle does not exist. (See Patent Document 1, for example).

JP 2010-224972 A

However, conventionally, when a regression test is performed by changing the program of the simulation device or the control device, the operator has to manually check the past simulation results and the changed simulation results. It takes time.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an evaluation apparatus and an evaluation method capable of efficiently performing a regression test.

In order to solve the above-described problems and achieve the object, an evaluation device according to the present invention includes a control section and a storage section. The control unit outputs the input data of the simulation to a simulation device that executes a control simulation between a control device and a simulation device that simulates a controlled object of the control device, and outputs the simulation result. Collect output data. The storage unit stores a data set that is a set of the input data and the output data. When the program of at least one of the control device and the simulation device is changed, the control unit executes the simulation using the input data stored in the storage unit, and calculates the data before and after the change of the program. Perform regression tests comparing sets.

ADVANTAGE OF THE INVENTION According to this invention, a regression test can be performed efficiently.

FIG. 1A is a diagram illustrating a configuration example of a simulation system including an evaluation device according to an embodiment; FIG. 1B is a diagram illustrating an outline of an evaluation method according to the embodiment; FIG. 2 is a functional block diagram illustrating a configuration example of the evaluation device according to the embodiment; FIG. 3 is a flowchart illustrating a processing procedure of data set storage processing executed by the evaluation device according to the embodiment. FIG. 4 is a flow chart showing a processing procedure of overall processing executed by the evaluation device according to the embodiment.

Hereinafter, embodiments of an evaluation apparatus and an evaluation method disclosed in the present application will be described in detail with reference to the accompanying drawings. In addition, this invention is not limited by embodiment shown below.

First, with reference to FIGS. 1A and 1B, an overview of the evaluation method executed by the evaluation device according to the embodiment will be described. FIG. 1A is a diagram illustrating a configuration example of a simulation system including an evaluation device according to an embodiment; FIG. 1B is a diagram illustrating an outline of an evaluation method according to the embodiment;

As shown in FIG. 1A, the simulation system S includes an evaluation device 1, a simulation device 10, a simulation device 11, and an ECU (Electronic Control Unit) 12. As shown in FIG. The simulation system S is a so-called HILS (Hardware In the Loop Simulation), in which each part of the vehicle (engine, transmission, etc.) to be controlled by the ECU 12 is virtually created by the simulation device 11 and the simulation device 10, so that the actual vehicle can be simulated. It is a system that operates the ECU 12 and a virtual vehicle in conjunction with each other to perform logic evaluation of a control program incorporated in the ECU 12 and the like. In addition, ECU12 is an example of a control apparatus.

The evaluation device 1 is, for example, a terminal device operated by a user such as a developer of the ECU 12 . The evaluation device 1 is, for example, a desktop or laptop PC (Personal Computer), a smart phone, a tablet terminal, or the like.

The evaluation device 1 receives, for example, input data (data necessary for model calculation of the simulation device 10 ) in the simulation system S from the user and outputs the data to the simulation device 10 . The input data can be said to be simulation conditions, and the specific data contents are the operation data of the driver who controls the vehicle (each operation details such as accelerator, brake, steering, shift, etc.), the driving environment of the vehicle (weather, road type, etc.) ), driving performance (horsepower, acceleration performance, deceleration performance, vehicle body shape, vehicle type, etc.).

In addition, the evaluation device 1 acquires from the simulation device 10 the output data, which are the results of the simulation performed by the simulation device 10, the simulation device 11, and the ECU 12 based on the simulation conditions, and displays the output data on the screen. Thereby, the user can verify the operation of the ECU 12 by checking the simulation result displayed on the screen.

In addition, the evaluation apparatus 1 automatically executes a regression test for comparing past simulation results and current simulation results, which will be described later.

The simulation device 10 executes a simulation by the simulation device 11 and the ECU 12 according to the simulation conditions input from the evaluation device 1 and transmits output data, which is the result of the simulation, to the evaluation device 1 .

Specifically, based on the simulation conditions, the simulation device 10 generates, for example, a vehicle model in which behavior of the vehicle in the driving environment and driving performance is modeled using mathematical formulas or the like for each control object (engine, transmission, etc.) of the ECU 12. do. Then, the simulation device 10 inputs input data included in the simulation conditions to the generated vehicle model, and outputs behavior information regarding the behavior of the vehicle output from the vehicle model to the simulation device 11 .

The simulation device 11 is a device that simulates an object controlled by the ECU 12 . The simulation device 11 generates a simulation signal that simulates a vehicle signal based on behavior information input from the simulation device 10 and outputs the simulation signal to the ECU 12 . This simulated signal is a signal that simulates a signal output to the ECU 12 from a real object to be controlled (each part of the vehicle). That is, in the simulation process, a signal simulating a signal actually input from the controlled object is input to the ECU 12 .

The simulation device 11 receives a response signal from the ECU 12 to the simulation signal, converts the response signal into response information that can be processed by the simulation device 10 , and outputs the response information to the simulation device 10 . This response signal corresponds to the control signal actually output from the ECU 12 to the controlled object. Note that the simulation device 11 may have a function of performing high-speed arithmetic processing by hardware on a complicated model (motor model, etc.) that the simulation device 10 cannot calculate within a simulation period.

The simulation device 10 converts the acquired response information into output data that can be processed by the evaluation device 1 and outputs the output data to the evaluation device 1 . As described above, in the simulation system S, the simulation device 10 causes the ECU 12 and the simulation device 11 to execute vehicle control simulation according to the simulation conditions from the evaluation device 1 .

Here, a case where a regression test is performed by the evaluation method executed by the evaluation device 1 according to the embodiment will be described with reference to FIG. 1B. The term "regression test" as used herein is a test for verifying whether or not regression occurs when the program of at least one of the simulation device 10 and the ECU 12 is changed. The term "regression" as used herein refers to a state in which a part of a program is changed, causing problems in other parts (other programs, calculation results, etc.).

The program of the simulation device 10 is the above-described program for generating the vehicle model. Also, the program of the ECU 12 is a control program incorporated in the ECU 12 .

In the evaluation method according to the embodiment, when the program of at least one of the simulation device 10 and the ECU 12 is changed and a simulation is executed, a regression test is performed to compare data sets (sets of input data and output data) before and after changing the program. to run.

Specifically, as shown in FIG. 1B, the evaluation device 1 first stores a data set (before change), which is a set of input data and output data in the simulation performed before the program change, in the storage unit 3. Keep

Then, the evaluation device 1 performs a simulation after changing the program, and performs comparison processing for comparing the obtained data set (after change) with the data set (before change) stored in the storage unit 3. Run regression tests for program changes.

Then, the evaluation device 1 displays the regression test results on a display unit (not shown) or the like, thereby allowing the developer to confirm the regression test results. That is, in the evaluation method according to the embodiment, the regression test is executed by automatically comparing the data sets before and after the program change.

Accordingly, in the evaluation method according to the embodiment, since the developer does not need to manually compare the simulation results, the regression test can be performed efficiently.

In addition, in the evaluation method according to the embodiment, the regression test can be performed in parallel with the simulation after the program change, that is, the regression test can be performed in real time. As a result, for example, when a regression occurs, the simulation can be stopped immediately, so the developer can start working on solving the problem without waiting for the end of the simulation.

Further, in the evaluation method according to the embodiment, it is possible to perform a regression test by specifying a part of the data set and to specify the start timing of the regression test, which will be described later in detail.

Next, a configuration example of the evaluation device 1 according to the embodiment will be described with reference to FIG. FIG. 2 is a functional block diagram showing a configuration example of the evaluation device 1 according to the embodiment. In addition, in the block diagram of FIG. 2, only the components necessary for explaining the features of the present embodiment are represented by functional blocks, and the description of general components is omitted.

In other words, each component illustrated in the block diagram of FIG. 2 is functionally conceptual and does not necessarily need to be physically configured as illustrated. For example, the specific forms of distribution and integration of each functional block are not limited to those shown in the figure, and all or part of them can be functionally or physically distributed in arbitrary units according to various loads and usage conditions.・It is possible to integrate and configure.

As shown in FIG. 2 , the evaluation device 1 includes a control section 2 , a storage section 3 and an output section 4 . The evaluation device 1 is also connected to the simulation device 10 .

The output unit 4 outputs various information output from the control unit 2 . The output unit 4 is, for example, a display (display unit) that displays information, a speaker that outputs information as sound, or the like.

Here, the evaluation device 1 includes, for example, a computer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk drive, an input/output port, and various circuits.

The CPU of the computer functions as the reception unit 21, the execution unit 22, and the comparison unit 23 of the control unit 2 by reading and executing programs stored in the ROM, for example.

At least some or all of the functions of the reception unit 21, the execution unit 22, and the comparison unit 23 of the control unit 2 are configured by hardware such as ASIC (Application Specific Integrated Circuit) and FPGA (Field Programmable Gate Array). You can also

The storage unit 3 is, for example, a storage unit configured by a storage device such as a nonvolatile memory, a flash memory, or a hard disk drive. Data set information 31 and various programs are stored in the storage unit 3 .

The data set information 31 is information including a data set that is a set of input data to the simulation executed in the simulation system S and output data that is the result of the simulation.

Next, each function of the control unit 2 (accepting unit 21, executing unit 22 and comparing unit 23) will be described.

The reception unit 21 receives various types of information. The receiving unit 21 receives, for example, input data used for simulation. The input data includes, for example, operation data of the driver who controls the vehicle (operation details such as accelerator, brake, steering, shift, etc.), vehicle driving environment (weather, road type, etc.), driving performance (horsepower, acceleration performance, deceleration performance, body shape, vehicle type, etc.). In addition, the reception unit 21 receives information on the start timing for starting the regression test. The start timing information may be, for example, time information such as after a predetermined time, or may be information specifying the above-described driving environment, driving performance, and the like.

Further, when accepting input data, the accepting unit 21 accepts whether or not to execute a regression test based on the input data.

Further, the receiving unit 21 may receive a partial data range to be compared in the regression test in the data set. The data range may be a range based on a period such as time, and may be a range of data items (which data among data items such as input operation data, driving environment, and driving performance are to be compared). may Also, the data range may be the range from the above-described start timing to the designated end timing.

Note that a plurality of data ranges may be set. Specifically, the receiving unit 21 may receive a plurality of data ranges to be compared in the regression test in the data set.

Further, when part of the data in the data set is different due to a change in the program, the receiving unit 21 receives the different part of the data as an exclusion target to be excluded from the comparison target. Note that some data to be excluded may be determined automatically based on, for example, change information indicating the content of program changes.

Alternatively, the receiving unit 21 compares the input data received after changing the program with the input data before the change, and if different input data is detected, excludes the input data and the output data corresponding to the input data. It may be automatically detected as a target.

In addition, the receiving unit 21 may receive the allowable range of comparison in the regression test. In other words, the allowable range is a range in which the difference between the data sets before and after the program change is not regarded as a regression in the comparison processing by the comparison unit 23 at the later stage.

The execution unit 22 generates simulation conditions based on the input data received by the reception unit 21, and executes simulation according to the simulation conditions.

The execution unit 22 also stores a data set, which is a set of input data in the simulation and output data as a result of the simulation, in the storage unit 3 as data set information 31 .

Further, when the accepting unit 21 accepts execution of the regression test, the executing unit 22 outputs a data set, which is a set of input data and output data after the program is changed, to the comparing unit 23 . Specifically, the execution unit 22 executes the simulation after changing the program using the input data used for the difference in the simulation before changing the program.

When the accepting unit 21 accepts the execution of the regression test, the comparison unit 23 performs the regression test for comparing the data sets before and after the program change.

Then, the comparison unit 23 detects occurrence of regression when there is a difference between the data sets before and after the program change, more specifically, when the output data after the change changes compared to the output data before the change.

Then, when the comparison unit 23 detects the occurrence of a regression, the comparison unit 23 notifies the developer via the output unit 4 . The information notified by the output unit 4 includes, for example, information indicating the occurrence of a regression, information indicating the data location where the regression has occurred, information indicating the content of the regression, and the like. In this manner, the comparison unit 23 notifies the result of the regression test via the output unit 4, so that the developer can immediately grasp the regression information.

Further, when the execution unit 22 is executing the simulation after the program change in parallel with the regression test, the comparison unit 23 stops the simulation. Note that the comparison unit 23 may continue to perform the simulation even after the regression is detected, taking into account that the regression occurs at multiple locations.

Note that, when the reception unit 21 specifies the start timing of the regression test, the comparison unit 23 may execute the regression test triggered by the arrival of the start timing. In other words, the comparison unit 23 starts executing the regression test at the start timing specified in advance during the simulation period. As a result, even if the timings of input data and output data in the simulation are different, the regression test can be performed with high accuracy.

Further, when the acceptance unit 21 accepts that the regression test is performed only for a part of the data range, the comparison unit 23 may perform the regression test by comparing the data sets from the start timing to the end timing of the data range. good. That is, the comparison unit 23 executes a regression test that compares some data ranges in the data set. This allows, for example, comparison of only data sets that are likely to be affected by program changes.

In addition, when part of the data in the data set is different due to a change in the program, the comparison unit 23 excludes the different part of the data from comparison targets of the regression test. Specifically, the comparison unit 23 executes a regression test that compares the data sets received by the reception unit 21 and excluding the data to be excluded. As a result, it is possible to avoid, with high accuracy, erroneously detecting the different data as a regression when some of the data in the data set are different.

Further, when performing a regression test, the comparison unit 23 may determine that the data sets are the same (no regression) if the difference between the data sets before and after the program change is within the allowable range accepted by the acceptance unit 21 . As a result, for example, even if the data set changes values for each measurement like an analog signal, it is possible to perform a regression test with high accuracy.

Next, a procedure of processing executed in the evaluation device 1 according to the embodiment will be described with reference to FIGS. 3 and 4. FIG. FIG. 3 is a flow chart showing a processing procedure of data set storage processing executed by the evaluation device 1 according to the embodiment. FIG. 4 is a flow chart showing the processing procedure of the overall processing executed by the evaluation device 1 according to the embodiment.

First, data set storage processing will be described with reference to FIG.

As shown in FIG. 3, the receiving unit 21 receives input data for simulation (step S101). Subsequently, the execution unit 22 executes the simulation by transmitting the input data to the simulation device 10 (step S102). Subsequently, the execution unit 22 stores the data set in which the input data and the output data, which is the result of the simulation, are associated as the data set information 31 in the storage unit 3 (step S103), and ends the process.

Next, the overall processing performed by the evaluation device 1 will be described with reference to FIG.

As shown in FIG. 4, first, the receiving unit 21 receives input data for simulation (step S201). Subsequently, the execution unit 22 executes the simulation by transmitting the input data to the simulation device 10 (step S202).

Subsequently, the comparison unit 23 determines whether or not it is time to start the regression test during execution of the simulation (step S203). When the start timing arrives (step S203: Yes), the comparison unit 23 compares the data set including the output data collected from the start timing with the past data set before the program change (step S204).

In addition, the comparison part 23 returns to step S202, when the start timing has not arrived (step S203: No).

Subsequently, the comparison unit 23 determines whether or not a regression has occurred in the regression test results (step S205). If a regression occurs (step S205: Yes), the comparison unit 23 stops the simulation (step S206).

Subsequently, the comparison unit 23 notifies the result of the regression test (step S207), and terminates the process. In the result notification, the occurrence of the regression and the location of the regression in the data set are notified.

On the other hand, in step S205, if no regression has occurred (step S205: No), the comparison unit 23 determines whether or not the simulation has ended (step S208).

When the simulation ends (step S208: Yes), the comparison unit 23 notifies the end of the simulation through the output unit 4 (step S209), and ends the process. Note that the notification of the end of the simulation also notifies that no regression has occurred.

On the other hand, in step S208, if the simulation has not ended (step S208: No), the comparison unit 23 returns to step S204.

It should be noted that if a regression occurs in step S205, the simulation may not be stopped immediately, but the simulation may be paused to ask the user whether to stop the simulation.

As described above, the evaluation device 1 according to the embodiment includes the control section 2 and the storage section 3 . The control unit 2 outputs simulation input data to a simulation device 10 that executes a control simulation between a control device (ECU 12) and a simulation device 11 that simulates a controlled object of the control device, and outputs simulation results. Collect the output data that is The storage unit 3 stores data sets that are pairs of input data and output data. When the control unit 2 changes the program of at least one of the control device and the simulation device 10 and executes the simulation, the control unit 2 executes a regression test for comparing data sets before and after changing the program. This allows efficient regression testing.

Further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the invention are not limited to the specific details and representative embodiments so shown and described. Accordingly, various changes may be made without departing from the spirit or scope of the general inventive concept defined by the appended claims and equivalents thereof.

1 evaluation device 2 control unit 3 storage unit 4 output unit 10 simulation device 11 simulation device 12 ECU
21 reception unit 22 execution unit 23 comparison unit 31 data set information S simulation system

Claims (6)

Input data for the simulation is output to a simulation device that executes a control simulation between a control device and a simulation device that simulates a control target of the control device, and output data as a result of the simulation is collected. a control unit;
a storage unit that stores a data set that is a set of the input data and the output data;
with
The control unit
When the program of at least one of the control device and the simulation device is changed, the simulation is executed using the input data stored in the storage unit, and the data set is compared before and after the program is changed. An evaluation device characterized by executing a test. The control unit
2. The evaluation apparatus according to claim 1, wherein said regression test is executed by comparing a partial data range of said data set. The regression test is
It is executed in parallel with the simulation by the changed program,
The control unit
3. The evaluation apparatus according to claim 1, wherein the regression test is started at a start timing specified in advance during the period in which the simulation is performed. The control unit
4. The method according to any one of claims 1 to 3, wherein if part of the data in the data set is different due to a change in the program, the part of the data is excluded from comparison in the regression test. Evaluation device. further comprising an output unit for outputting information,
The control unit
The evaluation apparatus according to any one of claims 1 to 4, wherein the result of the regression test is output via the output unit. Input data for the simulation is output to a simulation device that executes a control simulation between a control device and a simulation device that simulates a control target of the control device, and output data as a result of the simulation is collected. a control step;
a storage step of storing a data set, which is a set of the input data and the output data, in a storage unit;
including
The control step includes
When the program of at least one of the control device and the simulation device is changed, the simulation is executed using the input data stored in the storage unit, and the data set is compared before and after the program is changed. An evaluation method characterized by running a test.

Images