JP6430903B2 - Evaluation result analysis system and evaluation result analysis method - Google Patents

Description

  The present invention relates to an evaluation result analysis system and an evaluation result analysis method, and, for example, relates to an evaluation result analysis technique in a software control operation test.

  When checking the operation of the device under development, for example, a simulator or a real machine is used. However, the result of the operation check is that the test environment differs depending on the type of simulator, the actual machine, etc., and the output data format differs. For this reason, conventionally, dedicated devices with different functions and operation methods have been used as devices for evaluating the data of the operation check results.

  In this regard, for example, Patent Document 1 accepts a user instruction regarding selection of data to be output and an output format of the selected data, and a part of data obtained from a test system such as a simulator according to the instruction. An output data control apparatus that selects and outputs in a predetermined output format is disclosed. According to this, the user can easily set the selection of data obtained from the test system and the output format of the selected data. Therefore, it is possible to align the data formats of a plurality of operation confirmation results with different output data formats.

Japanese Patent No. 4967861

  According to Patent Document 1, it is certainly possible to share the format of output data. However, in Patent Document 1, only the data format is shared, and comparison of results measured in different test environments is not considered at all. With this, the operation confirmation result cannot be evaluated effectively.

  The present invention has been made in view of such circumstances, and provides a technique for enabling comparative evaluation of a plurality of operation confirmation results in different test environments.

  In order to solve the above problems, an evaluation result analysis system according to the present invention includes a function for converting a plurality of types of evaluation result data into a common data format, and a plurality of types of evaluation result data converted into a common data format. And a function of outputting the result of the comparison process.

  Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. The embodiments of the present invention can be achieved and realized by elements and combinations of various elements and the following detailed description and appended claims.

  The descriptions in this specification are merely exemplary, and are not intended to limit the scope of the claims or the application of the invention in any way.

  According to the present invention, a plurality of types of operation confirmation results (evaluation result data) from different test environments can be compared and evaluated. Therefore, for example, it becomes possible to compare an operation confirmation result by a simulator and an operation confirmation result using an actual machine.

It is a figure which shows schematic structure of the evaluation result analysis system according to embodiment of this invention. It is an example of the flowchart for demonstrating the content of the process which the evaluation result analysis system by this embodiment performs. It is a figure which shows the example of output data (evaluation result data of analysis object). It is a figure which shows the example of a comparison object selection screen. It is a figure which shows the example of a display form of the comparison information of an evaluation result. It is a figure which shows the outline | summary of an original data jump function. It is an example of the flowchart for demonstrating the processing content by a stored data (past data) comparison function.

  The present invention provides a technique that enables a plurality of types of evaluation results to be compared for the same evaluation item. For example, the present invention provides a technique that enables comparison between an operation confirmation result by a simulator in an upstream development process and an operation confirmation result using an actual machine. In addition, a plurality of types of operation confirmation results (evaluation result data) by different simulators can be compared.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be denoted by the same numbers. The attached drawings show specific embodiments and implementation examples based on the principle of the present invention, but these are for understanding the present invention and are not intended to limit the present invention. Not used.

  This embodiment has been described in sufficient detail for those skilled in the art to practice the present invention, but other implementations and configurations are possible without departing from the scope and spirit of the technical idea of the present invention. It is necessary to understand that the configuration and structure can be changed and various elements can be replaced. Therefore, the following description should not be interpreted as being limited to this.

  Furthermore, as will be described later, the embodiment of the present invention may be implemented by software running on a general-purpose computer, or may be implemented by dedicated hardware or a combination of software and hardware.

  In the following, each processing unit (input processing unit, operation processing unit, analysis unit, and display processing unit) as a program will be described with each processing in the embodiment of the present invention as a subject (operation subject). Since the processing determined by being executed by the CPU is performed using the memory and the communication port (communication control device), the description may be made with the processor as the subject. Further, the processing disclosed with the program as the subject may be processing performed by a computer such as a management server or an information processing apparatus. Part or all of the program may be realized by dedicated hardware, or may be modularized. Various programs may be installed in each computer by a program distribution server or a storage medium.

<Configuration of evaluation result analysis system>
FIG. 1 is a diagram showing a schematic configuration of an evaluation result analysis system 100 according to an embodiment of the present invention. The evaluation result analysis system 100 compares and analyzes a plurality of operation confirmation result data (sometimes referred to as evaluation result data) under different test environments, and outputs an analysis result.

The evaluation result analysis system 100 includes an input device 101 for inputting data, a memory 102 for storing various programs and information, a CPU (processor) 103 for reading various programs and information from the memory 102, and executing the programs. , An output device 104 for outputting the processing results, and a storage device 105 for accumulating and storing the processing results.

  The input device 101 corresponds to, for example, a keyboard, a mouse, a touch panel, a reading device that reads data from a portable memory such as a USB memory or a memory card. The input device 101 may be a communication device that receives data via a network.

  The memory 102 serves as an input processing unit (input processing program) 1021 for executing input processing (details will be described later) on input evaluation result data (output data) as a program, and to provide a user with a GUI. An operation processing unit (operation processing program) 1022 that executes the above processing, an analysis unit (analysis program) 1023 that compares and calculates the two evaluation result data, and a display that executes processing for displaying the comparison result and the like on the display screen A processing unit (display processing program) 1024 is stored. Further, the memory 102 stores file format information of data output by each tool (test environment) (for example, information on the resolution of the time axis label, the item of the signal label, and the arrangement order of the item). .

  The output device 104 corresponds to, for example, a display or a printer.

  The storage device 105 corresponds to, for example, an HDD (hard disk drive). In FIG. 1, the storage device 105 is provided in the system, but it may be installed remotely from the evaluation result analysis system 100 via a network.

  The evaluation result analysis system 100 includes, for example, output data (evaluation result data) A_10 that is an operation confirmation result in the test environment A and output data (evaluation result data) that is an operation confirmation result in the test environment B different from the test environment A. ) B_20. The data format of the output data A_10 and the output data B_20 may be the same or different. In FIG. 1, the output data to be input is two (two types), but may be three (three types) or more. Examples of output data formats include RAMScope (registered trademark) format (extension: csv), HEW Simulator format (extension: txt), CalDesk format (extension: csv), INCA format (extension: dxl), CANoe format (extension: asc), vHILS format (extension: csv), and the like are exemplified, but not limited thereto.

<Details of processing contents>
FIG. 2 is a flowchart for explaining details of processing contents in the evaluation result analysis system according to the embodiment of the present invention. Hereinafter, the processing contents will be described assuming that the input evaluation result data (operation check result) is two (two types), but the same applies to the case where there are three (three types) or more evaluation result data. .

(I) Step 101
When the user inputs two types of evaluation result data (operation check results) as output data A_10 and B_20, the input processing unit 1021 receives these evaluation result data. The input of the evaluation result data is executed, for example, when the user sets a portable memory in the evaluation result analysis system 100, reads a file corresponding to the output data stored in the portable memory, and is designated by the user. You may make it.

  FIG. 3 shows an example of output data (evaluation result data to be analyzed). 3A is an example of output data A_10, and FIG. 3B is an example of output data B_20. The output data may be evaluation result data (operation confirmation result) by a simulator, or evaluation result data (operation confirmation result) by an actual machine. As described above, in step 101, the input processing unit 1021 acquires a plurality of types of output data (evaluation result data) having different file formats.

(Ii) Step 102
The input processing unit 1021 interprets the data format of the output data acquired in step 101. Specifically, for example, the input processing unit 1021 checks the tool name included in the header information of the input output data, and acquires the file format information held in the memory 102 using the tool name as key information. Thus, the format of each output data (evaluation result data) is interpreted. Accordingly, in step 102, the input processing unit 1021 determines the time resolution (information indicating how long the signal value is measured), the signal label items, and the order of the items of each output data. Recognize what is

(Iii) Step 103
The input processing unit 1021 changes the data format of the output data A_10 and B_20 by rearranging the data name (item name), the data value (signal value), the time of the data change point, data interpolation, and thinning. Make common. If the signal labels are arranged differently, they may be unified to any one of the format of output data A_10, the format of output data B_20, or a format common to all data (a format different from the formats of output data A and B). For example, when the time resolution is different between the two output data A_10 and B_20, the signal value of the output data having the lower resolution (for example, see FIG. 3A) is interpolated, or the output data having the higher resolution (for example, the output data) The time scale is made common by thinning out the signal values of FIG. When the order of signal labels is different in the output data A_10 and B_20, the data (items) are rearranged to make the data order the same. Then, the input processing unit 1021 provides the data converted into a common data format to the analysis unit 1023.

(Iv) Step 104
The analysis unit 1023 generates a graph of the acquired output data A_10 and B_20 having a common format, and draws them on the display screen of the display device in cooperation with the display processing unit 1024 (401 and 402 in FIG. 4). reference). In addition, when there are a plurality of graphs to be drawn for each output data, the analysis unit 1023 uses the operation processing unit 1022 to generate selection buttons 403 and 404 and draws them as a UI on the display screen. .

(V) Step 105
The analysis unit 1023 accepts selection of a comparison target graph made by the user using the drawn UI. In FIG. 4, the user can select the graph group 401 of the output data A and the graph group 402 of the output data B, but the graph 4011 and the graph 4021, the graph 4012 and the graph 4022, and the graph 4013 and Since each graph 4023 is a pair (a graph corresponding to the same signal label (item)), when one of the pairs is selected, the other graph of the pair may be automatically selected.

(Vi) Step 106
The analysis unit 1023 identifies output data to be compared from the output data A_10 and B_20 converted into the common format by the input processing unit 1021 and selection information by the user. The analysis unit 1023 detects the difference between the two by comparing the output data to be compared. Then, the display processing unit 1024 displays the detected difference on the display screen. Thereby, the user can obtain comparison information of evaluation results. Since the difference between the output data A and B can be detected by the comparison calculation, the display processing unit 1024 may display only the difference on the display screen.

  FIG. 5 is a diagram illustrating a display form example of comparison information of evaluation results. FIG. 5A is an example in which comparison targets are displayed in parallel. FIG. 5B is an example in which a difference to be compared is displayed. In the parallel display, the graph of the output data A and the graph of the output data B are displayed in a vertical contrast so that the user can confirm the difference between them (see FIG. 5A). Further, in the comparison operation executed by the analysis unit 1023, a difference between the output data A and B may be taken. In this case, the output data A and B may be overlapped, and a portion where the difference occurs (mismatched portion) may be highlighted (see FIG. 5B).

  Then, the analysis unit 1023 includes the output data A_10 and B_20, the data before the format is made common, the data after the format is made common, and the graph data drawn based on the output data. It is stored in the storage device 105.

<Original data jump function>
FIG. 6 is a diagram showing an outline of the original data jump function. The user can execute the original data jump function in a state where the comparison graph 601 is displayed on the screen. The original data jump function is data in which the format of the original data (signal values of each item) of each output data at a specific location on the comparison graph (arbitrary time selected by the user) (for example, table format) This is the function to highlight above.

  When the user activates the original data jump function, a cursor dotted line 602 is displayed in the vertical direction over the output data A and B on the comparison graph 601. When the user moves the cursor in the horizontal axis (time axis) direction, the cursor dotted line 602 also moves in the horizontal axis direction in accordance with the movement of the cursor.

  When the original data jump function is activated, original data (table data) 603 and 604 corresponding to the output data A and B are displayed on the display screen. Then, the original data corresponding to the horizontal axis value (time value) where the cursor dotted line 602 is located is highlighted. By providing such an original data jump function, the user can not only visually compare output data using a graph but also compare output data based on numerical values.

<Stored data comparison function>
In the embodiment of the present invention, as described above, the process of comparing a plurality of evaluation result data (operation confirmation results) that are actually input is executed, but the present invention is not limited to this. In addition to this, for example, a process for comparing evaluation result data (evaluation result data accumulated in the storage device 105) that have been analyzed in the past, and evaluation result data that has been analyzed in the past, and new You may make it perform the process which compares with the evaluation result data input.

  FIG. 7 is an example of a flowchart for explaining the processing content by such a stored data (past data) comparison function.

(I) Step 201
When the stored data comparison function is activated in accordance with a user instruction, the input processing unit 1021 determines whether the user instruction is a comparison between data stored in the storage device 105, or newly input data and stored data. Judge whether it is a comparison with. If it is a comparison between stored data, the process proceeds to step 202. On the other hand, if it is a comparison between the accumulated data and new input data, the process proceeds to step 203.

(Ii) Step 202
The input processing unit 1021 acquires comparison result evaluation result data from the storage device 105 in accordance with a user instruction. After data acquisition, the process proceeds to step 206. In addition, the format of the acquired evaluation result data may be the same or different. Here, it is assumed that they are different. Further, there may be three (three types) or more data to be compared.

(Iii) Step 203
The input processing unit 1021 acquires evaluation result data (output data under a certain test environment) input by the user. The acquisition method is as described in step 101 of FIG. Note that the input evaluation result data may be one (one type) or more.

(Iv) Step 204
The input processing unit 1021 interprets the data format of the output data acquired in step 201. The specific contents of the interpretation of the data format are as described in step 102 of FIG.

(V) Step 205
The input processing unit 1021 acquires the other evaluation result data to be compared from the storage device 105 in accordance with a user instruction. At this time, the input processing unit 1021 also interprets the format of the evaluation result data acquired from the storage device 105 at the same time.

(Vi) Step 206
The input processing unit 1021 shares the format of the acquired evaluation result data. The specific contents of the common data format are as described in step 103 of FIG.

(Vii) Step 207
The analysis unit 1023 generates a graph of data having a common format, and draws it on the display screen of the display device in cooperation with the display processing unit 1024.

(Viii) Step 208
The analysis unit 1023 detects the difference between the two by comparing the evaluation result data to be compared. Then, the display processing unit 1024 displays the detected difference on the display screen. The form of comparison is as shown in FIG. The original data jump function (FIG. 6) can also be applied to the comparison result by the accumulated data comparison function.

  The stored data comparison function can be used, for example, when a part of a tool (test environment) program is changed. That is, by comparing the evaluation result data (output data A) by the tool before the program change stored in the storage device 105 with the evaluation result data (output data A ′) by the tool after the program change, the program change is performed. The impact can be confirmed.

<Summary>
(I) The evaluation result analysis system according to the embodiment of the present invention converts a plurality of types of evaluation result data (output data A and B) into a common data format, and converts the plurality of types of conversion into the common data format. Compare and output the evaluation result data. At this time, a process of drawing a plurality of types of evaluation result data converted into a common data format on a graph is executed. By doing in this way, the operation check result of each tool (test environment) can be effectively compared.

  Each of the operation confirmation results (evaluation result data) includes at least one signal item and a time change of the corresponding signal value as information. When multiple signal items are included, the evaluation result analysis system selects a graph to be compared for each of multiple types of evaluation result data, along with a graph of signal value temporal changes corresponding to the multiple signal items. Draw a selection display to make it possible. Then, a comparison process is executed in response to selection of the comparison target graph. By doing in this way, since an operation check result can be compared for every signal item, it becomes possible to verify an operation check result in more detail.

  Furthermore, in this embodiment, a plurality of types of input evaluation result data (for example, output data A and B) and a plurality of evaluation result data converted into a common data format (for example, a common data format) are stored in the storage device. Output data A and B) and at least one of the results of comparison processing (graph data in which the output data A and B are displayed in parallel, for example) are stored. Then, at least one piece of data acquired from the storage device may be compared with newly inputted evaluation result data. For example, the evaluation result data output from the test environment before the change (tool before the program change) is compared with the evaluation result data output from the test environment after the change (the same tool after the program change). Thereby, it becomes possible to check the influence of the program change on the operation confirmation result.

(Ii) This embodiment can also be realized by a program code of software that realizes a function. In this case, a storage medium in which the program code is recorded is provided to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention. As a storage medium for supplying such program code, for example, a flexible disk, CD-ROM, DVD-ROM, hard disk, optical disk, magneto-optical disk, CD-R, magnetic tape, nonvolatile memory card, ROM Etc. are used.

  Also, based on the instruction of the program code, an OS (operating system) running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing. May be. Further, after the program code read from the storage medium is written in the memory on the computer, the computer CPU or the like performs part or all of the actual processing based on the instruction of the program code. Thus, the functions of the above-described embodiments may be realized.

  Further, by distributing the program code of the software that realizes the functions of the embodiment via a network, it is stored in a storage means such as a hard disk or memory of a system or apparatus, or a storage medium such as a CD-RW or CD-R And the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage means or the storage medium when used.

  Finally, it should be understood that the processes and techniques described herein are not inherently related to any particular apparatus, and can be implemented by any suitable combination of components. In addition, various types of devices for general purpose can be used in accordance with the teachings described herein. It may prove useful to build a dedicated device to perform the method steps described herein. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined. Although the present invention has been described with reference to specific examples, these are in all respects illustrative rather than restrictive. Those skilled in the art will appreciate that there are numerous combinations of hardware, software, and firmware that are suitable for implementing the present invention. For example, the described software can be implemented in a wide range of programs or script languages such as assembler, C / C ++, perl, shell, PHP, Java (registered trademark).

  Furthermore, in the above-described embodiment, control lines and information lines are those that are considered necessary for explanation, and not all control lines and information lines on the product are necessarily shown. All the components may be connected to each other.

10 Output data A
20 Output data B
100 Evaluation Result Analysis System 101 Input Device 102 Memory 103 CPU (Processor)
104 Output device 105 Storage device 1021 Input processing unit 1022 Operation processing unit 1023 Analysis unit 1024 Display processing unit

Claims (5)

An evaluation result analysis system for analyzing evaluation result data output from a test environment, a memory storing a program for analyzing the evaluation result data;
A processor that reads the program from the memory and executes an analysis process on a plurality of types of input evaluation result data,
The processor is
A process of converting the format of the plurality of types of evaluation result data into a common data format;
A process of drawing the plurality of types of evaluation result data converted into the common data format on a graph;
A process of comparing the plurality of types of evaluation result data drawn on the graph ;
A process of outputting a result of the process to be compared;
An evaluation result analysis system that executes In claim 1 ,
Each of the evaluation result data indicates temporal changes in signal values corresponding to a plurality of signal items,
For each of the evaluation result data, the processor draws a selection display for enabling selection of a graph to be compared together with a graph of a temporal change in signal values corresponding to the plurality of signal items. An evaluation result analysis system that executes the comparison process in response to selection of a graph. In claim 1,
The processor stores at least one of the plurality of input evaluation result data, the plurality of evaluation result data converted into the common data format, and the result of the comparison process in a storage device,
The processor compares the at least one data acquired from the storage device with the newly input evaluation result data. In claim 3 ,
At least one data acquired from the storage device is evaluation result data output from the test environment before the change, and the newly input evaluation result data is an evaluation output from the test environment after the change. An evaluation result analysis system that is a result data and enables comparison of the evaluation result data before and after the test environment change. An evaluation result analysis method for analyzing evaluation result data output from a test environment,
A processor that reads the program from a memory that stores a program for analyzing the evaluation result data and executes analysis processing on the plurality of types of input evaluation result data. Performing a process of converting to a common data format;
The processor rendering the plurality of types of evaluation result data converted into the common data format on a graph;
The processor executing a process of comparing the plurality of types of evaluation result data drawn on the graph ;
The processor executing a process of displaying a result of the comparing process on a display screen;
Evaluation result analysis method including

Images