JP6409599B2 - Program, information processing method and information processing apparatus - Google Patents

Description

  The present invention relates to software testing.

  The software test is an operation for confirming whether the program operates correctly, does not operate unintentionally, or reaches a target quality.

  If it is a program developed by the person who performs the software test, create a test set (also called a test program) to cover the execution route of the program based on the source code, and confirm the effectiveness of the test set. Can do. Such a software test performed after understanding the internal structure of the program is called a white box test.

  On the other hand, when using OEM (Original Equipment Manufacturer) or OSS (Open Source Software), it is possible to develop a program without creating source code. Normally, a black box test is used to test a program created in this way. The black box test is a test for checking whether or not an output according to the specification can be obtained for various inputs by paying attention to the input and output of the program.

  By the way, the execution route of a program may be a different execution route even for the same input due to the difference in the state of computer resources or the internal state such as past access history. If you understand the source code of the program, you can specify the execution route. However, in the case of a program created using OEM, OSS, or the like, it is difficult to grasp the internal state, and it is difficult to specify the execution route. Therefore, there is a problem that it is not known whether the execution route is stable when a plurality of tests are executed for the same input.

JP 2012-133721 A

  Therefore, the objective of this invention is providing the technique for enabling the evaluation of the stability of the execution route of a program in one side.

  In one aspect of the present invention, when a specific program is executed a plurality of times, the execution time of the specific program is stored in the storage device for each of the executions of the plurality of times, and according to the execution time stored in the storage device The execution route of a specific program is stabilized according to the ratio of the number of executions belonging to the group generated by classification to the total number of executions calculated from the result of classification of multiple executions. It includes processing to determine whether or not.

  In one aspect, the stability of the program execution route can be evaluated.

FIG. 1 is a diagram illustrating a configuration of the information processing apparatus. FIG. 2 is a functional block diagram of the driver. FIG. 3 is a diagram for explaining the outline of the present embodiment. FIG. 4 is a diagram illustrating a processing flow of the prior setting process. FIG. 5 is a diagram for explaining the target range. FIG. 6 is a diagram for explaining the start block and the end block. FIG. 7 is a diagram illustrating an example of data stored in the range table. FIG. 8 is a diagram illustrating an example of data stored in the access table. FIG. 9 is a diagram showing a processing flow of processing executed by the trace processing unit when starting execution of the test program. FIG. 10 is a diagram illustrating an example of data stored in the mode storage area. FIG. 11 is a diagram illustrating a processing flow of the trace processing. FIG. 12 is a diagram illustrating an example of data stored in the termination address storage area. FIG. 13 is a diagram illustrating a processing flow of processing executed by the trace processing unit when the execution of the test program is finished. FIG. 14 is a diagram showing a processing flow of access flag setting processing. FIG. 15 is a diagram illustrating an example of data stored in the access table. FIG. 16 is a diagram for explaining suppression of the amount of data loaded into the memory. FIG. 17 is a diagram illustrating a processing flow of processing executed by the loading unit. FIG. 18 is a diagram illustrating an example of data stored in the map table and a reserved memory area. FIG. 19 is a diagram illustrating a processing flow of processing executed by the verification unit when starting verification processing. FIG. 20 is a diagram illustrating a processing flow of processing executed by the verification unit when starting each test. FIG. 21 is a diagram illustrating a processing flow of the verification processing. FIG. 22 is a diagram for explaining a dummy access request. FIG. 23 is a diagram illustrating a processing flow of verification processing. FIG. 24 is a diagram illustrating an example of data stored in the first result table. FIG. 25 is a diagram illustrating a processing flow of processing executed by the verification unit when each test is completed. FIG. 26 is a diagram illustrating a processing flow of processing executed by the verification unit when the verification processing ends. FIG. 27 is a diagram illustrating a processing flow of processing for collecting the results of the verification processing. FIG. 28A is a diagram for explaining the assignment of the process GID. FIG. 28B is a diagram illustrating an example of data stored in the second result table. FIG. 29 is a diagram showing an example of total execution time aggregation. FIG. 30 is a diagram illustrating a processing flow of processing for shaping the aggregated results. FIG. 31 is a diagram illustrating an example of data stored in the third result table. FIG. 32 is a diagram illustrating an example of disk access when the amount of usable memory is small. FIG. 33 is a diagram illustrating an example of disk access when the amount of usable memory is large. FIG. 34 is a diagram illustrating an example of the shaped result. FIG. 35 is a diagram illustrating an example of the shaped result. FIG. 36 is a diagram illustrating a processing flow of processing for evaluating the stability of the execution route. FIG. 37 is a functional block diagram of a computer.

  FIG. 1 shows the configuration of the information processing apparatus 1 in the present embodiment. The information processing apparatus 1 includes a work data storage unit 11 provided in a memory (for example, main memory) and a disk device 12 that is, for example, an HDD (Hard Disk Drive). The test program 14 calls the target program 15, which is a target program for the software test, N times (N is a natural number of 2 or more) times, and executes N tests. Input values in each test are the same. The kernel 13 program, test program 14, target program 15, and driver 10 program in the OS (Operating System) are loaded into the main memory and executed by a CPU (Central Processing Unit) in the information processing apparatus 1, so that Various functions described in the above are realized.

  FIG. 2 shows a functional block diagram of the driver 10. The driver 10 includes a trace processing unit 101, a load unit 103, and a verification unit 105. The trace processing unit 101 executes trace processing based on the data stored in the work data storage unit 11 and stores the processing result in the disk device 12. The load unit 103 loads data accessed from the target program 15 stored in the disk device 12 into the work data storage unit 11. The verification unit 105 executes verification processing based on the data stored in the disk device 12 by the trace processing unit 101 and the data loaded in the work data storage unit 11, and stores the processing result in the disk device 12.

  In addition, as a feature common to various programs, the following disk access can be mentioned. (1) Read the operating conditions of the program from the operating environment file on the disk device 12, (2) execute processing according to the operating conditions of the program, and update or add data on the disk device 12 according to the processing result (3) Use the area on the disk device 12 as a work area. (4) Record the contents of the error and the execution log in the disk device 12. In the present embodiment, attention is paid to such disk access.

  The outline of the processing executed in the present embodiment will be described with reference to FIG. In FIG. 3, an arrow between the target program 15 and the driver 10 represents a disk access execution instruction and a disk access result notification, and an arrow between the driver 10 and the disk device 12 represents disk access. In the present embodiment, first, the trace processing unit 101 in the driver 10 executes processing. The trace processing unit 101 checks the area of the disk device 12 accessed by the target program 15, and the execution time of the target program 15 (also called CPU time or CPU execution time. The unit is ms (millisecond)) from the kernel 13. A point to be acquired (hereinafter referred to as an acquisition point) is determined. In FIG. 3, the process of accessing the data 301 and the process of accessing the data 302 are acquisition points. The execution time is the time that has elapsed since the execution of the target program 15 was started.

  Next, in the load process, the load unit 103 in the driver 10 reads the data 301 and 302 accessed at the acquisition point from the area in the disk device 12 and copies them to the work data storage unit 11.

  In the verification process, the verification unit 105 in the driver 10 acquires the execution time from the kernel 13 at the acquisition point. At this time, the data 301 and 302 loaded in the work data storage unit 11 are accessed without accessing the data 301 and 302 in the area in the disk device 12. As a result, the time required for access can be shortened, so that the shortened time and the time required for obtaining the execution time can be offset. The acquired execution time and access destination address at the time of disk access are stored in the disk device 12.

  In the present embodiment, the verification process is executed for each of the N tests. Based on the time required to complete the execution of the target program 15 acquired in the verification process, the time required to reach each acquisition point, and the address string having the access destination address as an element, N times Tests are grouped. Since the user can check the variation of the execution route based on the number of groups and the number of tests belonging to each group, the stability of the execution route of the target program 15 can be evaluated.

  Next, the processing executed by the information processing apparatus 1 will be described more specifically with reference to FIGS. First, prior setting processing will be described with reference to FIGS. 4 to 8. This process is performed for each test program 14.

  The trace processing unit 101 receives designation of the start address and the end address of each target range from the user (FIG. 4: step S1). As shown in FIG. 5, the target range is a range of areas in the disk device 12 and is a range of areas accessed from the target program 15.

  The trace processing unit 101 identifies the start block, and stores the block number and address of the start block in the range table in the work data storage unit 11 (step S3). The start block is a block including the address X, where X is the youngest address in the target range (that is, the smallest address number). For example, in the example of FIG. 6, the block indicated by the arrow 601 is the start block.

  The trace processing unit 101 identifies the last block, and stores the block number and address of the last block in the range table in the work data storage unit 11 (step S5). The final block is a block that includes the address X, where X is the youngest address in the target range (that is, the largest address number). For example, in the example of FIG. 6, the block indicated by the arrow 602 is the final block.

  FIG. 7 shows an example of data stored in the range table. In the example of FIG. 7, the name of the test program 14 (hereinafter referred to as the test program name), the block number of the start block, the address of the start block, the block number of the last block, and the address of the last block are stored. The

  Returning to the description of FIG. 4, the trace processing unit 101 generates an entry in the access table in the work data storage unit 11 for each block from the start block to the last block (step S <b> 7).

  FIG. 8 shows an example of data stored in the access table. In the example of FIG. 8, a test program name, a block number, a range flag, and an access flag are stored. The range flag and the access flag are set to “ON” or “OFF”, but are set to “OFF” at the time of step S7.

  Returning to the description of FIG. 4, the trace processing unit 101 sets the range flag of the block included in the target range for which designation is accepted in step S <b> 1 among the range flags stored in the access table to “ON” (step S <b> 9). ). Then, the process ends.

  By executing the processing as described above, an access table in an initial state can be prepared.

  Next, a process executed by the trace processing unit 101 when starting the trace process will be described with reference to FIGS. 9 and 10.

  First, the trace processing unit 101 reads data from the access table corresponding to the test program name of the test program 14 to be executed (FIG. 9: Step S11).

  The trace processing unit 101 sets the information indicating the mode stored in the mode storage area in the work data storage unit 11 to “trace” (step S13). Then, the process ends. In the mode storage area, information representing the mode is stored in a mode as shown in FIG. 10, for example.

  By executing the processing as described above, the trace processing unit 101 can start the trace processing.

  Next, trace processing executed by the trace processing unit 101 will be described with reference to FIGS. 11 and 12. This process is executed every time a disk access is performed. The disk access parameters are the access destination address (here, the youngest address in the accessed area) and the size of the accessed data.

  First, the trace processing unit 101 determines whether the information indicating the mode stored in the mode storage area in the work data storage unit 11 is set to “verification” (FIG. 11: step S21). When the information indicating the mode is set to “verification” (step S21: Yes route), the process proceeds to step S101 in FIG. After step S101, the verification unit 105 called from the trace processing unit 101 executes the verification process. The verification process will be described later.

  When the information indicating the mode is not set to “verification” (step S21: No route), the trace processing unit 101 stores the information indicating the mode stored in the mode storage area in the work data storage unit 11 as “trace”. Is set (step S23).

  When the information indicating the mode is not set to “trace” (step S23: No route), the information indicating the mode is “normal”. Therefore, the driver 10 executes normal disk access according to the parameters (step S25). Then, the process ends.

  On the other hand, when the information indicating the mode is set to “trace” (step S23: Yes route), the trace processing unit 101 determines whether the access destination address is included in the block whose range flag is “ON”. (Step S27). The range flag is a flag stored in the access table (FIG. 8).

  If the access destination address is not included in the block whose range flag is “ON” (step S27: No route), the driver 10 executes normal disk access (step S25). Then, the process ends.

  On the other hand, when the access destination address is included in a block whose range flag is “ON” (step S27: Yes route), the trace processing unit 101 stores the end address stored in the end address storage area in the work data storage unit 11. The address is compared with the access destination address (step S29). In the end address storage area, for example, the end address is stored in a manner as shown in FIG. The end address is the last (that is, end) address in the area accessed in the previous disk access.

  The trace processing unit 101 determines whether the access destination address is an address next to the end address (step S31). By the processing in step S31, it is determined whether the previous disk access and the current disk access are a series of disk accesses. In the case of the first process, since the end address is not stored in the end address storage area, the process proceeds to the No route in step S31.

  When the access destination address is the next address after the termination address (step S31: Yes route), the process proceeds to step S37. On the other hand, when the access destination address is not the address next to the end address (step S31: No route), the trace processing unit 101 identifies the block number n of the block including the access destination address (step S33).

  The trace processing unit 101 increments the access number AN (n) for the block with the block number n by 1 (step S35).

  The trace processing unit 101 identifies the end address of the current disk access from the address and size of the access destination, and stores (overwrites here) in the end address storage area (step S37). Then, the process ends.

  By executing the processing as described above, the number of accesses can be counted for points at which the access destination addresses are discontinuous.

  Next, processing executed by the trace processing unit 101 when the trace processing is finished will be described with reference to FIGS.

  First, the trace processing unit 101 executes an access flag setting process (step S41). The access flag setting process will be described with reference to FIGS.

  The trace processing unit 101 sets a variable j representing the block number as j = 0 (FIG. 14: step S51).

  The trace processing unit 101 determines whether the access number AN (j) for the block with the block number j is larger than the threshold (step S53). The threshold is set in advance by the user. If AN (j) is equal to or smaller than the threshold value (step S53: No route), the process proceeds to step S57.

  When AN (j) is larger than the threshold (step S53: Yes route), the trace processing unit 101 sets the access flag for the block with the block number j among the access flags stored in the access table to “ON”. (Step S55).

  The trace processing unit 101 increments the variable j by 1 (step S57), and determines whether j is larger than the final block number (step S59). The final block number used in step S59 is the final number stored in the range table (FIG. 7).

  If j is less than or equal to the last block number (step S59: No route), the process returns to step S53 to process the next j. On the other hand, when j is larger than the final block number (step S59: Yes route), the trace processing unit 101 copies the access table in the work data storage unit 11 to the disk device 12. Then, the process returns to the calling process. By the access flag setting process, data as shown in FIG. 15, for example, is stored in the access table. As shown in FIG. 15, the range flag and the access flag are set to ON or OFF.

  Returning to the description of FIG. 13, the trace processing unit 101 deletes the end address in the end address storage area in the work data storage unit 11 (step S43). Further, the trace processing unit 101 sets the information indicating the mode stored in the mode storage area in the work data storage unit 11 to “normal” (step S45). Then, the process ends.

  If the acquisition point is determined as described above, the amount of data loaded into the memory can be suppressed. The suppression of the amount of data loaded into the memory will be described with reference to FIG. In FIG. 16, the target program 15 is a program for executing a process of “extracting a record satisfying a condition from huge size data, analyzing the content of the extracted record, and outputting the analysis result”. And According to the present embodiment, of the huge size data (data 1601 to 1604 here), the data 1601 is loaded into the memory, and the data 1602 to 1604 is not loaded into the memory. Therefore, loading data into the memory does not affect other processes (for example, delays occur in other processes).

  In addition, according to the present embodiment, disk access that does not occur frequently is also excluded from being loaded into the memory. Furthermore, disk accesses that are not related to disk access by the target program 15 (for example, regular patrol and log writing by the OS) are also excluded from being loaded into the memory. Thereby, it becomes possible to further narrow down the target area.

  Next, processing executed by the load unit 103 will be described with reference to FIGS. 17 and 18. This process is executed for each test program 14.

  First, the load unit 103 reads data in the access table in the disk device 12 (FIG. 17: step S71). When the access table remains in the work data storage unit 11, data in the access table in the work data storage unit 11 may be read.

  The load unit 103 counts the number b of blocks whose access flag stored in the access table is “ON” (step S73).

  The load unit 103 secures a memory area corresponding to the number of blocks b * block size in the work data storage unit 11 (step S75). Then, the load unit 103 copies the data in the block whose access flag is “ON” to the memory area secured in step S75 (step S77).

  The load unit 103 associates the block number of the block whose access flag is “ON” with the address on the memory where the data in the block is stored, and stores it in the map table in the disk device 12 (step S79). ). Then, the process ends.

  An example of a map table generated by executing the above processing is shown in the lower part of FIG. In the example shown in the lower part of FIG. 18, the block number and the address of the memory area are stored in association with each other. The upper part of FIG. 18 shows the secured memory area. Here, since the size of one block is 0x2000 (= 8192) (bytes), a memory area of 0x2000 * b (bytes) is secured.

  Next, a process executed by the verification unit 105 when starting the verification process will be described with reference to FIG. This process is performed for each test program 14.

  First, the verification unit 105 sets the information indicating the mode stored in the mode storage area in the work data storage unit 11 to “verification” (FIG. 19: step S81).

  The verification unit 105 reads data in the map table (step S83). Then, the verification unit 105 sets the variable TESTNUM representing the test number as TESTNUM = 0 (step S85). Then, the process ends.

  If the above processing is executed, verification processing can be started for the test program 14.

  Next, processing executed by the verification unit 105 when starting each test will be described with reference to FIG. This process is performed for each test.

  First, the verification unit 105 increments a variable TESTNUM that represents a test number by 1 (FIG. 20: step S91).

  The verification unit 105 sets the variable COUNT representing the acquisition point number as COUNT = 0 (step S93). Then, the process ends.

  If the process as described above is executed, the verification unit 105 can start the verification process for each test.

  Next, verification processing executed by the verification unit 105 will be described with reference to FIGS. This process is executed every time a disk access is performed. The disk access parameters are the access destination address (here, the youngest address in the accessed area) and the size of the accessed data.

  First, the verification unit 105 called by the trace processing unit 101 selects the first block among the blocks included in the accessed area in the disk device 12 based on the access destination address and the size of the accessed data. It identifies (FIG. 21: step S101).

  The verification unit 105 determines whether the range flag corresponding to the block identified in step S101 among the range flags stored in the access table in the disk device 12 is “ON” (step S103).

  If the range flag is not “ON” (step S103: No route), the driver 10 executes normal disk access according to the parameters (step S105). Then, the processing shifts to the processing in FIG. On the other hand, when the range flag is “ON” (step S103: Yes route), the verification unit 105 adds the access destination address to the address storage area in the work data storage unit 11 (step S107). The address storage area is an area (for example, an area in the kernel space) where an access destination address is accumulated.

  The verification unit 105 determines whether the access flag corresponding to the block identified in step S101 is “ON” (step S109). If the access flag is not “ON” (step S109: No route), the current disk access is not an acquisition point. Therefore, the driver 10 executes normal disk access according to the parameters (step S105). Then, the processing shifts to the processing in FIG.

  On the other hand, when the access flag corresponding to the block identified in step S101 is “ON” (step S109: Yes route), the verification unit 105 increments the variable COUNT representing the number of the acquisition point by 1 (step S111). .

  The verification unit 105 calculates the disk access division number DN from the address and size of the access destination (step S113). For example, the disk access division number DN is calculated by adding 1 to the value obtained by dividing the size by the amount of data accessible by one disk access. When an access destination address is required for calculating the amount of data accessible by one disk access, the access destination address is also used.

  The verification unit 105 sets a variable i representing the divided disk access number as i = 1 (step S115). Then, the verification unit 105 specifies a memory address corresponding to the access destination address from the map table, and accesses the memory area indicated by the specified memory address (step S117).

  The verification unit 105 issues a dummy access request to the disk device 12 (step S119). And the verification part 105 transfers to a sleep state (step S121). Then, the process proceeds to step S123 in FIG. The processes in steps S119 and S121 will be described with reference to FIG.

  When the work data storage unit 11 is accessed instead of accessing the disk device 12, the notification by the interrupt handler, which is normally performed after the access to the disk device 12 is not generated, is not performed. For this reason, the driver 10 cannot execute a process executed on extension of the notification from the interrupt handler. Therefore, in this embodiment, notification from the interrupt handler is performed by the method shown in FIG. As illustrated in FIG. 22, when the verification unit 105 of the driver 10 detects a disk access from the target program 15, the access point address is checked to determine whether it is an acquisition point. If it is an acquisition point, the verification unit 105 issues a dummy access request to the disk device 12 and shifts to the sleep state. Unlike the actual access request, the dummy access request does not read or write data, but the kernel 13 detects that a disk access has occurred. The interrupt handler notified from the kernel 13 that the disk access has occurred releases the sleep state of the verification unit 105 by notifying the driver 10.

  Shifting to the explanation of FIG. 23, the verification unit 105 cancels the sleep state according to the notification from the interrupt handler (step S123), and determines whether i = 1 is established (step S125). If i = 1 does not hold (step S125: No route), the driver 10 executes normal disk access according to the parameters (step S127). Then, the process proceeds to step S135.

  On the other hand, if i = 1 holds (step S125: Yes route), the verification unit 105 acquires the process ID (IDentifier) of the process of the target program 15 from the OS kernel 13 (step S129). Further, the verification unit 105 acquires the execution time corresponding to the process ID acquired in step S129 from the OS kernel 13 (step S131). The execution time is the time from the process execution start time to the processing time of step S131, but such information may not be acquired from the kernel 13. In such a case, the time information at the time of starting execution of the process may be stored after the process of step S93, and the difference from the time acquired from the kernel 13 in step S131 may be obtained.

  The verification unit 105 adds COUNT, the last address stored in the address storage area, and the execution time to the first result table in the work data storage unit 11 (step S133). However, when the first result table does not exist for the corresponding process ID (that is, when COUNT = 1), in step S133, the test program identifier, process ID, TESTNUM, COUNT, address, and execution time are set. A first result table containing is generated.

  FIG. 24 shows an example of data stored in the first result table. In the example of FIG. 24, the test program name (test program A in FIG. 24), process ID (1001 in FIG. 24), TESTNUM (No. 1 in FIG. 24), and COUNT (FIG. 24). , “1”, “2” and “3”), the address of the access destination, and the execution time are stored. Thus, the first result table is generated for each process ID.

  Returning to the description of FIG. 23, the verification unit 105 increments i by 1 (step S135), and determines whether i is greater than DN (step S137). If i is less than or equal to DN (step S137: No route), the process returns to step S125. On the other hand, when i is larger than DN (step S137: Yes route), the process ends.

  By executing the processing as described above, for each test, it is possible to record the time required to reach each acquisition point and the address string having the access destination address as an element. According to the present embodiment, since it is not necessary to modify the source code, it is possible to cope with a case where the source code is not disclosed.

  Next, processing executed by the verification unit 105 when each test is completed will be described with reference to FIG. This process is performed for each test.

  First, the verification unit 105 writes the address stored in the address storage area in the work data storage unit 11 to the disk device 12 (FIG. 25: step S141). Further, the verification unit 105 writes the first result table in the work data storage unit 11 into the disk device 12 (step S143).

  The verification unit 105 then tests the test program name, process ID, TESTNUM, and the time required to complete the test (ie, the time required to complete the execution of the target program 15; hereinafter referred to as the total execution time). Is written in the disk device 12 (step S145). Then, the process ends.

  By executing the processing as described above, the data stored in the disk device 12 can be used for subsequent processing.

  Next, a process executed by the verification unit 105 when the verification process is completed will be described with reference to FIG. This process is performed for each test program 14.

  First, the verification unit 105 writes back the data loaded in the work data storage unit 11 to the disk device 12 (FIG. 26: step S151), and releases the area in which the data was stored.

  The verification unit 105 sets the information indicating the mode stored in the mode storage area in the work data storage unit 11 to “normal” (step S153). Then, the process ends.

  If the above process is executed, the verification process for the test program 14 can be completed.

  Next, a process for collecting the results of the verification process will be described with reference to FIGS.

  First, the verification unit 105 acquires the process ID of a process that currently exists in the information processing apparatus 1 from the OS kernel 13 and generates a process ID list (FIG. 27: step S161). The process ID included in the process ID list includes a process ID of a resident process and a process ID of a non-resident process. Since the target program 15 is not executed at the time of execution of step S161, the process ID of the target program 15 is not included in the process ID list.

  The verification unit 105 extracts the process ID from the first result table (that is, the first result table generated for each process ID) in the disk device 12. Then, among the process IDs extracted from the first result table, process IDs not included in the process ID list are specified (step S163). The process ID extracted in step S163 includes the process ID of the target program 15. Therefore, the process ID specified in step S163 includes the process ID of the target program 15. The process ID specified in step S163 includes the process ID of a process that was resident during execution of the test program 14 but not resident at the time of processing in step S161.

  The verification unit 105 reads the result in the first result table for the identified process ID from the disk device 12 (step S165).

  The verification unit 105 performs grouping on the result read in step S165 so that those having at least a part of the address string are in the same group (step S167). FIG. 28A shows an example of grouping. In the example of FIG. 28A, a result 2801 for a process whose process ID is “211” and a result 2802 for a process whose process ID is “302” are shown. Since the address string (0x01008800,0x0156ab10,0x0100A800) in the result 2801 and the address string (0x01008800,0x0156ab10,0x0100A800) in the result 2802 match, grouping is performed so that they are in the same group.

  The verification unit 105 sets i = 1 as a variable representing the group number (step S169). Then, the verification unit 105 aggregates the results belonging to the group i, replaces the process ID belonging to the group i with a process GID (Group IDentifier), and stores the result after aggregation and replacement in the disk device 12 as a second result table. (Step S171).

  FIG. 28B shows an example of data stored in the second result table. In the example of FIG. 28B, the identifier of the test program, the process GID, the number indicating the acquisition point (that is, COUNT), the access destination address, and the execution time are stored. In FIG. 1 address string (0x01008800,0x0156ab10,0x0100A800) and No.1. 2 address strings (0x01008800, 0x0156ab10, 0x0100A800) completely match, but in the process of step S171, the result is stored in the same second result table even if at least part of the address strings match. The

  The verification unit 105 increments i by 1 (step S173), and determines whether i is larger than the number of groups (step S175). The number of groups is the number of groups generated by grouping in step S167. If i is equal to or less than the number of groups (step S175: No route), the process returns to step S171 to process the next group.

  On the other hand, if i is larger than the number of groups (step S175: Yes route), the verification unit 105 aggregates the total execution time stored in the disk device 12 in step S145 (step S177), and displays the aggregation result as the disk device 12. To store. Then, the process ends. The aggregation in step S177 is performed based on the grouping result in step S167.

  FIG. 29 shows an example of total execution time aggregation. In the example of FIG. 29, aggregation is performed by generating data including an identifier of a test program, a process GID, and a total execution time for each test.

  The process of the target program 15 is generated N times by execution of the test program 14, but the process ID of each time is different, so it is difficult to aggregate the results using the process ID. Therefore, if the address string is used as described above, the results for the target program 15 can be aggregated.

  Next, processing for shaping the aggregated result will be described with reference to FIGS. 30 to 35.

  First, the verification unit 105 identifies the number M of process GIDs included in the second result table in the disk device 12 (FIG. 30: step S181). As described above, since the second result table is generated for each process GID, M is the same as the number of the second result table.

  The verification unit 105 sets a variable i for counting the process GID as i = 1 (step S183). Then, the verification unit 105 identifies one unprocessed process GID among the process GIDs included in the result table in the disk device 12, and the result of the identified process GID is the second result corresponding to the process GID. Obtained from the table (step S185).

  The verification unit 105 changes the format of the result acquired in step S185 so that the execution time for the same address is in the same column (step S187). Then, the verification unit 105 stores the result after the format change in the disk device 12 as a third result table (step S189).

  FIG. 31 shows an example of data stored in the third result table. In the example of FIG. 31, the identifier of the test program, the process GID, the number indicating the acquisition point, the address of the access destination, the test number, and the execution time acquired at the acquisition point are stored. In the example of FIG. 31, an address string (0x01008800,0x0156ab10,0x0100A800,0x0156ab20,0x0100C800,0x0156ab30,0x0100E800,0x016100a0,0x015a0000) and an address string (0x01008800,0x016100a0,0x015a0000) appear.

  Returning to the description of FIG. 30, the verification unit 105 increments i by 1 (step S191), and determines whether i is greater than M (step S193). If i is equal to or less than M (step S193: No route), the process returns to step S185 to process the next i. On the other hand, when i is larger than M (step S193: Yes route), the process ends.

  If the above processing is executed, the verification processing result is changed to a format that is easier for the user to grasp. Note that the contents of the third result table may be output to a display device or the like after step S193 is completed.

  As described in the background section, even if the same target program 15 is executed, the execution route of the program may change depending on the resource state or the like in the information processing apparatus 1, and the address string may change as a result. .

  The variation of the address string will be described with reference to FIGS. FIG. 32 shows an example of disk access when the amount of usable memory is small. In the example of FIG. 32, since the amount of usable memory is small, the data 321 in the disk device 12 is read into the memory in three steps. Accordingly, a part of the data on the memory is discarded, so that the log 322 for recording the discard is also accessed. Further, the management file 323 and the log 324 for recording the completion are also accessed.

  FIG. 33 shows an example of disk access when the amount of usable memory is large. In the example of FIG. 33, since the amount of usable memory is large, the data 321 in the disk device 12 is read into the memory once. Therefore, since a part of the data on the memory is not discarded, access to the log 322 for recording the discarding is not performed. Similarly to the example of FIG. 32, the management file 323 and the log 324 for recording the completion are also accessed.

  If the trace process is executed when the amount of usable memory is small and the acquisition point is determined, the data shown in FIG. 34 is generated by the process shown in FIG. In the example shown in FIG. 34, execution times are shown for seven acquisition points. Since Test 1 and Test 2 are tests performed in a state where the amount of usable memory is small, execution times are acquired for all seven acquisition points. On the other hand, since the test 3 is performed in a state where the amount of usable memory is large, the data 321 is accessed only once and the log 322 is not accessed. Only the execution time has been acquired.

  On the other hand, if the trace processing is executed when the amount of usable memory is large and the acquisition point is determined, the data shown in FIG. 35 is generated by the processing shown in FIG. In the example shown in FIG. 35, the execution time is shown for three acquisition points regardless of whether the amount of usable memory is small or large. In Test 1 and Test 2, the process of accessing the log 322 is performed, so the execution time of Test 1 and Test 2 is longer than the execution time of Test 3. It can be seen that this cause is between the acquisition point for address 1 and the acquisition point for address 6.

  Next, processing for evaluating the stability of the execution route will be described with reference to FIG. Here, processing for evaluating the stability of the execution route for the target program 15 based on the third result table for the target program 15 will be described.

  First, the verification unit 105 reads data from the third result table in the disk device 12 (FIG. 36: step S201), and executes the first grouping for N tests based on the address string (step S203). In the first grouping, for example, grouping is performed so that tests having the same address string belong to the same group. This is because tests with the same address string are likely to have the same execution route.

  The verification unit 105 determines whether the grouping result in step S203 satisfies a predetermined condition (that is, the address string type converges to a predetermined number) (step S205). In step S205, for example, it is determined whether the condition that the ratio of the number of tests belonging to the group to the total number of tests is a predetermined value or more is satisfied for each group. The predetermined condition may be a condition that the number of groups is a predetermined number (for example, 2) or less. Further, the predetermined condition may be a condition that a ratio of the number of tests belonging to one or two groups to the total number of tests is equal to or greater than a predetermined value. When the grouping result in step S203 does not satisfy the predetermined condition (step S205: No route), the process proceeds to step S207.

  On the other hand, when the result of grouping in step S203 satisfies a predetermined condition (step S205: Yes route), the verification unit 105 determines the result of grouping performed in step S203 based on the total execution time. 2 grouping is executed (step S209). In step S209, the result of aggregation in step S177 is used. In the second grouping, for example, the grouping is performed so that tests having the same total execution time (here, the difference in execution time is shorter than a predetermined time) belong to the same group. This is because tests with the same total execution time are likely to have the same execution route.

  The verification unit 105 determines whether the grouping result in step S209 satisfies a predetermined condition (step S211). When the grouping result in step S209 does not satisfy the predetermined condition (step S211: No route), the process proceeds to step S207.

  On the other hand, when the grouping result in step S209 satisfies a predetermined condition (step S211: Yes route), the verification unit 105 executes the execution time to each measurement point with respect to the grouping result performed in step S209. Based on the above, the third grouping is executed (step S213). In step S213, grouping is performed so that tests having the same execution time to each measurement point (here, the difference in execution time is shorter than a predetermined time) belong to the same group. This is because a test with the same execution time to each measurement point is likely to have the same execution route.

  The verification unit 105 determines whether the grouping result in step S213 satisfies a predetermined condition (step S215). When the grouping result in step S213 does not satisfy the predetermined condition (step S215: No route), the verification unit 105 generates data indicating that the execution route of the target program 15 is not stable (step S207). .

  On the other hand, when the grouping result in step S213 satisfies the predetermined condition (step S215: Yes route), the verification unit 105 generates data indicating that the execution route of the target program 15 is stable (step S217). ).

  The verification unit 105 outputs the data generated in step S217 or S207 to a display device or the like (step S219). Then, the process ends.

  By executing the processing as described above, the stability of the execution route of the target program 15 can be evaluated. For example, if there are variations in the execution route even though the test is executed N times with the same input, it can be understood that the execution route of the target program 15 is not stable. In addition, when the test program 14 is executed a plurality of times for the purpose of confirming the completeness of the test program 14 and only a specific execution route passes, the user recreates the test program 14 accordingly. It becomes possible to take measures.

  Further, in the method for determining the state of the execution route using the OS call history, it takes time to record the OS call history. Therefore, the operation at the time of recording is the operation at the time of actual operation (that is, recording). This is different from the operation when no operation is performed. In addition, since the amount of history becomes large, the history cannot be held in the memory, and the history is stored in the disk, so that there is a problem that it takes a long time to complete the test set. However, these problems do not occur with the method of the present embodiment.

  Although one embodiment of the present invention has been described above, the present invention is not limited to this. For example, the functional block configuration of the driver 10 described above may not match the actual program module configuration.

  In addition, the data holding configuration described above is an example, and the above configuration is not necessarily required. Further, in the processing flow, the processing order can be changed if the processing result does not change. Further, it may be executed in parallel.

  Further, although the predetermined number is 2 in steps S205, S211, and S215, the predetermined number is not limited to 2.

  In the processing described with reference to FIG. 36, the determination is made based on the address string, the total execution time, and the execution time up to each measurement point. The determination may be made based on any two of them.

  The information processing apparatus 1 described above is a computer apparatus, and as shown in FIG. 37, a memory 2501, a CPU (Central Processing Unit) 2503, a hard disk drive (HDD: Hard Disk Drive) 2505, and a display device. A display control unit 2507 connected to 2509, a drive device 2513 for the removable disk 2511, an input device 2515, and a communication control unit 2517 for connecting to a network are connected by a bus 2519. An operating system (OS) and a program for executing the processing in this embodiment are stored in the HDD 2505, and are read from the HDD 2505 to the memory 2501 when executed by the CPU 2503. The CPU 2503 controls the display control unit 2507, the communication control unit 2517, and the drive device 2513 according to the processing content of the program, and performs a predetermined operation. Further, data in the middle of processing is mainly stored in the memory 2501, but may be stored in the HDD 2505. In the embodiment of the present invention, the above-described program for executing the processing described above is stored in a computer-readable removable disk 2511 and distributed, and installed in the HDD 2505 from the drive device 2513. In some cases, the HDD 2505 may be installed via a network such as the Internet and the communication control unit 2517. Such a computer apparatus realizes various functions as described above by organically cooperating hardware such as the CPU 2503 and the memory 2501 described above with the OS and programs such as the above-described programs.

  The embodiment of the present invention described above is summarized as follows.

  In the determination method according to the present embodiment, (A) when a specific program is executed a plurality of times, the execution time of the specific program is stored in the storage device for each of the plurality of executions, and (B) the storage device If the ratio of the number of executions belonging to the group generated by the classification to the total number of multiple executions is greater than or equal to a predetermined value in the result of classification of multiple executions according to the execution time stored in Including a process for determining that the execution route of the program is stable.

  When there are a plurality of execution routes of a specific program, the execution time is not constant, and a plurality of executions can be classified according to the execution time. Therefore, if it is described above, the stability of the execution route can be evaluated.

  In the process of storing the execution time of the specific program in the storage device, (a1) acquiring the first execution time that is the time from the start to the end of the specific program, and (a2) a plurality of the specific program For each point, a second execution time that is the time from the start of the specific program to the point is acquired. (A3) The first execution time and the second execution time acquired for each of the plurality of points May be stored in a storage device. In this way, it can be determined with higher accuracy whether or not they belong to the same group.

  In the process of storing the execution time of the specific program in the storage device, (a4) for each of the plurality of points, the address of the area in the second storage device that is accessed at the point is acquired and stored in the storage device It may be stored. In the process of classifying a plurality of executions, (b1) the plurality of executions may be classified based on the execution time and address stored in the storage device. In this way, it is possible to prevent a plurality of executions having the same execution time but different access destination addresses from being classified into the same group.

  The storage device described above may be a memory, and the second storage device described above may be a disk device. Then, this determination method is a process of (C) copying the data of the area in the second storage device that is accessed at each point for each of the plurality of points to the storage device before executing the specific program multiple times. May further be included. In the process of storing the execution time of the specific program in the storage device, (b2) data copied to the storage device may be accessed at each of the plurality of points. As described above, since it is not necessary to access the disk device, the time required to execute a specific program can be shortened. As a result, the time taken to acquire the execution time can be offset from the shortened time.

  Further, in the process of storing the execution time of the specific program in the storage device, (b3) when the data copied to the storage device is accessed at each of the plurality of points, a dummy is stored in the second storage device. An access request may be issued. In this way, the sleep state can be canceled by notification from the interrupt handler.

  Further, the first condition that the area accessed at each of the plurality of points is an area accessed a predetermined number of times or more by execution of a specific program may be satisfied. It is not realistic to obtain the execution time for all accesses that occur due to the execution of a specific program. So, as described above, you can narrow down the number of points.

  The address of the area accessed at each of the plurality of points may satisfy the second condition that the address is not continuous with the address of the area accessed at the point immediately before the point. When the addresses are continuous, the same processing often continues. Therefore, as described above, the execution time can be acquired at the point where the process is switched.

  The execution time may be a CPU (Central Processing Unit) execution time.

  A program for causing the processor to perform the processing according to the above method can be created, and the program can be a computer-readable storage medium such as a flexible disk, a CD-ROM, a magneto-optical disk, a semiconductor memory, a hard disk, or the like. It is stored in a storage device. The intermediate processing result is temporarily stored in a storage device such as a main memory.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Appendix 1)
On the computer,
When a specific program is executed a plurality of times, the execution time of the specific program is stored in a storage device for each of the plurality of executions,
The ratio of the number of executions belonging to the group generated by classification calculated from the result of classification of the plurality of executions according to the execution time stored in the storage device to the total number of the plurality of executions And determining whether the execution route of the specific program is stable,
Judgment program that executes processing.

(Appendix 2)
In the process of determining whether the execution route of the specific program is stable,
When the ratio is equal to or greater than a predetermined value, it is determined that the execution route of the specific program is stable.
The determination program according to attachment 1.

(Appendix 3)
In the process of storing the execution time of the specific program in the storage device,
Obtaining a first execution time which is a time from the start to the end of the specific program;
For each of the plurality of points in the specific program, obtain a second execution time that is the time from the start of the specific program to the point,
Storing the first execution time and the second execution time acquired for each of the plurality of points in the storage device;
The determination program according to attachment 1 or 2, which causes the process to be executed.

(Appendix 4)
In the process of storing the execution time of the specific program in the storage device,
For each of the plurality of points, an address of an area in the second storage device that is accessed at the point is acquired and stored in the storage device.
Let the process run,
In the process of determining whether the execution route of the specific program is stable,
Classifying the plurality of executions based on execution times and addresses stored in the storage device;
The determination program according to appendix 3, which causes the process to be executed.

(Appendix 5)
The storage device is a memory;
The second storage device is a disk device;
In the computer,
Before executing the specific program a plurality of times, for each of the plurality of points, the area data in the second storage device accessed at the points is copied to the storage device.
Let the process run further,
In the process of storing the execution time of the specific program in the storage device,
Accessing the data copied to the storage device at each of the plurality of points;
The determination program according to appendix 4, which causes the process to be executed.

(Appendix 6)
In the process of storing the execution time of the specific program in the storage device,
Issuing a dummy access request to the second storage device when accessing the data copied to the storage device at each of the plurality of points;
The determination program according to appendix 5, which causes the process to be executed.

(Appendix 7)
The determination program according to claim 3, wherein an area accessed at each of the plurality of points satisfies a first condition that the area accessed is a predetermined number of times or more by execution of the specific program.

(Appendix 8)
The address of the area accessed at each of the plurality of points satisfies the second condition that it is an address that is not continuous with the address of the area accessed at the point immediately before the point,
The determination program according to appendix 3, which causes the process to be executed.

(Appendix 9)
The determination program according to attachment 1, wherein the execution time is a CPU (Central Processing Unit) execution time.

(Appendix 10)
Computer
When a specific program is executed a plurality of times, the execution time of the specific program is stored in a storage device for each of the plurality of executions,
The ratio of the number of executions belonging to the group generated by classification calculated from the result of classification of the plurality of executions according to the execution time stored in the storage device to the total number of the plurality of executions And determining whether the execution route of the specific program is stable,
Judgment method to execute processing.

(Appendix 11)
A storage device;
When a specific program is executed a plurality of times, the execution time of the specific program is stored in the storage device for each of the executions of the plurality of times, and the plurality of times are executed according to the execution time stored in the storage device. Whether the execution route of the specific program is stable according to the ratio of the number of executions belonging to the group generated by classification to the total number of executions calculated from the result of classification of the execution of A determination unit for determining;
An information processing apparatus.

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 10 Driver 11 Work data storage part 12 Disk apparatus 13 Kernel 14 Test program 15 Target program 101 Trace processing part 103 Load part 105 Verification part

Claims (11)

On the computer,
When a specific program is executed a plurality of times, the execution time of the specific program is stored in a storage device for each of the plurality of executions,
The ratio of the number of executions belonging to the group generated by classification calculated from the result of classification of the plurality of executions according to the execution time stored in the storage device to the total number of the plurality of executions And determining whether the execution route of the specific program is stable,
Judgment program that executes processing. In the process of determining whether the execution route of the specific program is stable,
When the ratio is equal to or greater than a predetermined value, it is determined that the execution route of the specific program is stable.
The determination program according to claim 1. In the process of storing the execution time of the specific program in the storage device,
Obtaining a first execution time which is a time from the start to the end of the specific program;
For each of the plurality of points in the specific program, obtain a second execution time that is the time from the start of the specific program to the point,
Storing the first execution time and the second execution time acquired for each of the plurality of points in the storage device;
The determination program according to claim 1 or 2, wherein the process is executed. In the process of storing the execution time of the specific program in the storage device,
For each of the plurality of points, an address of an area in the second storage device that is accessed at the point is acquired and stored in the storage device.
Let the process run,
In the process of determining whether the execution route of the specific program is stable,
Classifying the plurality of executions based on execution times and addresses stored in the storage device;
The determination program according to claim 3, wherein the process is executed. The storage device is a memory;
The second storage device is a disk device;
In the computer,
Before executing the specific program a plurality of times, for each of the plurality of points, the area data in the second storage device accessed at the points is copied to the storage device.
Let the process run further,
In the process of storing the execution time of the specific program in the storage device,
Accessing the data copied to the storage device at each of the plurality of points;
The determination program according to claim 4, wherein the process is executed. In the process of storing the execution time of the specific program in the storage device,
Issuing a dummy access request to the second storage device when accessing the data copied to the storage device at each of the plurality of points;
The determination program according to claim 5, wherein the process is executed. The determination program according to claim 3, wherein an area accessed at each of the plurality of points satisfies a first condition that an area is accessed a predetermined number of times or more by execution of the specific program. The address of the area accessed at each of the plurality of points satisfies the second condition that it is an address that is not continuous with the address of the area accessed at the point immediately before the point,
The determination program according to claim 3, wherein the process is executed. The determination program according to claim 1, wherein the execution time is a CPU (Central Processing Unit) execution time. Computer
When a specific program is executed a plurality of times, the execution time of the specific program is stored in a storage device for each of the plurality of executions,
The ratio of the number of executions belonging to the group generated by classification calculated from the result of classification of the plurality of executions according to the execution time stored in the storage device to the total number of the plurality of executions And determining whether the execution route of the specific program is stable,
Judgment method to execute processing. A storage device;
When a specific program is executed a plurality of times, the execution time of the specific program is stored in the storage device for each of the executions of the plurality of times, and the plurality of times are executed according to the execution time stored in the storage device. Whether the execution route of the specific program is stable according to the ratio of the number of executions belonging to the group generated by classification to the total number of executions calculated from the result of classification of the execution of A determination unit for determining;
An information processing apparatus.

Images