JP2020048172A - Verification device for open facility, verification method, and program - Google Patents

Abstract

To provide a verification device for performing efficient function verification of an open network facility, a verification method, and a program.SOLUTION: A verification device 100 for an open facility includes: a test configuration management unit 110 for configuring virtual network devices by combining a hardware emulator for emulating operation of network device hardware with a network OS for controlling the network device; a test performing unit 120 for performing a necessary test on the virtual network devices from a predetermined test item; and a test result determination unit 130 for determining a result of a performed test.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a verification device, a verification method, and a program for performing efficient function verification of an open network facility.

  Opening of devices constituting a network is progressing. Opening here refers to disclosure of the physical specifications of the device. For example, in an organization called OCP (Open Compute Project), a design drawing of the inside of a network device is disclosed as “Hardware Specs”. Conventionally, network equipment hardware and the OS (software) that runs it have been provided by a single manufacturer. However, with the advent of white box switches, hardware and software must be procured separately. Became possible. It is anticipated that with the progress of openness, it will be possible to procure component units inside hardware from different vendors.

  Patent Literature 1 describes a technique for automating a test of a behavior according to a test procedure manual and reducing test man-hours without reducing the number of test items. Providing the expected operation determination unit is also useful for a function test of the target device.

  Patent Literature 2 relates to a performance prediction device that easily predicts the performance of a virtualization system even when one or more virtual machines operate on one physical machine, and includes a hardware layer and an application layer. By generating each performance model, it is possible to predict the performance of the virtualization system in which the dependency of each layer is reflected.

JP-A-2005-049659 JP 2014-132419 A

  As network devices become more open, the degree of freedom in procuring individual hardware components increases. On the other hand, when actually used as a network device, it is expected that performance tests and function confirmation tests will be very complicated between components and further in combination with software that operates them.

  Specifically, for example, an OS (software) usually mounted on a white box switch includes a device driver for executing an instruction on a component such as an ASIC, and operates on a general-purpose server without an ASIC. Need to prepare physical hardware. Also, in order to find the best combination from various components and software, it is necessary to prepare and test hardware for all of those combinations.

  However, although the technology described in Patent Document 1 or 2 can determine the expected operation and predict the performance of the virtualization system, it does not determine which of a number of combinations of hardware and software in an open facility is suitable. I can't. Also, it is not suitable for functional testing of virtualization systems.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a verification device, a verification method, and a program for performing efficient function verification of an open network facility.

  (1) In order to achieve the above object, the present invention has taken the following measures. That is, the verification device of the present invention is a verification device for open facilities, and combines a hardware emulator that emulates the operation of the hardware of a network device with a network OS that controls the network device to form a virtual network device. A test configuration management unit to be configured, a test execution unit that performs a necessary test from a predetermined test item on the virtual network device, and a test result determination unit that determines a result of the performed test. Prepare.

  This makes it possible to perform a performance test and a function confirmation test on a combination of the network device and the network OS without preparing the physical hardware of the open network device and installing the network OS.

  (2) In the verification device of the present invention, the hardware emulator is a combination of a plurality of hardware emulators that emulate the operation of components inside a network device.

  This makes it possible to prepare the components inside the open network device and configure the hardware, and to perform performance tests and functions on the combination of the components inside the network device and the network OS without installing the network OS. A confirmation test can be performed.

  (3) In the verification device of the present invention, the test configuration management unit sequentially configures a plurality of virtual network devices based on a predetermined combination pattern, and the test execution unit configures the plurality of virtual network devices. A necessary test is performed every time, and the test result determination unit compares test results of the plurality of virtual network devices and determines an optimum test result.

  Thereby, it is possible to easily compare using the test results of the virtual network devices without configuring all of the plurality of network devices.

  (4) Further, the verification device of the present invention extracts, from the predetermined combination patterns, combinations of combination patterns that have already been tested and combinations that do not need to be tested based on test results, The apparatus further includes a combination reduction unit that checks the predetermined combination pattern including the extracted combination, and the test configuration management unit does not configure the checked combination.

  As described above, since a combination that does not need to be tested is extracted and a combination pattern including the combination is not configured, for example, a combination having a high possibility that a test result is negative is not configured and tested, and the test is not performed. Time can be reduced.

  (5) In the verification device of the present invention, the test configuration management unit sequentially configures a virtual network composed of a combination of a plurality of virtual network devices based on a predetermined network configuration pattern. A necessary test is performed for each of the virtual networks, and the test result determination unit compares test results of the plurality of virtual networks and determines an optimal one.

  Accordingly, a network configuration including a combination of a plurality of network devices can be virtually configured by a combination of a plurality of virtual network devices without physically configuring the network configuration, and different network configurations can be easily compared. .

  (6) The verification device of the present invention further includes a test item reduction unit that determines a parent-child relationship that may have a logical inclusion relationship between test items based on test results of tests that have already been performed. When the test result of the test item determined as the parent by the test item reduction unit is negative, the test execution unit tests the test item determined to be a child of the test item determined by the test item reduction unit as the parent. Do not do.

  As described above, since a parent-child relationship that may have a logical inclusion relationship between test items is determined, and a test for a test item determined to be a child is not performed, a test having a high possibility of a negative test result is performed. There is no need to test items, and the test time can be reduced.

  (7) Further, the verification device of the present invention assigns weights to test items based on a test result of a virtual network device or a virtual network adopted by a user, and a test result output unit for outputting a judgment result of the test result judgment unit. A test result output unit, wherein the test result of the virtual network device or the test item of the virtual network to be output is negative and weighted, and the virtual network device or the virtual Display an alert with network output.

  As a result, when the user selects a virtual network device or a virtual network, necessary information is displayed, and the comparison speed can be increased.

  (8) The verification method of the present invention is a method of verifying an open facility, wherein a hardware emulator emulating the operation of hardware of a network device and a network OS controlling the network device are combined. Configuring a virtual network device on the machine, performing a necessary test from predetermined test items for the virtual network device, and determining a result for the performed test item. ,including.

  This makes it possible to perform a performance test and a function confirmation test on a combination of the network device and the network OS without preparing the physical hardware of the open network device and installing the network OS.

  (9) The program of the present invention is a program for verifying an open facility, and is a virtual network that combines a hardware emulator that emulates the operation of hardware of a network device and a network OS that controls the network device. A series of processes of configuring a device, performing a required test from a predetermined test item on the virtual network device, and determining a result for the performed test item On a computer.

  This makes it possible to perform a performance test and a function confirmation test on a combination of the network device and the network OS without preparing the physical hardware of the open network device and installing the network OS.

  ADVANTAGE OF THE INVENTION According to this invention, efficient function verification of open network equipment can be implemented.

FIG. 1 is a diagram illustrating an example of a schematic configuration of a verification device according to an embodiment. It is a figure showing an example of the concept of the verification system concerning this embodiment. 6 is a table illustrating an example of software stored in a software storage unit. 9 is a table illustrating an example of a combination pattern stored in a combination pattern storage unit. 6 is a table illustrating an example of a network configuration pattern stored in a network configuration pattern storage unit. 9 is a table showing examples of test items stored in a test item storage unit. 9 is a table illustrating an example of test results stored in a test result storage unit. FIG. 1 is a diagram illustrating an example of a schematic configuration of a verification device according to an embodiment. 9 is a table showing an example in a case where there is a combination pattern determined to be unnecessary. It is a table | surface which shows the example when there exists a test item judged to have a parent-child relationship. It is a table | surface which shows the example when there is a test item to which weight was given. (A), (b) is a figure which shows an example of the concept of the verification system which concerns on this embodiment. It is a lineblock diagram showing the verification device of an example. FIG. 4 is a sequence diagram illustrating an example of a test flow according to the embodiment. 9 is a flowchart illustrating an example of a test result determination algorithm of a test result determination unit according to the embodiment. 9 is a flowchart illustrating an example of an algorithm for determining a parent-child relationship in a test item reduction unit according to the embodiment. 9 is a flowchart illustrating an example of an algorithm for determining a parent-child relationship in a test item reduction unit according to the embodiment.

  The present inventors can emulate the hardware by describing the function or operation of the hardware of the network device or its components by a program and converting it into software, and the software of the network device can be emulated on the hardware emulator. By operating the, it is possible to perform tests based on the combination.Furthermore, by automating the combination test by listing the combinations and test items, it has been found that the test man-hours for the tests in various combinations can be greatly reduced. Invented the invention.

  That is, the verification device of the present invention is a verification device for open facilities, and combines a hardware emulator that emulates the operation of the hardware of a network device with a network OS that controls the network device to form a virtual network device. A test configuration management unit to be configured, a test execution unit that performs a necessary test from a predetermined test item on the virtual network device, and a test result determination unit that determines a result of the performed test. Prepare.

  As a result, the present inventors perform a performance test and a function confirmation test on a combination of a network device and a network OS without preparing physical hardware of an open network device and installing a network OS. Made it possible. Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings.

[Embodiment]
(Configuration of verification device)
FIG. 1 is a diagram illustrating an example of a schematic configuration of a verification device according to the present embodiment. The verification device 100 includes a test configuration management unit 110, a test execution unit 120, and a test result determination unit 130.

  The test configuration management unit 110 configures a virtual network device by combining a hardware emulator that emulates the operation of the hardware of the network device and a network OS (Operating System) that controls the network device. The hardware emulator may be a combination of a plurality of hardware emulators that emulate the operation of components inside the network device. FIG. 2 is a diagram illustrating an example of a concept of a verification system (described later) in a case where a virtual network device is tested using a combination of a hardware emulator and a network OS.

  The network device refers to, for example, a device having a network function such as a switch, a router, and a firewall. The components of the network device are components that are considered as a functional unit that performs packet processing, a communication protocol function, and the like of the network device, such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). ), NIC (Network Interface Card), GPU (Graphic Processing Unit), CPU (Central Processing Unit), and the like.

  The hardware emulator is a program that emulates the functions and operations of the hardware of the network device or its components. The network OS controls network devices. The hardware emulator and the network OS are prepared in advance and stored in the database (the software storage unit 210). Similar to the configuration of the network device by combining the components of the network device and the network OS, the network device can be virtually configured by combining the hardware emulator and the network OS of these components, and as shown in FIG. Testing can be performed.

  FIG. 3 is a table showing an example of software stored in the software storage unit 210. The software storage unit 210 stores a plurality of software such as a hardware emulator, which is a hardware version of network device software, and a network OS. The database of the software storage unit 210 and the like may be provided in the verification device 100 including those described later, but may be in another place. The database may be provided in a server on the cloud, for example, and the data may be acquired by a communication function.

  The combination of the hardware emulator and the network OS or the combination of the hardware emulator and the network OS of a plurality of components may be selected from a combination pattern that is defined and listed in advance. In this case, the combination pattern is stored in the database (combination pattern storage unit 220). FIG. 4 is a table showing an example of combination patterns stored in the combination pattern storage unit 220. As shown in FIG. 4, the combination pattern stored in the combination pattern storage unit 220 represents a single virtual network device including a hardware emulator and an OS of the network device. If the number of the combination pattern is different, the configuration of the virtual network device is different.

  In addition, the test configuration management unit 110 may sequentially configure a plurality of virtual network devices based on a predetermined combination pattern, and test each virtual network device.

  Further, the test configuration management unit 110 may sequentially configure virtual networks composed of a combination of a plurality of virtual network devices based on a predetermined network configuration pattern, and test each virtual network. In that case, the network configuration pattern is stored in the database (network configuration pattern storage unit 230).

  FIG. 5 is a table showing an example of a network configuration pattern stored in the network configuration pattern storage unit 230. The network configuration pattern stored in the network configuration pattern storage unit 230 is displayed using the combination pattern number indicating a single network device of the combination pattern storage unit 220. That is, the network configuration pattern stored in the network configuration pattern storage unit 230 represents a virtual network configured by a plurality of virtual network devices. However, when the test of the virtual network device alone and the test of the virtual network are performed continuously, one number can be displayed on the network configuration pattern as shown in FIG. If the numbers of the network configuration patterns are different, the configuration of the virtual network is different. As shown in FIG. 5, the network configuration pattern may be described so as to indicate a connection with "-", or the configuration may be described in a JSON or XML format (if the configuration is known, the description format is used). Does not matter).

  The test execution unit 120 performs a necessary test on a virtual network device from predetermined test items. Further, the test execution unit 120 performs a necessary test from a predetermined test item on a virtual network including a combination of a plurality of virtual network devices. The test items are stored in a database (test item storage unit 240). FIG. 6 is a table showing an example of test items stored in the test item storage unit 240. The test items describe the test contents and expected operations. The test execution unit 120 performs a test according to the test content described in the test item.

  The test result determination unit 130 determines the result of the performed test. In addition, the test result determination unit 130 compares test results of a plurality of virtual network devices and determines an optimum one. In addition, the test result determination unit 130 compares test results of a plurality of virtual networks and determines an optimal one. The determined test result is stored in the database (test result storage unit 250). The test results are stored for each combination pattern or each network configuration pattern. FIG. 7 is a table showing an example of test results stored in the test result storage unit 250. The test results are shown in FIG. As in 3, numerical values and the like may be used in addition to OK and NG. Various methods of the prior art can be used for the method of determining the test result, and any method may be used without departing from the scope of the present invention.

  The verification device 100 may further include one or more of the combination reduction unit 140, the test item reduction unit 150, the weighting unit 160, and the test result output unit 170. FIG. 8 is a diagram illustrating an example of a schematic configuration of a verification device 100 further including a combination reduction unit 140, a test item reduction unit 150, a weighting unit 160, and a test result output unit 170.

  The verification device 100 may include a combination reduction unit 140. The combination reduction unit 140 determines and extracts a combination that does not need to be tested based on a combination of combination patterns that have already been tested and a test result, among predetermined combination patterns, and includes a combination including the extracted combination. Check the specified combination pattern. If there is a checked combination, the test configuration management unit 110 does not configure the checked combination. Further, similar processing may be performed for a network configuration pattern.

  Thus, when constructing a new test configuration, the test configuration management unit 110 can prevent the combination pattern storage unit 220 from constructing any unnecessary ones. Also, the network configuration pattern storage unit 230 can be configured not to configure a network configuration including a combination that is unnecessarily checked in the combination pattern storage unit 220. As a result, the number of steps for performing a test in a configuration that is likely to fail can be saved.

  Note that the combination pattern storage unit 220 can be operated by the user, and can remove the unnecessary check if it is determined that the user is not unnecessary in the combination with the unnecessary check. In this case, a function may be provided in which the user's correction is true, and no unnecessary check is performed in the corresponding combination in the future. The same applies when there is an unnecessary column in the network configuration pattern storage unit 230.

  Specifically, when the verification device 100 includes the combination reduction unit 140, the test result determination unit 130 can perform a process of passing an incompatible combination pattern to the combination reduction unit 140. The combination reduction unit 140 retains the combination pattern, and when it is determined that the predetermined threshold value or more is inconsistent with the threshold value, the combination pattern or the network configuration pattern using the combination pattern is not tested. An unnecessary check is put in the storage unit 220. For example, assuming that the threshold value is three, the combination reduction unit 140 determines the combination of the software A and the hardware C in the combination pattern storage unit 220 when it is detected that the combination of the software A and the hardware C does not match three or more times. Check the unnecessary columns of the combination pattern including. FIG. 9 is a table illustrating an example in which there is a combination pattern determined to be unnecessary. FIG. 9 shows a state where the unnecessary column of the combination pattern is checked for the combination of software A and hardware C. Note that the threshold value in the above example may be set by the user.

  In the above example, it is determined that two combinations are unnecessary. However, three or more combinations may be determined. Also, the combination of hardware, software, and manufacturer information may be used as learning data, and unnecessary combinations may be deleted by machine learning.

  The verification device 100 may include a test item reduction unit 150. The test item reduction unit 150 determines a parent-child relationship that may have a logical inclusion relationship between test items based on the test results of the tests already performed. FIG. 10 is a table illustrating an example in which there is a test item determined to have a parent-child relationship. When it is determined that there is a parent-child relationship, as shown in FIG. 10, the number of the parent test item is entered in the parent column of the test item determined to be a child. Multiple parents may be entered in the parent column. Further, for example, another expression method such as a tree structure may be used.

  The relationship having a logical inclusion relationship between test items refers to a relationship between test items such that if A is NG, B is also NG. Specifically, for example, in BGP which is a communication protocol, if there is no connection, it is usually not possible to confirm the route information obtained therefrom. Therefore, if the item “confirm BGP connection” is NG, the test item “the number of BGP paths is as expected” must necessarily be NG. As described above, when there is a test item in which A is NG and B always becomes NG, A is treated as a parent and B is treated as a child.

  The test item reduction unit 150 may check the information in the test result storage unit 250 each time and determine the parent-child relationship of the test items. After the completion of the test, the test result storage unit 250 returns a completion notification and a test result to the test execution unit 120. In this way, if A is NG, the test execution unit 120 can determine that B need not be executed, and can prevent unnecessary test items from being executed. The parent-child relationship determined by the test item reduction unit 150 need not be a 100% correct logical inclusion relationship. That is, even if the test items that the test item reduction unit 150 has determined to have a parent-child relationship do not actually have a logical inclusion relationship, the user only needs to have a function that can correct the logical inclusion relationship.

  The test item reduction unit 150 returns the content determined as the parent or the child to the test item storage unit 240. The test item storage unit 240 can be operated by the user, and the user can input and correct the parent-child relationship in advance. In this case, a function may be provided in which the user's correction is true, and the relevant parent-child relationship is not corrected from the system in the future.

  The verification device 100 may include a weight assigning unit 160. The weight assigning unit 160 assigns a weight to a test item based on a test result of a virtual network device or a virtual network adopted by a user. The test result output unit 170 that outputs the determination result of the test result determination unit 130 outputs a virtual network device or virtual network device when the output virtual network device or virtual network test item test result is negative and weighted. Display an alert with network output.

  It is assumed that the user has a weight for each test item that is not adopted unless this test result passes. By providing the function of giving the test result, when the test result comparison output to the user is given, if the test item having the weight is NG, the user is alerted, and the comparison speed can be increased. . Specifically, for example, when the user inputs a recruitment result actually selected, the weighting unit 160 holds the recruitment result a plurality of times. The weighting unit 160 checks the weight column of the corresponding test item in the test item storage unit 240 when there is a test item that is always OK for a certain test item among the plurality of adoption results. Put on. FIG. 11 is a table showing an example in the case where there are test items to which weights are assigned. The weight assigning unit 160 can be operated by the user, and the user can input or modify the weight in advance. In this case, a function may be provided in which the user's correction is set to true and the corresponding weight is not corrected from the system in the future.

  The verification device 100 may include a test result output unit 170. The test result output unit 170 outputs the individual test results or the determination results of the test result determination unit 130. In addition, when the test result output unit 170 outputs the determination result including the test item whose test item to which the weighting unit 160 has given the weight is NG, the display is alerted together with the output of the virtual network device or the virtual network. .

(Configuration of verification system)
FIGS. 12A and 12B are diagrams illustrating an example of the concept of the verification system according to the present embodiment. FIG. 12A is a diagram illustrating an example of a concept of a verification system 300 in a case where a virtual network device is configured by a combination of a plurality of hardware emulators and a network OS and the virtual network device is tested.

  The verification system 300 includes the verification device 100, the computing resources 310, and the middleware 320. The computing resource 310 is a server or the like including a CPU, a memory, and the like. The middleware 320 means, for example, software for realizing virtualization such as KVM, and the technology of virtualization is not particularly limited. Further, the verification system 300 may include a necessary database 200 such as the software storage unit 210. Note that components C and E in FIG. 12A indicate hardware emulators of the corresponding components. FIG. 12B is the same.

  In the verification system 300, the verification device 100 and a virtual network device or a server or the like on which a virtual network is constructed may be provided in the same device, or the verification device 100 and the server or the like may be separate. . Further, for example, the verification system 300 itself is all on a cloud, and obtains a test result of a virtual network device or a virtual network built on a server on the cloud using a device having a communication function, an input / output function, and a display function. It is good also as composition which performs.

  FIG. 12B illustrates a configuration in which two virtual network devices are respectively configured by a combination of a plurality of hardware emulators and a network OS, a virtual network including the two virtual network devices is configured, and the virtual network is tested. It is a figure showing an example of the concept of a verification system in the case. As described above, the verification system 300 can perform a function test or the like of the virtual network device or the virtual network. The virtual network may be configured by three or more virtual network devices.

[Example]
(Configuration of verification device)
FIG. 13 is a configuration diagram illustrating the verification device of the example. In the present embodiment, the verification device 100 includes a test configuration management unit 110, a test execution unit 120, a test result determination unit 130, a combination reduction unit 140, a test item reduction unit 150, a weighting unit 160, and a test result output unit 170. Prepare. The verification device 100 also includes a software storage unit 210, a combination pattern storage unit 220, a network configuration pattern storage unit 230, a test item storage unit 240, and a test result storage unit 250, which are databases. In FIG. 13, each storage unit is displayed separately, but may be provided on one database 200. In addition, in addition to the above, a transmission unit 180, a reception unit 190, and a packet generation unit 400, and a transmission unit 410 and a reception unit 420 in the packet generation unit 400 are provided.

(Test execution flow)
FIG. 14 is a sequence diagram illustrating an example of a test flow according to the present embodiment. As pre-processing, the user stores and manages the program of the hardware emulator and the OS (software) of the network device in the software storage unit 210. Also, the combination of the hardware emulator to be tested and the OS of the network device is stored in the combination pattern storage unit 220 and managed. The “No.” of the combination pattern storage unit 220 is a single virtual network device including the hardware emulator to be tested and the OS of the network device. Store and manage. Further, the test items to be performed are stored in the test item storage unit 240 and managed (step S1). The test result storage section 250 refers to the information in the test item storage section 240 and stores the test contents and expected operations in advance.

  After completing the pre-processing, the test configuration management unit 110 checks the software storage unit 210, the combination pattern storage unit 220, and the network configuration pattern storage unit 230, and stores the corresponding test configuration on the computing resource 310 via the transmission unit 180. Build (step S2). After the completion of the construction, a request to execute a test is made to the test execution unit 120 (step S3). The test execution unit 120 creates test items with reference to the test item storage unit 240 (step S4), and executes the test items one by one (step S5).

  The test items can be broadly divided into functional tests and performance tests. The function test is an operation test related to the function of the virtual network device, and is a test for checking whether an expected operation is output when a certain process is executed. The performance test is a test related to the performance of the virtual network device, and is, for example, a test of quantitative evaluation for confirming whether or not a desired value is satisfied when a certain load is applied. For example, in FIG. No. 1 is a function test. 3 is a performance test. In the test execution in step S5, if a traffic load is particularly required in the performance test, the traffic is transmitted from the transmission unit of the packet generation unit in FIG. 13 and received by the reception unit to perform quantitative evaluation. And

  The test result storage unit 250 determines whether the result is OK or NG based on the test content and the content of the expected operation, and reflects the result in its own data (step S6). In the present embodiment, the test result storage unit 250 determines the result of each test, but the test result determination unit 130 may determine all the results, including the comparison of the test results. Thereafter, a test completion notification is transmitted to the test execution unit 120 (step S7). After the test is completed, the test execution unit 120 inputs the log and the like to the test result storage unit 250.

  If the created test item includes the next test, the test execution unit 120 performs the test on the test target (step S8). The same steps are repeated, and the test result storage unit 250 transmits a test completion notification to the test execution unit 120 (Step S9). If the test completion notification is for the last test item, the test execution unit 120 transmits a test execution completion notification to the test configuration management unit 110 (step S10). After the test execution completion notification is transmitted, the test configuration management unit 110 deletes the test configuration (Step S11).

  Then, the test configuration management unit 110 constructs the next test configuration on the computing resource 310 (Step S12), and requests the test execution unit 120 to execute the next test (Step S13). By repeating the same steps, the test result storage unit 250 transmits a test completion notification to the test execution unit 120 (step S14), and when the test completion notification is for the last test item, the test execution unit 120 A test execution completion notification is transmitted to the test configuration management unit 110 (step S15). After the test execution completion notification is transmitted, the test configuration management unit 110 deletes the test configuration (step S16), and when all the test configurations are completed (step S17), the test result determination unit 130 determines the test result. Is requested (step S18). The test result determination unit 130 determines the test result (step S19), and outputs the determined test result from the test result output unit 170 (step S20).

(Judgment and output of test results)
FIG. 15 is a flowchart illustrating an example of an algorithm for test result determination of the test result determination unit 130 according to the present embodiment. According to the algorithm shown in FIG. 15, the test result determination unit 130 transfers the test item and the test result to the test result output unit 170 when the combination pattern or the network configuration pattern and the test item that are considered to be suitable are required. The test result output unit 170 outputs the received combination pattern or network configuration pattern, and outputs a warning message if necessary. In addition, the test result determination unit 130 passes a combination pattern or a network configuration pattern that is not considered suitable to the combination reduction unit 140 according to the algorithm shown in FIG. Combination reduction section 140 stores this, and when the number of similar combinations is equal to or more than a predetermined number of times determined by the user, checks a combination including the combination. Note that X% in FIG. 15 can be specified by the user. The user can determine the compatibility between the hardware and the software based on the output of the test result output unit 170. The user inputs the combination pattern actually adopted based on the output result, and the weighting unit 160 holds the input adoption result.

  Specifically, the following is performed. First, the test result determination unit 130 calculates the ratio of test items that are OK to all test items (step T1). Next, it is determined whether the ratio is equal to or more than X% (step T2). If X% or more, it is determined whether or not the test item has a weight value (step T3). On the other hand, if it is not X% or more, the corresponding combination pattern or network configuration pattern is passed to the combination reduction section 140 (step T7). As a result of determining whether or not the test item has a weight value (step T3), if there is a weight value, it is determined whether or not all the test items having the weight value are OK (step T4). When all the test items having the weight value are OK, the corresponding combination pattern or network configuration pattern is passed to the test result output unit 170 (step T5). As a result of determining whether or not the test item has a weight value (step T3), if there is no weight value, the corresponding combination pattern or network configuration pattern is also passed to the test result output unit 170 (step T5). Also, as a result of determining whether or not all the test items having weight values are OK (step T4), when there is a test item which is NG, the corresponding combination pattern or network configuration pattern is determined together with the test result of the test item having weight value. The data is passed to the test result output unit 170 (step T6).

(Determination of parent-child relationship)
The test item reduction unit 150 determines a parent-child relationship between test items. FIG. 16 and FIG. 17 are flowcharts illustrating an example of an algorithm of the parent-child relationship determination of the test item reduction unit 150 of the present embodiment. The test item reduction unit 150 determines the parent-child relationship between test items according to the algorithm shown in FIG. 16 or FIG. Note that the specified number N in FIG. 16 or 17 can be specified by the user.

  Specifically, the following is performed. In the example of FIG. 16, first, the test item reduction unit 150 extracts and saves a result of the test item that is NG (step R1). Next, it is determined whether the test has been performed a specified number of times (step R2). If the test has not been performed the specified number of times, the test results are further accumulated. On the other hand, when the specified number of times has been performed, the test items having the closest NG number are grouped (step R3). Although various grouping methods are conceivable, for example, the grouping may be performed so that the number of elements becomes as equal as possible to a predetermined number K of groups, or a clustering method or the like may be used. In addition, the number of groups K may be specified by the user, or may be automatically changed according to the number of test items. Then, a group having the smallest number of NGs in each group is set as a parent, and others are set as children (step R3).

  In the example of FIG. 16, even if the parent is NG, the child is not always NG. Therefore, the example of FIG. 17 has a step of correcting that point. The example of FIG. 17 is the same as the example of FIG. 16 until grouping (steps P1 to P3). Next, the group with the smallest number of NGs in each group is set as a parent, and the others are set as temporary children (step P4). Then, it is determined whether or not the tentative child is always NG in the parent item of NG (step P5). As a result of the determination, when all the temporary children are always NG, the parent-child relationship is described in the test item storage unit 240 (step P6). As a result of the determination, if there is a provisional child having an item whose parent is NG and the provisional child is OK, the provisional child having an item whose parent is NG and the provisional child is OK is removed. , Etc. are described in the test item storage unit as a parent-child relationship (step P7).

  As described above, the verification device of the present invention can perform a performance test or a function confirmation test on a combination of a network device and a network OS without preparing physical hardware of the open network device and installing a network OS. It can be performed.

Reference Signs List 100 verification device 110 test configuration management unit 120 test execution unit 130 test result determination unit 140 combination reduction unit 150 test item reduction unit 160 weighting unit 170 test result output unit 180 transmission unit 190 reception unit 200 database 210 software storage unit 220 combination pattern Storage unit 230 Network configuration pattern storage unit 240 Test item storage unit 250 Test result storage unit 300 Verification system 310 Computing resource 320 Middleware 400 Packet generation unit 410 Transmission unit 420 Receiver

Claims (9)

A verification device for open facilities,
A test configuration management unit configured to configure a virtual network device by combining a hardware emulator that emulates the operation of the hardware of the network device and a network OS that controls the network device;
A test performing unit that performs a necessary test from a predetermined test item on the virtual network device;
A verification device comprising: a test result determination unit configured to determine a result of the performed test.   The verification device according to claim 1, wherein the hardware emulator is a combination of a plurality of hardware emulators that emulate the operation of a component inside a network device. The test configuration management unit sequentially configures a plurality of virtual network devices based on a predetermined combination pattern,
The test performing unit performs a necessary test for each of the plurality of virtual network devices,
The verification apparatus according to claim 1, wherein the test result determination unit compares test results of the plurality of virtual network devices and determines an optimum one. Among the predetermined combination patterns, a combination that does not need to be tested based on the combination of the combination patterns already tested and the test result is extracted, and the predetermined combination pattern including the extracted combination is extracted. Further equipped with a combination reduction unit that checks
The verification apparatus according to claim 3, wherein the test configuration management unit does not configure the checked combination. The test configuration management unit sequentially configures a virtual network composed of a combination of a plurality of virtual network devices based on a predetermined network configuration pattern,
The test performing unit performs a necessary test for each of the virtual networks,
The verification apparatus according to claim 1, wherein the test result determination unit compares test results of the plurality of virtual networks and determines an optimal one. Further comprising a test item reduction unit that determines a parent-child relationship that may have a logical inclusion relationship between test items based on test results of tests that have already been performed,
When the test result of the test item determined as the parent by the test item reduction unit is negative, the test execution unit determines the test item that is determined to be a child of the test item determined by the test item reduction unit as the parent. The verification device according to claim 1, wherein a test is not performed. A test result output unit that outputs a determination result of the test result determination unit,
A weighting unit that weights a test item based on a virtual network device or virtual network test result adopted by the user,
The test result output unit, when the test result of the test item of the virtual network device or the virtual network to be output is negative and weighted, displays a display that alerts the user along with the output of the virtual network device or the virtual network. The verification device according to any one of claims 1 to 6, wherein: It is a method of verifying open facilities,
Configuring a virtual network device on a physical machine by combining a hardware emulator that emulates the operation of the hardware of the network device and a network OS that controls the network device;
Performing a necessary test on the virtual network device from a predetermined test item;
Determining a result for the performed test item. It is a verification program for open facilities,
Processing of configuring a virtual network device by combining a hardware emulator that emulates the operation of the hardware of the network device and a network OS that controls the network device;
A process of performing a necessary test from a predetermined test item on the virtual network device;
A program for causing a computer to execute a series of processes including a process of determining a result for the performed test item.

Images