KR20200129232A - Test automation framework system for medical software and method of test automation - Google Patents

Abstract

Disclosed is a test automation system for medical software. According to the present invention, the test automation system comprises: a development automation machine performing an automation test or ordering a build machine to perform the automation test in accordance with a test request from an integrated development environment; the build machine performing building in accordance with a build request from the integrated development environment and performing the automation test in accordance with a test request of the development automation machine; and a shape management machine storing a performance result of the automation test and providing a test case for performing the automation test to the development automation machine or the build machine. Accordingly, a test automation framework system for medical software is provided, thereby providing an effect of providing test automation functions considering major automation elements of the medical software.

Description

Test automation framework system for medical software and test automation method {TEST AUTOMATION FRAMEWORK SYSTEM FOR MEDICAL SOFTWARE AND METHOD OF TEST AUTOMATION}

The present application relates to a test automation framework for medical software, and more particularly, to a test automation framework system and a test automation method for medical software.

Devops is a compound word for software development and operation. DevOps is a concept to enable rapid development and distribution of software products/services by integrating work processes that were separately performed by each department such as the existing development team and quality team in software development and enabling quick feedback between them.

One of the important elements of DevOps is the issue of automation. When software test automation is implemented, it is possible to continuously perform repetitive tasks to quickly provide feedback on test results and improvements to users, thereby reducing test time. In addition, test accuracy can be improved through extensive testing through test automation and the use of proven tools.

However, the current test automation application degree has a problem in that the entire software test is not sufficiently performed because only test automation is implemented for some parts rather than the entire software development process. In particular, in the case of medical software, since a high level of software accuracy and reliability is required, a higher level of test automation is required during software development.

Patent Document 1: Unexamined Patent Publication No. 10-2007-0037808 (2007.04.09)

In order to solve this problem, an object of the present invention is to provide a test automation framework system and a test automation method for medical software.

According to an embodiment of the present invention, a test automation system for medical software is presented. The system includes: a developer fairy tale machine for performing an automated test or instructing a build machine to perform the automated test according to a test request from an integrated development environment; A build machine that performs a build according to a build request from the integrated development environment, and performs the automated test according to a test request from the developer fairy tale machine; And it may include a configuration management machine for storing the execution result of the automated test.

According to an embodiment of the present invention, an automated smoke test method for medical software is provided. The method includes determining a subset of test cases associated with a reference model of the medical software; Generating input image data based on the subset of test cases; Performing an automation test through a UI automation tool using the input image data; And storing and managing the execution result of the automated test in the automated test storage.

According to an embodiment of the present invention, a UI automation test method for medical software is provided. The method includes determining test cases for all interface functional elements of the medical software; Performing an automated test on the test cases through a UI automation tool; And storing and managing the execution result of the automated test in the automated test storage.

According to an embodiment of the present invention, a method for automating interoperability testing for medical software is provided. The method includes determining test cases based on a data standard associated with the medical software; Performing an automated test on the test data of the test cases through a test oracle; And storing and managing the execution result of the automated test in the automated test storage.

According to an embodiment of the present invention, a computer program stored in a computer-readable medium is provided, and the computer program may include computer-executable instructions for executing the automated test methods of the present disclosure.

According to the present disclosure, by providing a test automation framework system for medical software, there is an effect of providing test automation functions in which major automation elements of medical software are reflected.

Further, according to the present disclosure, there is an effect of providing a smoke test automation function that enables test automation of main data and main functions according to a representative reference model.

Further, according to the present disclosure, there is an effect of providing a UI automation test function that enables test automation for all interface functional elements.

In addition, according to the present disclosure, there is an effect of providing an interoperability test automation function capable of verifying compatibility with heterogeneous equipment.

1 is a diagram showing a test automation framework system for medical software according to an embodiment of the present invention.
2 is a view showing an automated smoke test method for medical software according to an embodiment of the present invention.
3 is an exemplary view visually showing steps 220 and 230 of FIG. 2.
4 is a diagram illustrating a UI automation test method for medical software according to an embodiment of the present invention.
FIG. 5 is an exemplary diagram illustrating an order in which test cases are tested in the UI automation test of FIG. 4.
6 is a diagram illustrating a method for automating interoperability testing for medical software according to an embodiment of the present invention.
FIG. 7 is an exemplary diagram illustrating a test oracle for an automated interoperability test according to FIG. 6.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. First of all, in adding reference numerals to elements of each drawing, it should be noted that the same elements have the same numerals as possible even if they are indicated on different drawings. In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

Various aspects of the invention are described below. It is to be understood that the inventions presented herein may be embodied in a wide variety of forms and any particular structure, function, or all of them presented herein are exemplary only. Based on the inventions presented herein, a person of ordinary skill in the art to which the present invention pertains may implement one aspect presented herein independently from any other aspects, and two or more such aspects may be implemented in various ways. You will understand that it can be combined with. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects described herein. In addition, such an apparatus may be implemented or such a method may be practiced using a structure, function, or structure and functionality other than or in addition to one or more aspects described herein.

The test automation framework system and the medical software to which the test automation method of the present disclosure is applied will be exemplarily described as dental CAD (Computer Aided Design) software for convenience of description. Such dental CAD software may be software designed to design a dental prosthesis based on an image scanned with an oral scanner and to produce a prosthesis through a prosthesis processor. However, those of ordinary skill in the technical field to which the present disclosure belongs is not limited to dental CAD software as medical software to which the present disclosure can be applied, and the technical idea of the present disclosure may be applied to software for other medical purposes. Will understand.

1 is a diagram showing a test automation framework system for medical software according to an embodiment of the present invention.

The test automation framework system 100 according to the present disclosure may include a configuration management machine 120, a developer fairy tale machine 130, and a build machine 140. In addition, for software development and testing, the developer computer 110 may be interconnected with the configuration management machine 120, the developer fairy tale machine 130, and the build machine 140. To this end, the test automation framework system 100 of the present disclosure may be implemented in an internal network environment such as an on-premise environment or may be implemented in an external network environment such as a cloud environment.

The developer computer 110 may include an integrated development environment 111. The integrated development environment 111 may be implemented as an integrated development environment software such as an integrated development environment (IDE) capable of processing all tasks required for a software development process in one software.

The configuration management machine 120 may store a result of performing the automated test according to the present disclosure. In addition, the configuration management machine 120 may store a backup file of each tool chain tool as well as a committed source code in the integrated development environment. To this end, the configuration management machine 120 may include a repository 121. The repository 121 may be implemented as, for example, Gitlab.

The developer fairy tale machine 130 may perform an automated test according to a test request from the integrated development environment 111 or may instruct the build machine 140 to perform the automated test. The developer assimilation machine 130 is a CI tool (continuous integration tool) module 131, an analysis/review module 132, a visualization module 133, an issue management module 134, a messenger module 135, and interoperability test automation. A module 136 and a code test automation module 137 may be included.

The build machine 140 may perform a build according to a build request from the integrated development environment 111. In addition, the build machine 140 may perform an automated test according to a test request from the developer fairy tale machine 130. The build machine 140 may include a build module 141, a UI test automation module 142, and a packaging/distribution module 143.

According to the present disclosure, the integrated development environment 111 may request a test/build from the test automation framework system 100 for testing coded medical software. In response to the request of the integrated development environment 111, the analysis/review module 132 analyzes and reviews the source code of the corresponding program, and if the review is successful, the program is stored in the storage 121 of the configuration management machine 120. ) Can be committed. The analysis performed by the analysis/review module 132 may include static analysis that detects code complexity, defects, etc. of the source code. The review performed by the analysis/review module 132 may be implemented through a code collaboration tool such as Gerrit. In addition, the CI tool module 131 may control the execution of the automated test in response to a request from the integrated development environment 111. The CI tool module 131 may be implemented with Jenkins, an open source automation server implemented in Java. Specifically, the CI tool module 131 may request a build from the build machine 140, and the build module 141 may perform a software build. The build module 141 may be implemented with a build engine such as MSBuild. Then, the UI test automation module 142 may perform a UI test on the built result. The UI test may be performed using a UI automation tool of an automatic functional testing platform such as Testcomplete. Then, the packaging/distribution module 143 may generate a software package for distribution. The packaging/distribution module 143 may be implemented as an installer/software packaging creation tool such as Installshield.

Here, the visualization module 133 may visually provide software build and automation test results. The visualization module 133 may be implemented as a program such as a Grafana dashboard to provide visualized data. When a failure or a specific event occurs during the above-described build and automation test process, the messenger module 134 may notify the developer of this fact. In addition, issues occurring during the build and automation test process may be managed by the issue management module 134. The issue management module 134 may be implemented, for example, with jira, which is an issue tracking software.

The test automation framework system 100 according to the present disclosure can provide a smoke test, a UI automation test, an interoperability automation test, and a source code automation test using a golden model for test automation of medical software, as will be described later. have.

The above-described automated test methods may be performed in association with the build type. In this regard, the build type may include a commit build, a daily build, and a deploy build. The commit build can be performed when the developer attempts to commit, and the automated smoke test, interoperability automation test, and source code automation test can be performed according to the commit build request. The daily build may be performed according to a predetermined schedule (eg, once a day) for tasks that take a long time to execute. The UI automation test, which will be described later, may be performed according to a daily build request because it is performed on all functional elements of medical software. The deployment build is a build for deployment only, and may be performed according to a build request from the integrated development environment 111 when deployment is required. On the other hand, according to another embodiment, at least one of the above-described automated test methods may be implemented to be performed together when building the deployment.

2 is a view showing an automated smoke test method for medical software according to an embodiment of the present invention.

In order to solve the CAD test complexity of medical software such as dental CAD software, the present disclosure proposes an automated smoke test method based on golden model modeling as a primary fast automated test method for a representative reference model.

The automated smoke test method of the present disclosure includes determining a subset of test cases associated with a reference model of medical software (210), generating input image data based on the subset of test cases (220), A step 230 of performing an automated test through a UI automation tool using the input image data, and a step 240 of storing and managing the execution result of the automated test in an automated test storage. Such an automated smoke test method may be performed by the UI test automation module 142 of the build machine 140 at the time of commit build.

Since the automated smoke test of the present disclosure needs to be able to secure the sufficiency of the automated test through a short test, a reference model having representativeness for, for example, dental CAD software is determined based on the golden model modeling in step 210. I can. Specifically, in the case of dental CAD software, based on the basic elements of dental treatment such as tooth composition, treatment type, tooth position, and number of teeth, it can cover a large number of cases and easily verify errors, and the frequency of use You can model a golden model that can verify high behavior.

As an example, modeling elements for implementing a golden model can be classified into CAD input images and work routines, and classification criteria for each can be set as shown in Table 1 below.

-Golden model modeling object Classification criteria Input video Type of tooth composition (general, clinical, difficult) Type of treatment (laminate, crown, inlay, etc.) Tooth position (maxilla, mandible) Tooth position (anterior, posterior) Number of teeth (single, 6 incisors) Work routine General routine related to input video Routine you want to emphasize in the target product

In addition, 12 detailed cases (ie, a subset of test cases) may be determined according to a criterion that considers clinical effectiveness based on the factors presented in Table 1 as an example.

-Golden model detailed case design division Tooth composition Treatment type Position (up and down) Position (back and forth) Number of teeth One Normal Laminate Maxillary transposition 6 incisors 2 Normal Crown Maxillary Posterior single 3 Normal Inlay Maxillary Posterior single 4 Normal Inlay Mandible Posterior single 5 Normal Crown Mandible Posterior single 6 clinical bridge Maxillary transposition 3 copies 7 clinical bridge Upper/mandibular Posterior 4/3 copies 8 clinical bridge Maxillary transposition 7 copies 9 Difficult bridge Maxillary transposition single 10 Difficult Crown Maxillary Posterior single 11 Difficult Inlay Maxillary Posterior single 12 Difficult Laminate Maxillary transposition 6 incisors

In step 220, input image data may be generated based on a subset of test cases as illustrated in Table 2. Specifically, input image data may be generated by scanning a tooth model produced based on each test case with a scanner. The input image data may be 3D data as an example, and may also be 3D model data in STL format as an example. In step 230, an automated smoke test may be performed using input image data through a UI automation tool such as Testcomplete. In step 240, the execution result of the automated test may be stored and managed in the storage 121 of the configuration management machine 120, and the test execution result stored in the storage 121 may be stored to be inquired by the tester. 3 is an exemplary diagram visually showing steps 220 and 230.

4 is a diagram illustrating a UI automation test method for medical software according to an embodiment of the present invention.

The automated smoke test described in connection with FIG. 2 provides a quick automated test for important functions of the target software based on the golden model, but does not cover the entire interface functional elements of the software. Accordingly, the present disclosure proposes a UI automated test method for automated testing of all interface functional elements of medical software such as dental CAD software.

The UI automation test method of the present disclosure includes determining test cases for all interface functional elements of medical software (310), and performing an automated test on the determined test cases through a UI automation tool (320). And storing and managing the result of the execution of the automated test in the automated test storage (330). This UI automation test method may be performed in the UI test automation module 142 of the build machine 140 during daily build. In addition, step 310 is based on defect analysis or additional functional elements of medical software so that additional test cases by defect analysis or test cases for additional functional elements generated in the development process can be additionally tested in the UI automation test. Thus, an operation of updating the test case may be included.

Test cases tested in the UI automation test of the present disclosure may include an Unattended case, an Attended case, and a specification-based case. The unattended case is a test case that can be performed by the UI automation tool from test execution to result determination. For example, for an unattended case, P/F (Pass/Fail) is automatically determined by a UI automation tool, and a test result may be output. In the Attended case, the test is automatically performed by the UI automation tool, but the result judgment is a test case made by the tester. For the attended case, the UI automation tool may provide, for example, an expected image, a result image, and a test result including a similarity analysis result of the two images so that the tester can judge the result. Specification-based cases are test cases that cannot be automatically tested by UI automation tools. Specification-based cases require direct analysis and result judgment by testers.

FIG. 5 is an exemplary diagram illustrating an order in which test cases are tested in the UI automation test of FIG. 4. As shown in FIG. 5, the UI automation test of the present disclosure may be performed first for an unattended case, which is an automatic case, and then, may be performed for an Attended case, which is a semi-automatic case, and then a specification-based case. Can be done against However, the UI automation test of the present disclosure is not limited to the above test order, and may be performed in a different order from the above-described order according to embodiments.

In the present disclosure, test cases may be generated for a software unit function based on test feedback data such as an error case, a specification-based function list, and a defect report. In addition, each test case may be generated in the form of a UI event-based script so as to be independent of UI location change. In addition, in order to prevent generation of unnecessary test cases, the present disclosure may reduce the number of redundant test cases by using a redundancy removal method such as a pairwise technique. In addition, after the UI automation test of the present disclosure is performed on all test cases, test coverage is measured through a verification tool such as Bullseye, so that the efficiency of the UI automation test may be verified.

6 is a diagram illustrating a method for automating interoperability testing for medical software according to an embodiment of the present invention.

A smart dental system capable of using exemplary dental CAD software may consist of a scanner, CAD and Computer Aided Manufacture (CAM). The scanner delivers the work result (file) to CAD, and the CAD can process the received file and deliver the result to CAM. However, since these dental CAD software, scanners, and CAMs can be manufactured by different manufacturers, interoperability between such medical software and heterogeneous equipment must be verified. Accordingly, the present disclosure proposes an interoperability automation test method for testing interoperability of medical software such as dental CAD software.

The interoperability automation test method of the present disclosure includes determining test cases based on a data standard associated with medical software (410), performing an automated test on test data of the test cases through a test oracle (420). ) And storing and managing the execution result of the automated test in the automated test storage (430). Step 420 may include an operation of determining interoperability of the corresponding test case by comparing the output for each test case with the expected output based on the data standard by testing the input and output of test data through a test oracle. have. This interoperability automation test method may be performed in the interoperability test automation module 136 of the developer fairy tale machine 130 at the time of commit build.

The data standard described above may be an ISO 18618 standard established in July 2018 to facilitate the transfer of dental case data and CAD/CAM data between software systems. In addition, test data of test cases may be configured in the form of a mark-up language according to a data standard established for dental CAD software. For example, the markup language may be an Extensible Mark-up Language (XML).

FIG. 7 is an exemplary diagram illustrating a test oracle for an automated interoperability test according to FIG. 6.

As shown in FIG. 7, test data in the form of an XML file may be input from the storage 121 to the interoperability test oracle module 450 of the interoperability test automation module 136. The test oracle module 450 may attempt to import an XML file into the CAD software 460. CAD software 460 can import files with all nodes and data specified in the data standard. In addition, when an XML file that does not conform to the data standard is input, the CAD software 460 loads only data conforming to the data standard excluding a corresponding node, and outputs and transmits the XML file to the test oracle module 450.

The scenario of the interoperability automation test of the present disclosure may include a normal input, an abnormal input 1, an abnormal input 2, and a normal output according to input/output types. An exemplary test scenario is shown in Table 3.

-Interoperability automation test scenario ID Test scenario TS01 Check whether normal XML loading is possible TS02 Checking whether error handling is possible when loading abnormal XML (including nodes not specified in the standard) TS03 Checking whether error handling is possible when loading abnormal XML (attribute or data format specified in the standard is different) TS04 Check whether normal XML loading and EXPORT is possible

As an example, test cases specifying preconditions, inputs, and expected outputs based on the scenarios presented in Table 3 may be defined as shown in Table 4.

-Interoperability automation test case ID Test case TC01 -Precondition: xml file including all elements and attributes specified in the standard, XSD file conforming to the standard
-Input: xml load
-Expected Output: Load all attribute data included in xml file TC02 -Precondition: xml file including all elements and attributes specified in the standard, including nodes not specified in the standard, XSD file conforming to the standard
-Input: xml load
-Expected Output: xml load failed TC03 -Precondition: xml file including all elements and attributes specified in the standard or specific attribute data format is different, XSD file conforming to the standard
-Input: xml load
-Expected Output: xml load failed TC04 -Precondition: xml file including all elements and attributes specified in the standard, XSD file conforming to the standard
-Input: xml load
-Expected Output: All newly created files are the same except for the input xml file and the changed part.

The interoperability automation test of the present disclosure may be used for an API (Application Program Interface) unit test for XML input/output during a commit build.

Meanwhile, according to the present disclosure, when a source code level test is required, the code test automation module 137 of the developer assimilation machine 130 performs a source code automation test in response to a commit build request from the integrated development environment 111. Can be done.

It is to be understood that any particular order or hierarchy of steps in any of the presented processes is an example of exemplary approaches. Based on the design priorities, it is to be understood that within the scope of the present invention a specific order or hierarchy of steps in processes may be rearranged. The appended method claims provide elements of the various steps in an exemplary order, but are not meant to be limited to the specific order or hierarchy presented.

As used herein, the terms "component", "machine", "module", "system" and the like may refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or execution of software. For example, a module may be, but is not limited to, a processing process executed on a processor, a processor, an object, an execution thread, a program, and/or a computer. For example, both an application running on a computing device and a computing device may be modules. One or more modules may reside within a processor and/or thread of execution, and a module may be localized within one computer, or may be distributed between two or more computers. In addition, these modules can execute from various computer readable media having various data structures stored therein.

Computer-readable media, whether direct and/or indirect, whether in raw state, formatted state, organized state, or any other accessible state, include relational repositories, non-relational repositories, in-memory Repositories, or other suitable repositories that may store data and allow access to such data through a storage controller, may include repositories, including decentralized ones. In addition, the storage medium includes a primary storage device, a secondary storage device, a tertiary storage device, an offline storage device, a volatile storage device, a nonvolatile storage device, a semiconductor storage device, a magnetic storage device, an optical storage device, and a flash device. Storage devices, hard disk drive storage devices, floppy disk drives, magnetic tapes, or other suitable data storage media.

In this specification, instructions are assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or object-oriented such as Smalltalk, C++, etc. It may be either source code or object code written in any combination of programming languages and one or more programming languages including conventional procedural programming languages such as "C" programming languages or similar programming languages.

The description of the presented embodiments is provided to enable any person skilled in the art to use or implement the present invention. Various modifications to these embodiments will be apparent to those of ordinary skill in the art, and the general principles defined herein can be applied to other embodiments without departing from the scope of the present invention. Thus, the present invention is not to be limited to the embodiments presented herein, but is to be construed in the widest scope consistent with the principles and novel features presented herein.

110: developer computer
111: Integrated development environment
120: configuration management machine
121: storage
130: developer fairy tale machine
131: CI tool module
132: Analysis/Review module
133: visualization module
134: issue management module
135: messenger module
136: Interoperability test automation module
137: code test automation module
140: build machine
141: build module
142: UI test automation module
143: packaging/distribution module

Claims (20)

In the test automation system for medical software,
A developer fairy tale machine for performing an automated test according to a test request from the integrated development environment or instructing a build machine to perform the automated test;
A build machine that performs a build according to a build request from the integrated development environment, and performs the automated test according to a test request from the developer fairy tale machine; And
Including a configuration management machine for storing the execution result of the automated test,
Test automation system. The method of claim 1,
The developer assimilation machine operates as an automation server and includes a CI tool (continuous integration tool) module that controls the execution of the automation test,
Test automation system. The method of claim 1,
The build machine includes a UI (user interface) test automation module that performs a smoke test as the automation test according to a commit build request from the integrated development environment,
The UI test automation module is configured to perform the smoke test on a subset of test cases associated with the reference model of the medical software,
Test automation system. The method of claim 3,
The UI test automation module performs an automation test through a UI automation tool using input image data configured based on a subset of the test cases,
Test automation system. The method of claim 1,
The build machine includes a UI test automation module that performs a UI automation test as the automation test according to a daily build request from the integrated development environment,
The UI test automation module is configured to perform the UI automation test on test cases for all interface functional elements of the medical software,
Test automation system. The method of claim 5,
The UI test automation module classifies the test cases into an unattended case, an attended case, and a specification-based case, and performs an automation test through a UI automation tool,
Test automation system. The method of claim 1,
The developer assimilation machine includes an interoperability test automation module that performs an interoperability automation test as the automation test according to a commit build request from the integrated development environment,
Test automation system. The method of claim 7,
The interoperability test automation module verifies input/output compatibility of the medical software by receiving test data in the form of a markup language and performing an automated test through a test oracle,
Test automation system. The method of claim 1,
The developer assimilation machine includes a code test automation module that performs a source code automation test as the automation test according to a commit build request from the integrated development environment,
Test automation system. As an automated smoke test method for medical software,
Determining a subset of test cases associated with the reference model of the medical software;
Generating input image data based on the subset of test cases;
Performing an automation test through a UI automation tool using the input image data; And
Including the step of storing and managing the execution result of the automated test in an automated test storage,
Automated smoke test method. The method of claim 10,
The automation test is performed in response to a commit build request from the integrated development environment,
Automated smoke test method. The method of claim 10,
The reference model of the medical software is determined based on golden model modeling,
Automated smoke test method. As a UI automation test method for medical software,
Determining test cases for all interface functional elements of the medical software;
Performing an automated test on the test cases through a UI automation tool; And
Including the step of storing and managing the execution result of the automated test in an automated test storage,
How to test UI automation. The method of claim 13,
The automation test is performed in response to a daily build request from an integrated development environment,
How to test UI automation. The method of claim 13,
The step of determining the test cases includes determining test cases including an Unattended case, an Attended case, and a specification-based case,
How to test UI automation. The method of claim 13,
The step of determining the test cases includes updating test cases of the automated test based on a defect analysis or an additional functional element of the medical software,
How to test UI automation. As an automated test method for interoperability for medical software,
Determining test cases based on a data standard associated with the medical software;
Performing an automated test on the test data of the test cases through a test oracle; And
Including the step of storing and managing the execution result of the automated test in an automated test storage,
Automated interoperability test method. The method of claim 17,
The test data of the test cases is configured in the form of an extensible mark-up language (XML),
Automated interoperability test method. The method of claim 17,
In the performing of the automated test, by testing the input and output of the test data through the test oracle, the output of each test case and the expected output based on the data standard are compared to determine interoperability of the corresponding test case. Comprising steps,
Automated interoperability test method. A computer program stored on a computer-readable medium comprising computer-executable instructions for executing a method according to any one of claims 10 to 19.

Images