JP2021507325A - Systems and methods to improve the reliability of medical imaging devices - Google Patents

Abstract

Systems and methods for generating device test cases for medical imaging devices that not only reflect the current actual field usage of the medical imaging device but also provide an outlook for future usage are disclosed. A probabilistic model of usage patterns is generated from historical data present in the device field log by mining the current usage patterns. A deep-length, short-term memory neural network model of usage patterns is constructed to predict future usage patterns. In addition, predictive models are continually updated in real time to capture changing trends in device usage patterns in the field, and the test cases generated by the models are integrated into an automated test system.

Description

[0001] The following generally relates to improving the reliability of medical imaging devices.

[0002] Since malfunctions in medical device hardware and software adversely affect the patient's diagnostic process, comprehensive testing of medical devices to ensure consistent and reliable functionality is crucial. In addition, equipment failures can reduce the productivity of hospitals that use them. In traditional machine testing methodologies, field experts prepare operational profiles similar to the clinical workflows performed by engineers in hospitals. Operational profiles often include typical workflows performed to image various anatomical structures in terms of scanning workflows and corresponding software operations. The device verification team then uses this profile to verify the functionality of the machine.

[0003] Preparing a device operational profile that contains a comprehensive list of test scenarios presents several challenges. The imaging device facilitates the generation of an image of the desired contrast by allowing the technician to change various contrast parameters to achieve the desired image contrast. The number of such different contrast configurations used in the field is very large. It is necessary to include some such configurations in the operational profile, depending on the actual field usage (hereinafter referred to as field usage).

[0004] In addition, the imaging device allows the technician to perform many management and post-image processing operations via the device UI. As a result, technicians often multitask and work in parallel with multiple patients with the help of such UIs. For example, the technician prints the image and transfers the image in response to activities such as post-processing the image of the previous patient or scan while the current scan is in progress. In order to ensure that the medical device operates reliably in such situations, it is necessary to include the usage status of all provided UI operations in the operation profile according to the actual field usage status.

[0005] In addition, some of the advanced imaging features packaged with medical devices are computational and memory intensive, which can lead to device software system hangs and even crashes. Therefore, it is very important to include the usage of those features in the operational profile.

[0006] Fluctuations in the actual usage of imaging devices in the field are another aspect that makes it very difficult to envision comprehensive usage scenarios. The clinical scan sequence varies greatly depending on the patient's condition. For example, imaging techniques for studying cellularity can be very different from techniques for analyzing cerebral hemorrhage. In addition, clinical scan sequences vary depending on the clinical practice of the hospital. In addition, hospitals are continually producing effective and improved scan sequences. Therefore, it is necessary to estimate currently known scan configurations in order to anticipate future progress and include them in operational profiles. In addition, operational profiles need to be continually updated to capture ongoing trends in clinical practice.

[0007] The cost and time involved in performing the actual tests of the imaging device imposes a limit on the number of tests that can be included in the operational profile. For example, the cost of operating and maintaining an MRI machine is very high, estimated at about $ 1000 per day. In addition, the average time required to complete one clinical protocol is estimated to be about 30 minutes. This suggests that the protocols contained in the operational profile should briefly describe the field usage of the imaging device.

[0008] The imaging device records a description of all performed scans and UI operations in a log file with a time stamp. In addition, the log contains ancillary information indicating the start and end of the clinical workflow. Event timestamps stored in log files can be used to align log information in chronological order of machine operations, and start and end indicators can be used to extract clinical workflows. Therefore, log files are a good way to get insights into actual device usage in the field. Log files are also available daily from machine equipment around the world, making them suitable for analyzing and capturing ongoing clinical trends. Moreover, since the log file is a description of the actual usage of the machine, the test cases derived from the log file require little validation.

[0009] Traditional test methodologies are driven by operational profiles prepared by field experts. However, their profiles are limited to expert knowledge and often do not adequately represent the entire clinical workflow. Moreover, such profiles are rarely updated and therefore may not adequately reflect the ongoing progress of clinical practice.

[0010] In most busy hospitals, technicians often multitask and work in parallel for multiple patients or tests. For example, the technician prints the image and transfers the image in response to activities such as post-processing the image of the previous patient or scan while the current scan is in progress. The number of different combinations of such activities is very large. For example, by changing various machine parameters, a technician can configure more than 200 unique scans. The examination is then constructed by selecting a subset of such scans in the sequence most suitable for the current patient scenario. A typical test includes 10 such scans, which suggests that the total number of permutations that can be scanned exceeds 1 million. Therefore, it is very difficult for an expert to identify the most likely combinations of such activities and include them in the operational profile.

[0011] Another drawback of traditional test methodologies is that they often do not have modules for predicting future usage patterns and incorporating them into the test system. These methodologies are often limited to the use of explicitly specified test scenarios, which increase the risk of device failure when faced with unfamiliar usage patterns.

[0012] The field log file contains a wealth of information about the actual usage of the medical device. However, there are many challenges in deriving test usage patterns using field log files. First, accurately identifying the most prominent workflow is not trivial due to fluctuations in device usage. For example, a technician can perform the same scan set in many different orders. In addition, some machine parameters can be modified to suit the patient's condition, so usage patterns are often sparsely distributed. Therefore, the derived test patterns do not completely represent the actual device usage. Moreover, it is difficult to guarantee that the test patterns derived by estimating the usage patterns found in the field logs match the actual clinical usage of the machine.

[0013] Seamlessly integrate field clinical workflows with workflows derived by estimating actual workflows from field log files in light of the above challenges in operational profile preparation and log file usefulness. , Innovative testing methodologies are needed.

[0014] Some improvements are disclosed below.

[0015] In one aspect disclosed, a system that facilitates the generation of test scripts for medical imaging devices provides field log files containing data related to examinations performed by one or more imaging devices. It has a stored field log file database. The system also comprises one or more processors, the one or more processors processing the field log file stored in the field log file database with a mining module that identifies prominent field usage patterns. , Executes a prediction module that predicts future workflow patterns by processing the field log file stored in the field log file database. The system further includes a test case preparation and execution module that generates and executes at least one test script for the imaging device, at least in part based on the identified prominent field usage patterns and the future workflow patterns. Be prepared.

[0016] In another disclosed embodiment, a method of generating a test script for a medical imaging device comprises receiving a field log file containing data related to a test performed by one or more imaging devices. It has a step of processing the field log file to identify prominent field usage patterns and a step of constructing and training a long short-term memory (LSTM) neural network to identify future workflow patterns. The method further comprises generating and executing at least one test script for testing the imaging device, at least partially based on the identified prominent field usage patterns and the future workflow patterns.

[0017] In another disclosed embodiment, a method of predicting future workflow patterns for a medical imaging device identifies all examinations E performed by a plurality of medical imaging devices in terms of the sequence of scans. And by building long short-term memory (LSTM) neural network models to predict future workflows and collecting unique scan names across all tests, scan names and user interface (UI) operations It has a step of generating a dictionary D. The method further comprises the step of generating a one-hot encoding of test E by identifying the index of each dictionary word of test E in dictionary D.

[0018] In another disclosed embodiment, a system that facilitates the generation of test cases for execution in an imaging device using information from data log files from multiple imaging devices provides log files. A plurality of imaging devices (120, 122, 124) to be generated, a log file database (20) for storing the log file, and one or more processors (14) are provided, and the one or more processors (the one or more processors). 14) identifies a workflow that includes a sequence of scan operations performed by the imaging device. The one or more processors further identifies unintended invalid usage patterns and unintended valid usage patterns in the identified workflow and generates at least one test case sequence for new scan products and scan product updates. And schedule the test case sequence to run on one or more of the plurality of imaging devices to test at least one of the new scan product and the scan product update.

[0019] One advantage is improved test cases across a group of medical imaging devices.

[0020] Another advantage is the dynamic update of workflow forecasts.

[0021] A given embodiment may provide one, two, more, or all of the above advantages and / or others as will be apparent to those skilled in the art upon reading and understanding the present disclosure. May provide the benefits of.

[0022] The present invention may take the form of various components and arrangements of components and various steps and arrangements of steps. The drawings are intended merely to illustrate preferred embodiments and should not be construed as limiting the invention.

[0023] FIG. 6 illustrates a system that facilitates the generation of test cases for testing medical imaging devices, according to one or more aspects described herein. [0024] It is a figure which shows the method for generating the neural network described in this specification by one or more aspects described in this specification. [0025] It is a diagram showing a method for training a neural network by one or more features described herein. [0026] It is a block diagram showing a detailed aspect of innovation by one or more features described herein. [0027] It is a diagram schematically showing various aspects of an analyzer module according to one or more aspects described in the present specification. [0028] It is a figure which shows the method of identifying a workflow by one or more aspects described herein. [0029] FIG. 6 illustrates various examples of criteria by which workflows can be stratified, according to one or more aspects described herein. [0030] FIG. 6 illustrates a method for analyzing a tier workflow, eg, processor 14 (FIG. 1), according to one or more aspects described herein. [0031] FIG. 6 illustrates a method for analyzing a workflow according to one or more aspects described herein. [0032] It is a figure which shows the method for generating the workflow summary. [0033] FIG. 6 illustrates a method for estimating test cases according to one or more features described herein. [0034] FIG. 6 illustrates a method for predicting future usage test cases according to one or more features described herein. [0035] It is a figure which shows the method of generating the usage report by one or more features described in this specification. [0036] FIG. 6 illustrates a method for analyzing performance as may be performed by a layer analysis module, with one or more features described herein. [0037] In a diagram illustrating a method for scheduling test cases to be performed by one or more imaging devices, as can be done by a dynamic model builder scheduler, according to the various aspects described herein. is there.

[0038] This described system and method limits the limitations of traditional test methodologies by extensively analyzing field log files and enhancing the workflow provided in the operational profile with the actual workflow performed in the hospital. Regarding test methodologies aimed at overcoming. The technique begins by processing log files to identify different workflows and derives the most important usage patterns by integrating the workflows throughout the hospital. The technique also improves operational profiles by using machine learning techniques to predict possible future fluctuations in ongoing workflows and adding those predicted profiles to the test system. In addition, the proposed method predicts the usage of newly introduced product characteristics and generates a new operational profile consisting of those newly introduced product characteristics. In addition, this method continuously updates the constructed machine learning model in real time with newly available field logs. In addition, this approach provides a fully automated test system, providing continuous integration of new test cases derived from field logs with predicted workflows.

[0039] Along with generating operational profiles, the innovations described generate usage reports for all available scan types, UI operations, and other modules in the system. This report includes (but not limited to) widely used scan types, widely used UI operations, rarely used scan types, and rarely used UI operations. In addition, the innovations described generate extensive performance reports for medical devices. This report includes (but not limited to) CPU usage, memory usage, time analysis for each scan type, usage time analysis for each UI operation, performance of various machine learning modules, image quality analysis, etc. including.

[0040] The proposed approach solves the challenge of deriving prominent usage patterns using field log files. Along with identifying the most frequent usage patterns, this methodology builds a probabilistic model of usage patterns and explores multiple patterns of fluctuating probabilities. This ensures that the derived pattern contains a sparse space of usage patterns. Since the predicted pattern is composed of the actual usage pattern, the obtained pattern is expected to match the actual clinical usage of the machine. In addition, this methodology builds a multi-layer neural network (eg, long short-term memory (LSTM)) model of usage patterns, thereby providing multiple advantages. The multi-layer neural network model can learn appropriate weighting for the sample in the input sequence regardless of the input position. Therefore, the usage patterns predicted by the multi-layer neural network efficiently summarize the actual device usage even if the device usage is out of order.

[0041] In addition, multi-layer neural network models have the unique ability to learn semantic relationships of data. This not only allows the proposed methodology to predict fluctuations in usage patterns, but also ensures that the prediction patterns match the actual clinical usage of the machine. In addition, the proposed approach produces an operational profile consisting of possible unintended usage of the medical device. This is done in two ways. First, by analyzing the field log file, we can identify unintended usage in the field. Second, they are synthesized using the multi-layer neural network and statistical model described above. Ultimately, the proposed systems and methods have the ability to generate operational profiles on any subset of the data identified using stratification procedures.

[0042] FIG. 1 shows a system 10 that facilitates the generation of test cases for testing medical imaging devices, according to one or more aspects described herein. The system 10 includes a user interface (UI) 13, a processor 14, and a test module or device 12 coupled to a memory 16 (ie, a computer-readable medium) and a usage pattern for a medical device being tested by the system 10. Is configured to identify. A "usage pattern", also referred to herein as a "workflow pattern," is defined as a sequence of scans in an inspection and parallel user interface (UI) operations performed during the inspection.

[0043] The processor 14 executes one or more computer-executable modules for performing various functions, methods, etc. described herein, and the memory 16 stores those modules. There is. As used herein, "module" refers to a computer executable algorithm, routine, application, or program, and / or a processor that executes such a computer executable algorithm, routine, application, or program.

[0044] The memory 16 is a computer-readable medium such as a disk or a hard drive in which a control program is stored. Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic storage medium, CD-ROM, DVD, or any other optical medium, RAM, ROM, etc. A PROM, EPROM, FLASH®-EPROM, variants thereof, other memory chips or cartridges, or any other tangible medium that can be read and executed by the processor 14. In this context, the systems described are one or more general purpose computers, dedicated computers, programmed microprocessors or microprocessors and peripheral integrated circuit elements, ASICs or other integrated circuits, digital signal processors, discrete element circuits. It may be implemented in or as hard-wired electronic circuits or logic circuits such as, or programmable logic devices such as PLDs, PLAs, FPGAs, graphics processing units (GPUs), or PALs.

[0045] The test module 12 receives the new field log file 18 and stores it in the field log file database 20. That is, the log file created from the imaging device installed in the hospital or the like is stored in the field log file database. The stored log file is provided from the database 20 to the prediction module 22 that predicts the future workflow and the mining module 24 that mines the workflow pattern from the log file. Further, the new log file 18 is provided to the mining pattern update module 26 that updates the mined workflow pattern based on the new input, and the prediction pattern update module that updates the pattern predicted by the prediction module 22. The test case preparation module 30 receives inputs from each of the prediction module 22, the mining module 24, the mining pattern update module 26, and the prediction pattern update module 28, and also generates a test case on a given imaging device 32. It is configured to perform one or more tests.

[0046] The prediction module 22 predicts the future workflow by processing the log file stored in the database 20, and predicts the future fluctuation in the currently remarkable workflow pattern. The log file contains details of various operations performed on the imaging device. The prediction module first identifies all tests in terms of the sequence of scans. The prediction module then builds a multi-layer neural network model and uses the following procedure to predict future workflows.

[0047] Data preparation: Let E be the set of all tests. Here, the i-th inspection Ei is defined as follows.
Ei = Ei1, Ei2, ..., Eip, but i = 1,2, ..., n,
Here, Eij is the name of the jth scan in Ei, ip is the number of scans and UI operations in Ei, and n is the total number of tests in E. The start and end of the examination are represented as special scans, called start scan and end scan, respectively. All tests of E are modified to include start and end scans so that Ei1 is the start scan and Eip is the end scan.

[0048] The prediction module 22 generates a dictionary D of scan names and UI operations by collecting unique names across all checks. It is defined as:
D = D1, D2, ..., Dl,
Here, Di is the i-th word in the dictionary, and l is the number of elements in the dictionary.

In addition, the prediction module generates a "one-hot encoding" of E by identifying the index of each dictionary word of E in D. "One-hot encoding" is defined as follows.
OHEdiction word = [x0, x1, ..., xn-1],
Here, when it is the index of the dictionary word at i = D, xi = 1, otherwise it is 0.

[0050] The input tensor X and the output tensor y of the LSTM are prepared as follows.
X = X1, X2, ..., Xn, where the i-th element Xi is defined as follows.
Xi = OHE (Ei1), OHE (Ei2), ..., OHE (Eip-1), and
y = y1, y2, ..., Yn, where the i-th element yi is defined as follows.
yi = OHE (Ei2), OHE (Ei3), ..., OHE (Eip)

Further referring to FIG. 1, the prediction module 22 performs a procedure for predicting a workflow using an LSTM model constructed during training of network steps as follows.

[0052] Let U be the input tensor of the LSTM model, and define it as follows.
U = OHE (U1), OHE (U2), ..., OHE (Uk),
Here, for i = i to k, Ui is an input dictionary word and U1 = start scan.

[0053] Let pred be an LSTM prediction function, and its format is
predU = [V1, V2, ..., Vk],
Here, for i = i to k, each Vi is a vector of the class probability of each word in the dictionary D.
Let V1 = [p0, p1, ..., pl-1].
Here, p0 is the probability of the first word in the dictionary, p1 is the probability of the second word in the dictionary, and so on. Let land (p0, p1, ..., pl-1) be a random index generator in the range (0, l-1) according to a given probability distribution p0, p1, ..., pl-1.

[0054] The predicted dictionary word can then be derived using the following equation.
scanpredU = D [landlast element of predU]

[0055] The initial input tensor I for prediction is prepared as follows.
I = [OHE start scan]

[0056] Further, the predicted test P is initialized as follows.
P = [start scan]

[0057] The predictive scan name for I is
c = scanpred (I)
Given by.

[0058] Next, I and P are updated using the predicted scan name as follows.
I = [I, OHE (c))] and P = [P, c]

[0059] The updated I is further used to predict the next dictionary word. The prediction is repeated until c = end scan. When the end scan is encountered, the updated P is recognized as a predicted future workflow. Since the dictionary contains both scan names and UI operations, the predicted future workflow will include both scan names and UI operations. Repeat the above steps to get multiple future workflows.

[0060] The prediction pattern update module 28 updates the multi-layer neural network model of the usage pattern generated by the prediction module with the newly available field log file. First, the prediction module identifies all checks in terms of the sequence of scans in the new log file. The prediction pattern update module 28 generates a new data set by combining the inspection identified in the database log file stored by the prediction module with the new inspection when predicting a future workflow, and the procedure is similar to that of the prediction module 22. Generates an updated set of predicted usage patterns according to.

[0061] The mining module 24 processes the log files stored in the database 20 to determine the most prominent field usage patterns. The log file contains details of various operations performed on the medical device. The mining module 24 first identifies all inspections in terms of the sequence of scans and parallel UI operations performed during the inspection. The mining module 24 then represents the identified inspection in a tree data structure. Each scan of the inspection is represented as a node in the tree, and the transition from one scan to another is represented as an edge in the tree. Each node stores a count of encounters with the node while populating the tree. The start and end of the check are represented as special nodes in the tree, called start and end nodes, respectively. Further, the mining module 24 stores UI operations performed during the inspection of the end node. After populating the tree with all the checks in the log file database, the count stored in the end node will represent the count of the particular check that exists in the log file database. The mining module 24 then uses the following procedures, called Procedures 1 and 2, to generate two sets of outstanding usage.

[0062] Step 1: This procedure identifies the most commonly used tests and proposes them as prominent usage patterns using the following procedure: Store Ci in the i-th end node. The count is counted. The k-th most prominent pattern is then defined as follows.
Ek = [Ek0, Ek1, ..., Ekn], but k = 1,2, ...,
Here, Ek0 is the start node, and Ekn is the kth most occurring end node.

[0063] Step 2: In this step, a probabilistic model of usage patterns is generated and the following steps are used to identify the most prominent field usage patterns: Cij, the jth of the i-th node of the tree. It is the count stored in the child node of. Next, the probability Pij of transitioning from the i-th node to the j-th child node of the i-th node is defined as follows.
Pij = CijSi
Here, Si = Cij, and n is the number of child nodes of the i-th node.

[0064] Furthermore, the k-th most prominent pattern is defined as follows.
Ek = [Ek0, Ek1, ..., Ekn], but k = 1,2, ...,
Here, Ek0 is the start node, Ekn is the end node, Ek1 is the kth most probable child node of Ek0, and so on.

[0065] The mining pattern update module 26 uses the newly available field log file 18 to update the probabilistic model of the usage pattern generated by the mining module 24 when mining a workflow pattern. First, the mining pattern update module 26 identifies all checks in terms of the sequence of scans and parallel UI operations performed during the check in the new log file 18. The mining pattern update module 26 then generates a new dataset by combining the tests identified by the mining module 24 with the new tests when mining the workflow pattern, generating an updated prominent field usage pattern. To do so, follow the same procedure performed by the mining module.

[0066] The test case preparation module 30 generates test scripts using usage patterns identified by other modules, tests them on imaging devices, and generates test reports.

ここで、Ｚは関数に対するＫ次元の入力であり、ｅｘｐは指数関数である。 [0067] With reference to FIG. 1, FIG. 2 is a neural described herein such that it can be performed by the prediction module 22 of FIG. 1 in one or more aspects described herein. Shows how to create a network. A multi-layer long short-term memory (LSTM) network is constructed using 2 (or 3) layer LSTM as shown in FIG. However, it will be understood that more or fewer layers may be utilized when building neural networks. The input tensor X is received as an input layer at 50. The input layer is tightly coupled to the first LSTM layer at 52. The LSTM layer is composed of one or more LSTM cells. Each LSTM cell contains three gates, namely an input gate, an oblivion gate, and an output gate, the formulas of which are given below.
i (t) = g (WxiX (t) + WhyH (t-1) + Bi], where t = 0,1,2, ...,
f (t) = g (WxfX (t) + WhfH (t-1) + Bf], where t = 0,1,2, ...,
o (t) = g (WxoX (t) + WhoH (t-1) + Bo], but t = 0,1,2, ...,
Here, i (t) is the input gate, f (t) is the forgetting gate, and o (t) is the activation of the output gate at time t, Wx is the weight, B is the bias, and X. (T) is the input at time t, and H (t-1) is the cell output at the previous time step. The input is converted by tanh as follows.
c_in (t) = tanh (WxcX (t) + WhcH (t-1) + Bc_in], where t = 0,1,2, ...,
Here, c_in (t) is the converted input, Wx is the weight, B is the bias, X (t) is the input at time t, and H (t-1) is the previous time. It is a cell output in a step. In addition, the hidden state is updated as follows:
c (t) = f (t) * c (t-1) + i (t) * c_in (t), where t = 0,1,2, ...,
h (t) = o (t) * tanh (c (t)), but t = 0,1,2, ...,
Here, * is a product operation for each element, c (t-1) is the cell activation in the previous time step, and h (t) is the cell output at the time t. At 54, a dropout function is applied to the first LSTM layer to improve network regularity. The dropout function ignores some network nodes and their corresponding activations during training. The dropout function randomly selects a new set of nodes, replacing each training iteration. At 56, the second LSTM layer is tightly coupled to the first LSTM layer, and at 58, the dropout function is applied to the second LSTM layer. Therefore, each LSTM layer is further tightly coupled to the next level LSTM layer, and the dropout function is applied to further improve network regularity. The final LSTM layer is tightly coupled to the "softmax" output layer at 60. The softmax function is given as follows.

Here, Z is a K-dimensional input to the function, and exp is an exponential function.

[0068] FIG. 3 shows a method for training a neural network with one or more of the features described herein. When training the network, at 80, the input tensor X is split into batches of tests of equal scan length. Then, during training, at 82, the network is rolled over to match the batch scan length. Here, rollover refers to the process of replicating LSTM cells to match the input length so that all such replicated cells use the same copy of network weights and biases. Become. The category cross entropy is used as the loss function, and at 84, the output tensor y is utilized to calculate the loss. The category cross entropy is defined as follows.
H (p, y) = −Σ _x p (x) log (y (x)),
Here, p is the true distribution. However, it will be appreciated that some other loss function (eg mean squared error, Kullback-Leibler divergence, etc.) may be utilized when constructing the neural network. In addition, at 86, network weights are updated during each epoch using adaptive moment estimation (ADAM) optimization (ie, one forward path and one reverse path across all training examples). However, it will be appreciated that some other optimization function (eg, stochastic gradient descent, RMSrop, Adagrad, etc.) may be utilized when training the neural network.

[0069] FIG. 4 is a block diagram showing a detailed embodiment of the innovation, such as configured by the system 10 of FIG. 1 and executed by the processor 14, according to one or more features described herein. is there. The database 20 receives and stores log files and monitoring agent reports from each of the plurality of user devices 120, 122, 124 (ie, medical imaging devices). Each of the plurality of user devices 120, 122, 124 includes performance monitoring agents 121, 123, 125, respectively. The dynamic model builder scheduler module 126 receives all available log files and reports from database 120, as well as scheduler configuration information for scheduling test cases. The analyzer module 128, which can be executed by processor 14 (FIG. 1), has all available log files and reports from the scheduler module 126, in addition to analyzer configuration information and a rule base in which rules are defined for performing analysis. To receive. The analyzer sends performance updates to user devices 120, 122, 124, respectively. In one or more embodiments, the analyzer module performs the functions of the mining module 24 and / or the future workflow prediction module 22 of FIG. In this regard, the mining module and / or future workflow prediction module of FIG. 1 can be considered to include the analyzer of FIG. 4, and thus all functions, procedures, methods, etc. performed by the analyzer module 128, etc. Can be similarly executed by the mining module 24 and / or the future workflow prediction module 22.

[0070] Analyzer 128 also identifies additional information about the imaging device being analyzed, including but not limited to unintended invalid use cases, unintended valid use cases, performance reports, usage reports, and the like. Send to the product marketing server 130. Based on this information, the product and marketing server 130 deploys one or more improved product versions 132 that are sent to the user device as product updates. In addition, analyzer 128 sends a set of selected optimal test cases to the evaluation team server 134 for test case evaluation and test report generation, which is a product and marketing server to improve the product version. Provided to 130.

[0071] FIG. 5 graphically illustrates various aspects of the analyzer module 128, according to one or more aspects described herein. The analyzer module receives all available log files and monitoring agent reports, the workflow identification module 140 then identifies the workflow and provides the identified workflow to the stratification module 142. The layer workflow is analyzed by a plurality of layer analysis modules including a first layer analysis module 144 that analyzes the first layer workflow and an nth layer analysis module 146 that analyzes the nth layer workflow. Each layer analysis module 144, 146 generates test cases and is provided to test set optimizer module 148, which produces and outputs an optimal test set based on one or more predetermined criteria. Each layer analysis module 144, 146 provides them to the effective use case coupling module 150, which identifies unintended valid use cases and outputs a combination of unintended valid use cases. Unintended invalid use cases are also identified by the layer analysis module and sent to the invalid use case coupling module 152 which outputs a combination of unintended invalid use cases. The performance report is provided by the layer analyzer to the performance report combination module 154 that outputs the combined performance report. Further, the usage report is provided by the layer analyzer to the usage report combination module 156 which outputs the usage report in combination.

[0072] FIG. 6 is a workflow according to one or more aspects described herein, such as that performed by the workflow identification module 140 when executed by a processor (eg, processor 14 in FIG. 1). Shows how to identify. The workflow identification module receives all available log files and analyzer configuration information. For each log file, the workflow identification module 140 identifies sequential operations at 180, parallel operations at 182, start and end of each workflow at 184, performance metrics at 186, and attributes of each workflow at 188. At 190, the workflow identification module 140 generates and outputs one or more workflows from the identified information.

[0073] FIG. 7 shows various examples of criteria by which the workflow can be stratified by the stratification module 142 according to one or more aspects described herein. The stratification module receives the workflow generated by the workflow identification module 140 (FIG. 6) and the analyzer configuration information, and stratifies the workflow based on one or more criteria or parameters that can be specified in the analyzer configuration information. To do. In the first example, workflows are stratified based on anatomical structures such as the patient's anatomical region imaged using each workflow. In the second example, workflows are stratified based on the area in which the device using each workflow is installed (eg, physical location or area, hospital or department store, etc.). In the third example, workflows are stratified based on the type of hospital in which they are used (eg, surgical center, cancer center, etc.). In the fourth example, the workflow is stratified based on the type of patient (eg, pediatrics, geriatrics, etc.) or based on a given preset. In a fifth example, the workflow is stratified based on the presence of one or more errors, crashes, scan aborts, etc. detected on the device using the workflow. In another example, all available workflows pass through without stratification, as indicated by the analyzer configuration information. In either case, the workflow of each layer (1, 2, ..., N) is output.

[0074] FIG. 8 is such that it is performed by the layer analysis module 144 (and 146) when executed by a processor such as processor 14 (FIG. 1), according to one or more aspects described herein. , Demonstrates a method for analyzing layered workflows. The stratified workflow output by the stratified module 142 is provided by the stratified analysis module along with a monitoring agent report, analyzer configuration information, and rule-based information that defines one or more rules to analyze the stratified workflow. Received. Workflow analysis is performed at 220 to identify and output test cases, unintended valid use cases, and unintended invalid use cases. Further, usage distribution information for all operations included in the layer workflow is output, and at 222, usage analysis is performed to generate and output a usage report. Performance analysis is also performed at 224 and generates and outputs a performance report describing the performance of the workflow and / or the imaging device that utilizes the workflow.

[0075] FIG. 9 illustrates a workflow, such as that performed by the workflow analysis module 220 when executed by a processor (such as processor 14 in FIG. 1), according to one or more aspects described herein. The method for analysis is shown. The workflow of a given workflow layer is received by the workflow analysis module 220. At 240, the workflow analysis module produces a workflow summary. The summary contains usage distribution information as output for all operations in the workflow contained in the layer. At 242, a decision is made as to whether the test case identified using the rule base is valid. Test cases implemented in the field (and generated by mining the log file) but not validated according to the rules base are considered valid unintended test cases. Test cases executed in the field and verified according to the rule base are output as valid test cases.

At 244, test case estimation is performed. This step may be similar or identical to the process performed by the mining module 24 (FIG. 1). Test case estimation can be performed using, for example, a conditional probability tree of sequential operations (ie, operations performed sequentially during a workflow) and a distribution of parallel operations for each sequential operation. At 246, a decision is made as to whether the estimated test case is valid according to the rule base. Invalid cases are output as unintended invalid use cases, while inferred test cases that are valid according to the rule base are output to a pool of valid test cases.

[0077] In addition, the identified test cases at 240 can be analyzed at 248 to generate future usage test cases. This step may be similar or identical to the processing performed by the future workflow prediction module 22 (FIG. 1). At 250, a decision is made as to whether the predicted future usage test case is valid according to the rule base. Invalid test cases are output as unintended invalid use cases, while predicted future usage test cases that are valid according to the rules base are output to a pool of valid test cases.

[0078] FIG. 10 shows a method for generating a workflow summary, as done by the workflow analysis module 220 described in connection with FIG. The workflow analysis module receives one or more workflows for a given layer, and analyzer configuration information, and at 260 identifies the frequency of each sequence of sequential operations performed in each workflow. In 262, the corresponding parallel operation is identified for each sequential operation, and distribution information is generated for the test case 264 output by the workflow analysis module.

[0079] In parallel, at 266, a conditional probability tree is generated to represent the probabilities of sequential operating paths in a layered workflow. Sequential operations are identified at node S and the probabilities are represented as edges in the tree. Further, at 268, usage distribution information 270 is generated for parallel operations of each sequential operation. In 272, usage status distribution information 274 for all operations (parallel and sequential) is generated and output.

[0080] FIG. 11 shows a method for estimating test cases, such as that which can be done by the workflow analysis module 220 in step 244 of FIG. 9, according to one or more of the features described herein. After receiving information about the conditional probability tree for sequential operations, the usage distribution for parallel operations for each sequential operation, and the analyzer configuration information, at 280, the workflow analysis module 220 traverses the conditional probability tree for sequential operations. To form a test case by generating a sequence of. At 282, parallel operation integration is performed using the parallel operation usage distribution for each sequential operation to generate an estimated test case 284.

[0081] FIG. 12 is for predicting future usage test cases, such as that which can be done by the workflow analysis module 220 in step 248 of FIG. 9, with one or more of the features described herein. The method is shown. After receiving information about the identified test cases in 240, the usage distribution of parallel operations for each sequential operation, and the analyzer configuration information, in 300, the workflow analysis module 220 predicts usage based on the semantic analysis of the workflow. Build and output the model. According to one example, when a new scan type is introduced for a given imaging modality, there may be few usage examples of the scan type from the field. Semantic analysis models make it easy to predict future usage of new scan types based on usage of similar scan types (eg, scan types with similar parameters) without the need to rewrite specific rules. ..

In 302, a usage model is used to predict a sequence of sequential operations, which includes a newly introduced sequential operation to generate sequential operation test cases. In parallel, at 304, a sequence of sequential operations is unconstrainedly predicted (ie, including all sequential operations as well as newly introduced sequential operations) to generate additional sequential operation test cases. In 306, parallel operation integration is performed on the sequential operation test cases generated in 302 and 304 in order to generate and output future usage test cases 308.

[0083] FIG. 13 provides a usage report such that it is output to the product and marketing server 130 of FIG. 4 by the layer analysis module 144 of the analyzer module 128 according to one or more features described herein. The method of generating is shown. At 320, widely used sequential operations are identified. In one embodiment, "widely used" refers to usage above a first predetermined threshold level (eg, 20% of workflow, 40% of workflow, or some other predetermined threshold level). At 322, lesser-used sequential operations are identified. In one embodiment, "less used" indicates a usage situation below the first predetermined threshold level. At 324, widely used parallel operations are identified. The widespread use of parallel operations can be defined as exceeding the first threshold used for the widely used sequential operations, or different (second) threshold levels. At 326, less used parallel operations are identified. The lesser use of parallel operations can be defined as the use below the threshold used to identify widely used parallel operations.

[0084] Further, at 328, future usage patterns for newly introduced operations are predicted by identifying similar types of operations in the usage distribution. For example, when a new scan type or user interface is introduced in a given imaging device, the layer analysis module 144 may have a similar operation type before the new scan type or user interface is released to the market as a new build version. Based on usage, use usage distribution information to predict future usage patterns for scan types or user interfaces. It is understood that any two or more of the steps performed during usage analysis can be performed in parallel (parallel) or sequentially (in any desired order) to generate and output a usage report. Yeah.

[0085] FIG. 14 is such that one or more of the features described herein can be performed in step 244 (FIG. 9) by the layer analysis module 144 (and 146) (see, eg, FIG. 5). Demonstrates a method for analyzing performance. Examples of parameters that can be analyzed to generate performance reports and / or updates are given. After receiving the tier workflow information, at 340, the amount of time required to perform each of the multiple scan types is analyzed and / or calculated. At 342, the memory requirements for each scan and / or UI operation are analyzed and / or calculated. At 344, the image quality of each imaging modality used is analyzed and / or calculated. At 346, the amount of time the user spends on each of the plurality of UI operations is analyzed and / or calculated. At 348, the performance of each of the plurality of machine learning models deployed in a given product is analyzed and / or calculated. Any or all of the above analyzes and / or calculations may be included in the performance report output by the layer analysis module described herein.

After receiving one or more reports from Performance Monitoring Agents 121, 123, 125 (FIG. 1), at 350, a processing (CPU) usage pattern for a given product is analyzed and / or calculated. To. At 352, a memory usage pattern for a given product is analyzed and / or calculated. At 354, a runtime pattern for a given product is analyzed and / or calculated. Any or all of the above analyzes and / or calculations may be included in the performance updates output by the layer analysis modules described herein.

[0087] FIG. 15 is for scheduling test cases to be performed by one or more imaging devices, as can be done by the dynamic model builder scheduler 126, according to various aspects described herein. The method is shown. After receiving all available log files and monitoring reports, at 370, give a given test case based on one or more criteria (time since last run, imaging device availability, operational status, etc.) A decision is made as to whether it can be scheduled for execution. If the decision at 370 is negative because it does not meet one or more criteria, this method loops repeatedly to keep trying to schedule the test case protocol. If the decision at 370 is affirmative, at 372 the test case is scheduled for execution.

[0088] The proposed method creates some test cases that represent actual device usage in the field. In the case of magnetic resonance imaging (MRI), test cases are converted into MRI test cards to perform device tests.

[0089] The present invention has been described with reference to preferred embodiments. Reading and understanding the detailed description above may reveal corrections and changes to others. The exemplary embodiments are intended to be construed as including all such modifications and modifications as long as they fall within the appended claims or their equivalents.

Claims (27)

A system that facilitates the generation of test scripts for medical imaging devices.
A field log file database containing field log files containing data related to inspections performed by one or more imaging devices, and
It comprises one or more processors, said one or more processors.
A mining module that processes the field log files stored in the field log file database and identifies prominent field usage patterns.
A prediction module that predicts future workflow patterns by processing the field log file stored in the field log file database.
Execute a test case preparation and execution module that generates and executes at least one test script for an imaging device, at least in part based on the identified prominent field usage patterns and the future workflow patterns. system. A mining pattern update module that analyzes the new field log file and updates the identified prominent field usage patterns,
The system according to claim 1, further comprising a prediction pattern update module that analyzes the new field log file and updates the predicted future workflow pattern. The mining module further
The system of claim 1 or 2, which identifies all inspections in terms of parallel user interface (UI) operations and scanning sequences performed during each inspection. The mining module further
The identified tests are represented by a tree data structure so that each scan of the test is represented as a node in the tree and the transition from one scan to another is represented as an edge in the tree. The system according to claim 3. The fourth aspect of claim 4, wherein each node stores a count of the number of times the node has occurred while populating the tree, and the start and end of each check is represented as a start node and an end node, respectively. system. The mining module further
Stores UI operations performed during the given check on the end node for a given check
After populating the tree with all the checks in the log file database, each end node stores a value representing the count of a particular check that exists in the log file database and corresponds to the end node. , The system according to claim 5. The mining module further
Identify the most commonly used set of tests and propose the most commonly used set of tests as a prominent usage pattern.
The system of claim 6, which generates a probabilistic model of usage patterns and identifies the most prominent field usage patterns. The prediction model further
Receive the input layer
The input layer is tightly coupled to the first long short-term memory (LSTM) layer.
Execute a dropout function on the first LSTM layer to improve network regularity.
The second LSTM layer is tightly coupled to the first LSTM layer.
The dropout function is executed on the second LSTM layer,
The system according to any one of claims 1 to 7, wherein an LSTM neural network is constructed by tightly coupling the second LSTM layer to the softmax output layer. The prediction model further builds a long short-term memory (LSTM) neural network, and the prediction model is
Divide the input tensor X into batches of tests of equal scan length and
Roll over the LSTM neural network to match the batch scan length and
Execute the category cross entropy function to calculate the loss using the output tensor y,
The system according to any one of claims 1 to 8, wherein the LSTM neural network is trained by performing an adaptive moment estimation (ADAM) optimization function to update network weights during each epoch. A method of generating test scripts for medical imaging devices, said method.
A step of receiving a field log file containing data related to a test performed by one or more imaging devices, and
Steps to process the field log file and identify prominent field usage patterns,
Steps to build and train long short-term memory (LSTM) neural networks to identify future workflow patterns,
A method comprising generating and executing at least one test script for testing an imaging device, at least partially based on the identified prominent field usage patterns and the future workflow patterns. The claim further comprises a step of analyzing the new field log file and updating the identified prominent field usage pattern and the predicted future workflow pattern based on the information contained in the new field log file. 10. The method according to 10. The method of claim 10 or 11, further comprising a step of identifying all tests in terms of parallel user interface (UI) operations and scan sequences performed during each test. Steps to represent the identified tests in a tree data structure so that each scan of the test is represented as a node in the tree and the transition from one scan to another is represented as an edge in the tree. The method according to claim 12, further comprising. A claim that each node further has a step of storing a count of the number of times the node has occurred while populating the tree, and a step of representing the start and end of each check as start and end nodes, respectively. 13. The method according to 13. A step to store UI operations performed during the given check on the end node for a given check, and
A step of populating the tree for all the checks in the log file database and then storing in each end node a value representing the count of a particular check that exists in the log file database and corresponds to the end node. The method according to claim 14, further comprising. With the steps of identifying the most commonly used test set and proposing the most commonly used test set as a prominent usage pattern.
15. The method of claim 15, further comprising generating a probabilistic model of usage patterns and identifying the most prominent field usage patterns. The step of constructing the long short-term memory (LSTM) neural network is
Steps to receive the input layer and
The step of tightly coupling the input layer to the first LSTM layer,
The step of executing the dropout function on the first LSTM layer to improve the network regularity, and
A step of tightly coupling the second LSTM layer to the first LSTM layer,
The step of executing the dropout function on the second LSTM layer, and
The method according to any one of claims 10 to 16, further comprising a step of tightly coupling the second LSTM layer to the softmax output layer. The step of training the LSTM neural network is
With the step of dividing the input tensor X into batches of inspections of equal scan length,
A step of rolling over the LSTM neural network to match the batch scan length,
Steps to execute the category cross entropy function that calculates the loss using the output tensor y,
The method of any one of claims 10-17, comprising performing an adaptive moment estimation (ADAM) optimization function to update the network weights during each epoch. A method of predicting future workflow patterns for medical imaging devices, said method.
In terms of the sequence of scans, the steps to identify all tests E performed by multiple medical imaging devices,
Steps to build a long short-term memory (LSTM) neural network model to predict future workflows,
A step to generate a dictionary D of scan names and user interface (UI) operations by collecting unique scan names across all checks.
A method comprising the step of generating a one-hot encoding of test E by identifying the index of each dictionary word of test E in dictionary D. A step of determining the probabilities for each of the plurality of words in the dictionary D,
19. The method of claim 19, further comprising predicting a future workflow as a function of the determined probabilities. A system that facilitates the generation of test cases for execution on imaging devices using information from data log files from multiple imaging devices.
With multiple imaging devices that generate log files,
A log file database that stores the log file and
It comprises one or more processors, said one or more processors.
Identify a workflow that includes a sequence of scan operations performed by the imaging device.
Identify unintended invalid usage patterns and unintended valid usage patterns in the identified workflow.
Generate a test case sequence for at least one of the new scan product and scan product update,
A system that schedules the test case sequence to run on one or more of the plurality of imaging devices to test at least one of the new scan product and the scan product update. The one or more processors further
Identify the frequency of each occurrence of multiple sequences of sequential operations
Generate a conditional probability tree that describes the sequential operation
Identify the parallel operation corresponding to the identified sequential operation
For each sequential operation, the usage distribution information of the identified parallel operation corresponding to the sequential operation is generated.
21. The system of claim 21, which generates at least one test case sequence based on the identified sequential operations, corresponding parallel operations, conditional probability trees, and usage distribution information. The one or more processors further
Generate usage and performance reports containing a summary containing information about the frequency and identity of the identified sequential operations, the corresponding parallel operations, the conditional probability tree, and the usage distribution information.
22. The system of claim 22, which sends the usage report and the performance report to a remote server. The one or more processors further
Generate the at least one test case sequence by predicting future workflows based on the identified sequential operations, the corresponding parallel operations, and the conditional probability tree.
22 or 23. The system of claim 22 or 23, wherein the at least one test case sequence is submitted for verification. 24. The system of claim 24, wherein the usage report and the performance report include information for triage of field service problems based at least in part on conditional probability distribution information. The system according to any one of claims 22 to 25, wherein the one or more processors repeatedly updates the at least one test case sequence based on the usage distribution information. 26. The system of claim 26, wherein the at least one test case sequence is a scan sequence for a newly introduced scan protocol.

Images