KR101404113B1 - A quality evaluation tool and its development method for detailed clinical models and the evaluation method thereof - Google Patents

Abstract

The present invention provides qualitative and quantitative quality requirements that a detailed clinical model must have for a developer by providing an objective evaluation of the quality of the detailed clinical model and provides a quantitative evaluation result of the detailed clinical model to the user, In addition, the developer of the detailed clinical model should be able to develop a detailed clinical model that meets the criteria, and provide the user with an objective basis for the quality evaluation of the detailed clinical model to improve the reliability of the detailed clinical model. An evaluation tool, an evaluation tool development method, and an evaluation method thereof.

Description

TECHNICAL FIELD [0001] The present invention relates to a method for evaluating the quality of a detailed clinical model, a method for developing an evaluation tool, and a method for evaluating the evaluation tool,

The present invention relates to a quality evaluation tool, an evaluation tool development method, and an evaluation method of a detailed clinical model, and more particularly, to a method and an apparatus for evaluating a detailed clinical model by defining evaluation factors to be provided by a detailed clinical model (DCM) A quality evaluation tool for evaluating the qualitative completeness of a detailed clinical model, a method for developing an evaluation tool, and an evaluation method thereof.

In general, the Detailed Clinical Model (DCM) is an electronic health record system (EHR) that is used to store and transmit medical and clinical content, It means a model that defines structure and contents.

Although the detailed clinical model (DCM) is a core model for the utilization of patient care and medical information, there has been no evaluation tool that can objectively evaluate the quality of the detailed clinical model. In other words, in the past, items for evaluating the detailed clinical model were not designed to obtain an objective quality evaluation tool, and there was no quality evaluation tool for the detailed clinical model proved to be valid and reliable.

Accordingly, in the past, it has been difficult for users and developers to grasp the qualities of the detailed clinical model (DCM). As described above, since the user or the authenticator can not be objectively evaluated on the completeness of the detailed clinical model, There is a problem in practical use.

The present invention provides a quality evaluation tool, an evaluation tool development method, and an evaluation method of a detailed clinical model that can objectively evaluate the quality of a detailed clinical model (DCM) There is a purpose.

The present invention has been developed in order to overcome the above problems. The present invention proposes qualitative and quantitative quality requirements that a detailed clinical model must have, and provides a quality evaluation result of a detailed clinical model A tool, an evaluation tool, and an evaluation method thereof.

The present invention has been made in order to solve the above problems, and it is an object of the present invention to provide a detailed clinical model capable of developing a detailed clinical model conforming to a standard for a developer and providing an objective basis for a quality evaluation of a detailed clinical model to a user A quality evaluation tool, an evaluation tool development method, and an evaluation method thereof.

The quality evaluation tool of the detailed clinical model according to an aspect of the present invention includes an input unit for receiving information related to at least one detailed clinical model (DCM) to be evaluated; A database for storing information on at least one evaluation item for evaluating the quality of the detailed clinical model (DCM) and at least one or more detailed evaluation indicators included therein, evaluation criteria, and evaluation results; A central processing unit for evaluating the information of the detailed clinical model (DCM) using the evaluation item, the detailed evaluation index and the evaluation standard, giving a score corresponding thereto, and evaluating the quality based on the given score; And an output unit for outputting a result of the evaluation by the central processing unit through a specific device or a medium, wherein the evaluation item includes a purpose and a scope as a first evaluation item; Participation of stakeholders as a second evaluation item; Strictness in development as a third evaluation item; Clarity as the fourth evaluation item; Compliance with standards as a fifth evaluation item; General methodology as the sixth evaluation item; Metadata as the seventh evaluation item; Management and maintenance as an eighth evaluation item; And at least one of a combination thereof.

A method for evaluating the quality of a detailed clinical model according to an aspect of the present invention includes: a first step of receiving information related to at least one detailed clinical model (DCM) to be evaluated; The evaluation items stored in the database and the detailed evaluation indexes and the evaluation criteria are used to evaluate the information of the detailed clinical model and to give a score corresponding thereto and to evaluate the quality of the detailed clinical model based on the given score, step; And a third step of outputting the evaluation result through a specific apparatus or a medium, wherein the evaluation item includes a purpose and a scope as a first evaluation item; Participation of stakeholders as a second evaluation item; Strictness in development as a third evaluation item; Clarity as the fourth evaluation item; Compliance with standards as a fifth evaluation item; General methodology as the sixth evaluation item; Metadata as the seventh evaluation item; Management and maintenance as an eighth evaluation item; And at least one of a combination thereof.

According to another aspect of the present invention, there is provided a method for developing a quality assessment tool for a detailed clinical model, the method comprising: a first step of defining an evaluation item of a quality assessment tool of a detailed clinical model and an evaluation index for the evaluation item; A second step of performing a validity evaluation on the evaluation item and the evaluation indexes defined above; A third step of evaluating the reliability of the evaluation item and its evaluation indicators defined above; A fourth step of redetermining an evaluation index for which the evaluation results are inconsistent based on the evaluation result, re-executing the validity evaluation of the improved evaluation index, and finalizing the final evaluation item and the evaluation index; And stores information on the finally determined evaluation item, the evaluation index, and the evaluation method in a database.

The present invention provides qualitative and quantitative quality requirements that a detailed clinical model must have for a developer, and provides a quantitative evaluation result of a detailed clinical model to a user by allowing the quality of the detailed clinical model to be objectively evaluated.

In addition, the present invention has the effect of improving the reliability of the detailed clinical model by allowing the developer of the detailed clinical model to develop a detailed clinical model that meets the standard, and providing the user with an objective basis for the quality evaluation of the detailed clinical model.

FIGS. 1A to 1I are diagrams for explaining evaluation items to be evaluated using a quality evaluation tool of a detailed clinical model according to an embodiment of the present invention, and detailed evaluation indicators thereof. FIG.
2 is a block diagram schematically illustrating a configuration of a quality assessment tool of a detailed clinical model according to an embodiment of the present invention;
3 is a flow chart illustrating the operation of a quality assessment tool of a detailed clinical model according to an embodiment of the present invention;
4 is an exemplary diagram illustrating a method for developing a quality assessment tool for a detailed clinical model (DCM) according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, one embodiment of a quality assessment tool and method for evaluating a detailed clinical model according to the present invention will be described with reference to the accompanying drawings.

FIGS. 1A to 1I are diagrams for explaining evaluation items and detailed evaluation indexes of evaluation items using a quality evaluation tool of a detailed clinical model according to an embodiment of the present invention.

As shown in FIG. 1A, the quality evaluation tool of the detailed clinical model evaluates a total of eight evaluation items, and more specifically, a detailed clinical model is evaluated based on a total of 29 evaluation indicators included in each evaluation item do.

Information on the eight evaluation items and the 29 evaluation indicators included therein (reference value for evaluation) is stored in the database DB1. The present embodiment is not intended to limit the above-described evaluation items and the number of the evaluation indexes, and it is possible to add the numbers or change the contents for further evaluation.

Accordingly, the central processing unit 101 of the evaluation tool calculates a score (for example, 0 or or) for a detailed clinical model to be evaluated based on a specific reference value of each information (detailed evaluation index for each evaluation item) stored in the database DB1 1). In addition, it can be configured to give more points to relatively more important evaluation indicators. After the scores are added, the detailed clinical model is evaluated based on the sum of the scores.

The meanings and contents of the eight evaluation items and the 29 evaluation indicators included therein will be described below.

First, the quality evaluation tool of the detailed clinical model evaluates whether the detailed purpose and scope of the detailed clinical model are clearly provided to the user in an understandable level.

For example, as shown in FIG. 1B, the first evaluation index (purpose of DCM) for the first evaluation item (purpose and scope) is to evaluate whether to describe the purpose of developing and using the detailed clinical model content (instance) The detailed clinical model content (instance) (DCM content (instance)) or detailed clinical model metadata (DCM metadata) shall be evaluated and checked for the existence of the description of such purpose. If it is listed, it is scored as 1, and if it is not stated, it is scored as 0 and evaluated.

As shown in FIG. 1B, the second evaluation index (appropriate description of DCM application target) for the first evaluation item is used to evaluate whether or not the application target group to which the DCM content (instance) (Instance) or DCM metadata is evaluated, and it is checked whether or not the information related to the application target and the target populations to which the DCM contents can be applied is written, and information related to the application target and the application target group 1 ", and if the information related to the application target and the application target group is not described, the score is scored as " 0 ". Here, the target group includes descriptive information about a group, such as symptom, disease, sex, age, department of departments and DCM users Or at least one of them.

As described above, the first evaluation item includes the purpose of the detailed clinical model, the setting of information related to the target group, and the details of the user of the detailed clinical model. The DCM user's interdisciplinary, major, Whether there is information about the patient group to be applied, the applicable age and sex, and the like.

The quality evaluation tool of the detailed clinical model is the second evaluation item, and it evaluates whether stakeholder involvement (stakeholder involvement) is involved in the development of the detailed clinical model. The second evaluation item is for evaluating whether or not the demands of the field users have been sufficiently converged.

For example, as shown in FIG. 1C, the first evaluation metric (the clinician-participants included in the development (or design) of DCM) for the second metric (stakeholder engagement) participated in the development or design of DCM Evaluate whether DCM content (instance) or DCM metadata is being evaluated by evaluating whether there were clinicians who have helped to reflect end-user requirements. (clinicians) participated in the study, and if they participated, they scored 1 if they did not participate or 0 if they did not.

And, as shown in Figure 1c, the second metric for the second metric (the number of clinicians participating in the review / approval of DCM - the number of participants) is used to examine the appropriateness of the usage pattern definition DCM content (instance), language (ADL, XML, UML, RDF, etc.), development policies and guidelines by evaluating the number of clinicians participating in the approval The number of clinicians participating in the review and / or approval of the appropriateness of the usage pattern definition of the detailed clinical model is checked for administrative documents, If there are no clinicians participating in the study or if participation is not known, the score is scored as zero.

As described above, the second evaluation item is to evaluate whether the clinician is included in the review and / or approval process for the design (development) of the detailed clinical model, and whether the clinician is included in the evaluation index.

The quality evaluation tool of the detailed clinical model is the third evaluation item and evaluates whether all the data elements (or contents) of the detailed clinical model are developed using the evidence-based clinical knowledge. That is, the third evaluation item is for evaluating the rigor of development in the detailed clinical model development.

For example, as shown in Fig. 1d, the first evaluation index (referring to reference (s) used in DCM development) for the third evaluation item (rigor in development) refers to whether the detailed clinical model data elements refer to the identified source (Eg, clinical practice guidelines, medical textbooks, current practice, and so on) to assess DCM content (instances) and DCM metadata. And evaluates the content by scoring 1 for an appropriate reference and 0 for an inadequate reference.

As shown in FIG. 1d, the second evaluation index (accuracy (the preferred embodiment of the usage pattern)) for the third evaluation item indicates whether the detailed clinical model reflects current universal use or preferred use, The data element and its possible values are finely defined to evaluate whether the required data elements meet the possible values and to evaluate the data elements of the DCM content (instance) 0 if there is little or no usage pattern captured in the data elements, 0 if there is a usage pattern captured in the data elements, according to the clinical evaluation of the preferred use case of the usage pattern (or the treatment pattern) .

As described above, the data elements of the detailed clinical model are derived from clinical care guidelines, medical textbooks, and main care path guidelines. In addition, they are made by reflecting current medical practice patterns. Therefore, And the like. That is, the detailed clinical model to be evaluated is composed of indicators that evaluate whether the field user expresses the clinical data element (data element: name, value, object, attribute) accurately, detailed and concretely.

The quality evaluation tool of the detailed clinical model is the fourth evaluation item and evaluates whether the language or format of the detailed clinical model is clearly expressed. In other words, the fourth evaluation item is for evaluating clarity and presentation, including whether or not to use a formal modeling language of the detailed clinical model, coding for data processing of a computer, maintenance of a consistent data element representation, The accuracy of cardinality is included as an evaluation index.

For example, as shown in FIG. 1E, a first evaluation index (a formalized syntax) for the fourth evaluation item (clarity and presentation) is evaluated as to whether or not the expression is unambiguously expressed in a formalized syntax. UML, RDF, etc.), and checks whether the DCM content is unambiguously expressed in the formatted syntax. If the formatted syntax is not used, it is scored as "0" do.

As shown in FIG. 1E, the second evaluation index for the fourth evaluation item is an indicator for evaluating whether or not the DCM text data elements are coded (DCM text data elements). The DCM contents are encoded . At this time, the item should be coded if possible, except for developer name or numeric value. The evaluation index is used to evaluate data elements and languages (e.g., codes of data elements described by ADL, XML, UML, RDF, etc.), and checks whether codes of each data element exist. 1 if the code is applicable, and 0 if no data elements have a code.

As shown in FIG. 1E, the third evaluation index for the fourth evaluation item is an indicator for evaluating whether or not the cardinality is appropriate use of cardinality, and is an evaluation index of whether cardinality appropriate to the DCM content is used to be. The evaluation indexes are evaluated for DCM content, and cardinality used for data elements of the DCM content is checked for appropriateness and scored as 1 if appropriate and 0 for inappropriate cardinality.

As shown in FIG. 1E, the fourth evaluation index for the fourth evaluation item is an indicator for evaluating whether or not the DCM domain is a precise mention of the DCM domain. If the DCM domain name is the home page for the DCM content to be evaluated or the DCM domain name It is an index of whether or not it is mentioned correctly in places such as repositories (DCM repositories). It should be precisely addressed to the domain name and the extent to which the domain instance is mutually matched, such as, for example, the medication domain and the medication order. The evaluation index is used to evaluate the homepage of the DCM content (instance) or the document in which the domain is specified, checks whether the DCM content (instance) belongs to which domain, and if the domain and the instances of the domain match well And 1 if the domain and instances of the domain are not matched at all.

The quality evaluation tool of the detailed clinical model is the fifth evaluation item. It evaluates compliance to standard as a core requirement for medical information exchange and recyclability guarantee based on the detailed clinical model. Type, and evaluation indicators for evaluating the use of standard measurement units.

For example, as shown in FIG. 1F, the first evaluation index for the fifth evaluation item (conformity to standards) is an indicator for evaluating whether or not the use of international standard terminology exists. Evaluate whether DCM includes international standard terminology or whether DCM has mapped international standard terminology (if applicable and if applicable). The evaluation indexes are evaluated on the terms combination of DCM contents (instance), language (codes of data elements described by ADL, XML, UML, RDF, etc.) It checks whether the term is mapped and scored 1 if the international standard terminology is used or not, or 0 if the international standard terminology is not used.

As shown in FIG. 1F, the second evaluation index for the fifth evaluation item is an index for evaluating whether or not the use of the international standard data type is used, and the international standard data type or the DCM content (Instance) data type to use or map the HL7 V3 XML data type. The evaluation index is used to evaluate DCM instance data type references. It is checked whether an international standard data type is used for DCM contents (instance). If the international standard data type is used, 1 is used, and the international standard data type is used If not, the score is scored as 0.

As shown in FIG. 1F, the third evaluation index for the fifth evaluation item is an indicator for evaluating whether or not the use of the international standard units of measure is the UCUM in the detailed clinical model (DCM) Evaluate whether international standard measurement units such as SNOMED CT are being used or whether the DCM content (instance) uses measurement units that are mapped to international measurement units. The evaluation indexes are evaluated for DCM instance measurement unit combinations, and it is checked whether there is code for each appropriate data element. If the international standard measurement unit is used, 1, and if the international standard measurement unit is not used 0 "

As shown in FIG. 1F, the fourth evaluation index for the fifth evaluation item is an index for evaluating an examination of use of data types, and indicates whether the data type is appropriately used for the DCM content (instance) . The evaluation index is used to evaluate the DCM content (instance), and checks whether the data type is appropriately used in the DCM content (instance). If the data type is well specified and appropriately used, If it is not used properly or not used properly, it is scored as 0 and evaluated.

As shown in FIG. 1F, the fifth evaluation index for the fifth evaluation item is an indicator for evaluating exception handling (s) to the use of international terms (DCM) organization does not have exceptions at the organizational level (eg, provision of local codes, international standard terminology systems of interest) Requests for addendum) should be specified and evaluated. The evaluation index should be related to the document (for example, internal guidelines or policy), and if the use exception is processed through the review of related documents or interviews, do.

As shown in FIG. 1F, the sixth evaluation index for the fifth evaluation item is an index for evaluating the examination of the quality of the mapping to the international standard terminology, Evaluate whether there is a process to test the suitability of the mapping in standard terms. The evaluation index is an evaluation target of a quality assurance policy (a guideline), an implementation document (a guideline), a homepage post, and the like. It is checked whether the mapping quality control process is specified to the evaluation targets, 1, and if the mapping quality control process is not specified in the evaluation subjects, the score is scored as 0 and evaluated.

The quality assessment tool of the detailed clinical model is the sixth evaluation item, and the general methodology classifies the main attributes of the detailed clinical model and evaluates whether it satisfies it. If the other items of evaluation are items for individual detailed clinical models, the general methodological evaluation item corresponds to an evaluation item on the overall modeling methodology, and includes a modification (or modifier), a source of information, And an evaluation index for evaluating whether the detailed clinical model supports the attributes related to the negation of the data element.

For example, as shown in FIG. 1G, the first evaluation index for the sixth evaluation item (general methodology) is an indicator for evaluating whether or not the use of modifiers is used, and whether modifiers are used for all DCMs The donor's information in the chart of the patient, the prenatal baby, the transplanted patient listed on the maternal chart). The metric can be used in DCM developer interviews (applicable if no related documents are found, general access is not allowed, or there are no instances found in DCM instances), documents (for data elements and attributes) , And DCM content (instances), and through interviews and document reviews of relevant people, score 1 if the modifier is not used or applicable, and 0 if the modifier is not used.

As shown in FIG. 1G, the second evaluation index for the sixth evaluation item is an index for evaluating information provenance, and it is evaluated whether or not the specification of the source of the information provided by the patient or the clinician is evaluated do. The metric can be used in DCM developer interviews (applicable if no related documents are found, nor is access allowed to the public, or there are no instances found in DCM instances), documents (data elements and attributes ), The DCM content (instance), and the reference model, and if the information source attribute is present (or used) or not applicable through interviews and document reviews of relevant people, 1, the information source attribute is valid If not, the score is scored as 0.

As shown in FIG. 1G, the third evaluation index for the sixth evaluation item is an indicator for evaluating whether or not an appropriate support for negation is provided. The negative evaluation mechanism is applied to the detailed clinical model (DCM) Is supported. The metric can be used in DCM developer interviews (applicable if no related documents are found, general access is not allowed, or there are no instances found in DCM instances), documents (for data elements and attributes) , DCM content (instance), and if the negative mechanism is not supported or applicable through the interview and document review of related people, 1 is scored and scored as 0 if the negative mechanism is not supported.

As shown in FIG. 1G, the fourth evaluation index for the sixth evaluation item is an index for evaluating whether or not the support is appropriate support for past history, and is a time for past specific time representation of the statement Evaluates whether historical time stamps are supported. The metric can be used in DCM developer interviews (applicable if no related documents are found, general access is not allowed, or there are no instances found in DCM instances), documents (for data elements and attributes) , And detailed clinical model (DCM) are evaluated. In cases where past history exists or past history can not be applied through relevant interviews, search and document review on the homepage, 1, there is a past support mechanism If not, the score is scored as 0.

As shown in FIG. 1G, the fifth evaluation index for the sixth evaluation item is an indicator for evaluating whether or not an appropriate provision of narrative text is provided. Instead of the coded value itself, "In addition to the coded value, to support character comments in the DCM content (instance). The evaluation index is used for evaluating a document, a DCM content (instance) or a language (ADL, XML, UML, RDF, etc.), checking whether an allowed value and attributes exist, 1 if the text can not be applied, and 0 if the descriptive text for the data element is not allowed.

The quality evaluation tool of the detailed clinical model is the seventh evaluation item and evaluates the faithfulness of the metadata. The metadata is not directly related to the quality of the detailed clinical model, but it provides important information to the user, and it is composed of the evaluation index which evaluates the detailed clinical model developer, the version, the accessibility of the metadata, and the range of the metadata.

For example, as shown in FIG. 1H, the first evaluation index for the seventh evaluation item (metadata) evaluates whether or not information on the developer (or organization) of the DCM content (instance) is described. The evaluation index is used for evaluating DCM contents (instance), DCM metadata or languages (ADL, XML, UML, RDF, etc.), checking whether there is a description for the developer (or organization) , 1 if there is no description of at least one of them, and 0 if there is no description of the developer or organization.

As shown in FIG. 1h, the second evaluation index for the seventh evaluation item is an indicator for evaluating whether or not there is an appropriate description of the DCM user's training. The DCM content (instance) (Eg, doctors, surgeons, nurses, and pharmacists) who are trained in the use of the drug. The evaluation index is an evaluation target of the DCM content (instance) and the DCM metadata. The evaluation index indicates the user and a description of sufficiency, and if the description exists, the evaluation index is scored as 1, .

As shown in FIG. 1H, the third evaluation index for the seventh evaluation item is an indicator for evaluating whether version information (DCM version) for the DCM content (instance) is included. The DCM content (instance) , The language (ADL, XML, UML, RDF, etc.) is evaluated and whether or not version information and its actual meaning (e.g., whether it is published or draft) If there is no number, the score is scored as 0.

As shown in FIG. 1H, the fourth evaluation index for the seventh evaluation item is an index for evaluating the accessibility of metadata, and evaluates whether the metadata related to the DCM content (instance) is accessible to the public do. The evaluation index is used for evaluating DCM contents (instance) or DCM metadata. It is checked whether the metadata is publicly accessible in at least one of the home page and the DCM repository. If the metadata is publicly accessible, If the data is not publicly accessible, score it as 0 and evaluate.

As shown in FIG. 1H, the fifth evaluation index for the seventh evaluation item is an index for evaluating the extent of metadata, and evaluates how much metadata is provided in the DCM content (instance) . The evaluation metrics are targeted for evaluation of DCM content (instance) or DCM metadata and include a development date, an expiry date, a last-update date, an approval organization, The metadata element (s), such as the clinical keyword (s), are generated and checked for accessibility. If the metadata is sufficient (fully specified), the metadata element If insufficient, it is scored as 0 and evaluated.

As shown in FIG. 1h, the sixth evaluation index for the seventh evaluation item evaluates whether DCM provides open access to the review and / or approval results of DCM. As an indicator for DCM, the public accessibility of DCM's review and / or approval results is assessed. The evaluation index is evaluated based on the DCM metadata (approver, approval authority), administrative document, homepage, etc., and it is checked whether the result is accessible and whether the approval date, the content If there is information that can be accessed, it is scored as 1, and if there is no information about it, it is scored as 0.

The quality assessment tool of the detailed clinical model is the eighth evaluation item. It evaluates the activities for management and maintenance of the detailed clinical model, and the responsible maintenance agency, the feedback mechanism that can collect the improvement opinion of the user, etc. And the like.

For example, as shown in FIG. 1i, the maintenance organization of DCM for the eighth evaluation item (management and maintenance) is for evaluating the public accessibility of the DCM maintenance agency. The evaluation indexes are DCM metadata, administrative documents, and homepages, and it is checked whether the information is accessible. If the accessibility of the maintenance agency is 1, if the accessibility is not possible, 0 .

As shown in FIG. 1I, a second evaluation index (DCM) for the eighth evaluation item is an indicator for evaluating whether or not a user feedback mechanism exists for the DCM. Instance), or a request for additions to data elements. The evaluation index includes a DCM homepage or a repository as an evaluation target, and checks whether feedback can be provided through the DCM homepage or repository. If the feedback mechanism is valid, the evaluation index is 1; if the feedback mechanism is not valid, .

In evaluating the quality of the detailed clinical model as described above, the evaluation of the score is based on a predetermined standard, and it is determined whether or not the evaluation index is satisfied, and a score is given to a specific value according to at least one evaluation index , And the number of evaluation indicators can be increased for more detailed evaluation.

2 is a block diagram schematically illustrating a configuration of a quality assessment tool of a detailed clinical model according to an embodiment of the present invention. The quality evaluation tool of the detailed clinical model includes a central processing unit 101, an input unit 102, an output unit 103, and a database DB1.

The input unit 102 is a means for receiving data related to a detailed clinical model (DCM) to be evaluated, and receives data of characters, figures, voices, numbers, and codes in an online or offline manner.

The off-line method may be applied to a variety of devices such as a keyboard, a mouse, a joystick, a tablet, a touch screen, an optical mark reader (OMR), a barcode reader, a magnetic ink character reader (MICR), a scanner, SD, CD-ROM, DVD-ROM), or a separate computer system, or a combination thereof. The on-line method receives data using at least one of a wired / wireless network (e.g., LAN, Internet, WiFi, Bluetooth), a broadcasting network, and a mobile communication network or a combination thereof.

The input unit 102 receives data related to the detailed clinical model (DCM) to be evaluated manually or automatically.

In order to manually input the data, the user interface information related to the evaluation item is output, and the evaluator can proceed with the evaluation according to the user interface information. For example, a user interface screen capable of inputting each evaluation item (or evaluation index) of the detailed clinical model (DCM) is output, and the evaluator (or developer) And then input the data in the format of

In order to automatically receive the data, the data structure of the detailed clinical model (DCM) is analyzed and the data is detected in a specific area (for example, a data header part, an index part) storing data related to the evaluation item, Automatically convert to the specified format and output.

The database DB1 (or database server) stores eight evaluation items for evaluation of the detailed clinical model (DCM), information on the 29 evaluation indicators included therein, and reference values for evaluation. As described above, the evaluation item and the number of the evaluation indexes can be further added for detailed evaluation, and it is also possible to modify or delete the contents of the evaluation items and the evaluation indexes (or reference values).

At this time, the user interface program for receiving data related to the detailed clinical model (DCM) to be evaluated may be provided with a separate memory (not shown) or may be stored in a specific area of the database DB1.

In addition, the evaluation data of the detailed clinical model (DCM) is created and managed in a separate area (for example, a folder or a separate database) so that it can be distinguished from other detailed clinical models (DCM). The evaluation scores assigned to each detailed clinical model (DCM) during the evaluation are stored in a folder of the corresponding detailed clinical model (DCM). In addition, the evaluation score given to each detailed clinical model (DCM) can be modified by the evaluator (or the reviewer). When the evaluation score is modified as described above, the name of the evaluator (or the reviewer), the reason for revising the evaluation score, and the date of modification (time) are also stored.

The central processing unit 101 may be a control unit such as a CPU or a separate computer system and may be provided with a reference value for an evaluation item stored in the database DB1 and its detailed evaluation indexes and a reference value for a detailed clinical model (Or manually input data) are compared and evaluated. (E.g., 0) when the data related to the first evaluation index of the detailed clinical model (DCM) does not satisfy the predetermined reference value, assuming that the first evaluation index of the first evaluation item is evaluated. If the predetermined reference value is satisfied, a corresponding score (for example, 1) is given.

At this time, for more detailed evaluation of the detailed clinical model (DCM), a more varied score may be given according to the quality. That is, a more diverse reference value is set for one evaluation index, and a corresponding score is given as the reference value is satisfied. For example, if it is assumed that the first evaluation index is evaluated from 0 to 3 points, the first to third reference values are set, and when the first reference value is not satisfied, 0 is set, and between the first reference value and the second reference value 2 points are satisfied when the first reference value is satisfied, and 3 points are satisfied when the second reference value and the third reference value are satisfied.

As described above, each evaluation index is evaluated and a score is added to each evaluation score. In the case of constructing to give more various scores, average value may be calculated for evaluation.

According to the evaluation method, if the summed score is equal to or greater than a preset reference score, it is determined to be acceptable, and if it is less than the reference score, it is determined to be rejected.

At this time, the evaluation method may be configured to judge the quality level (for example, upper / middle / lower) of the quality rather than merely determine the acceptance and rejection, or to output only the objective evaluation score for each evaluation item, The quality of the model can be determined.

The output unit 103 may be a device (e.g., a speaker or a monitor) for outputting a voice or an image, a printing device (e.g., a printer) or a separate computer system The results of the evaluation are provided either offline or online according to a predetermined level of security policy for the user or the authorized user.

3 is a flowchart illustrating an operation of a quality assessment tool of a detailed clinical model according to an embodiment of the present invention.

The quality evaluation tool of the detailed clinical model receives information related to the detailed clinical model (DCM) to be evaluated (e.g., evaluation item and data corresponding to the evaluation index) (S101). The information related to the detailed clinical model (DCM) is input manually or automatically.

Then, the reference value for each evaluation item and the detailed evaluation index stored in advance is compared with the information related to the input detailed clinical model (DCM), and a score corresponding to the comparison result is given (S102). The comparative evaluation is preferably performed automatically according to a preset program, and it may also involve the evaluator's opinion if necessary.

If the evaluation is not completed for all of the evaluation indexes (NO in S103), the scores evaluated for the respective evaluation indexes are added up (S104), and the evaluation indexes are switched to evaluation indexes in the next step so as to be evaluated ).

When the evaluation is completed for all the evaluation indexes (YES at S103) by repeating steps S102 and S105, the summed evaluation score is compared with the designated reference score (S106). If the combined evaluation score is equal to or higher than the reference score, it is determined to be acceptable, or a quality level (e.g., upper / middle / lower) is determined based on the combined score (S107). However, if the summed evaluation score is less than the reference score, it is determined to be rejected (S108).

(Such as monitor, printer, speaker, or other computer system) if the pass / fail or quality level is determined using the combined score, as described above, Or via a medium (e.g., home page, e-mail, text message, broadcast, printed matter) (S109).

On the other hand, when the evaluation is completed without judging the pass / fail or the quality level, the evaluation score is merely disclosed so that the user can directly judge the quality of the detailed clinical model (DCM) by referring to the evaluation score.

4 is an exemplary diagram illustrating a method for developing a quality assessment tool for a detailed clinical model (DCM) according to an embodiment of the present invention.

The development of a quality assessment tool for the detailed clinical model corresponds substantially to setting a quality evaluation item and its detailed evaluation index stored in the database (or database server). Hereinafter, the quality evaluation items of the detailed clinical model and the method of setting the detailed evaluation indexes thereof will be described.

As a first step, the evaluation of the quality assessment tool of the detailed clinical model (DCM) (S201) based on the existing quality requirements (or items) with reference to the core document (or any evaluation tool) An evaluation index is defined as a sub-structure for the item and the evaluation item (S202). The above-described evaluation index corresponds to an abstract quality requirement (or item) since it has not yet undergone the specification step.

At this time, evaluation item (or evaluation area) of the clinical care instruction evaluation tool (AGREE tool) is utilized to define the quality evaluation item. Here, the quality criteria of the archetype of the existing quality requirements are used first in the definition of the above-mentioned evaluation index because the archetype is the international standard model of the detailed clinical model and the quality criterion is most comprehensive. Here, the definition of the evaluation index, the evaluation object, the evaluation standard, and the scoring method are also defined as the sub-structure of the evaluation index.

Then, the above-described evaluation index is specified at a measurable level (S203). The work for this specification refers to the conversion of abstract existing quality requirements / items into measurable metrics. Through the above operation, the evaluation items of the quality assessment tool of the detailed clinical model (DCM) and the evaluation indexes as its subordinate structure are specified (S204).

For reference, the methodology of the ISO / IEC 9126 software quality evaluation model is utilized to carry out the specification work of the abovementioned evaluation indexes. At this time, in defining the quality requirements that have been abstractly defined as abstract evaluation criteria, it is necessary to include all of the 'clinical use' as the criteria for inclusion and exclusion of existing quality requirements, (Eg quality requirements for building a repository) are excluded.

On the other hand, if the evaluation items of the detailed clinical model and the evaluation indexes, which are the sub-structures thereof, are defined as described above, the validity of the evaluation items and the evaluation indexes defined above is performed as the second step.

An expert panel will be formed for the above validity evaluation.

The method for constructing the expert panel is as follows. First, a keyword (eg, DCM, archetype, clinical model, quality criteria) is input through a specific search means (eg, Medline and Google Scalar) . You can specify the search target (eg Journal, JAMIA, Methods of Information in Medicine) and the search period (for example, 1990 to 2010). That is, a list of experts of a predetermined number of persons is collected through the search operation.

Second, obtain a list of experts working in working groups related to detailed clinical models in international standards bodies (eg ISO, HL7, CEN). At this time, the experts collected in the first method and the experts working in the international standards organization may overlap.

Third, panel participation of experts in representative models of detailed clinical models (eg, clinical factor models, archetypes) is made at the same rate. The selected panel is divided into two expert groups (first expert group and second expert group). The reason is that if a panel evaluates both the validity and the reliability of the evaluation tool, the workload of the panel is overloaded so as to induce high participation. The first expert group performs the surface validity evaluation (S205), and the second expert group performs the content validity evaluation using the Delphi technique (S207).

Here, the above-mentioned face validity evaluation includes the suitability as a quality evaluation index of a selected evaluation index, suitability as a quality improvement index, applicability as another detailed clinical model evaluation index, and suitability of an evaluation method.

The qualitative and quantitative validity evaluation result performed by the first expert group is fed back to the researcher (or developer) to modify the evaluation index (S206). At this time, the surface validity evaluation should be divided into the first and the second for the validity evaluation of the evaluation index which does not agree with the evaluation results of the first expert group (for example, two or more experts).

The content validity evaluation is conducted over two rounds using the Delphi technique.

First, the first round of Delphi is an open-ended questionnaire survey, and if a structured questionnaire is available based on extensive literature review, a first round of modified Delphi is possible and acceptable. For example, a questionnaire is mailed to international expert panels over a specific period of time, and the expert panel provides information on research objectives, research content, and Delphi techniques. (2) the descriptive description of the evaluation index; (3) the appropriateness of the evaluation index; (4) the possibility of improving the quality of the evaluation index; and (5) Applicability to clinical models); and (6) professional modification or amendment. The scale of the questionnaire response uses a 4-point scale (eg, scale 4 - strong consent, scale 3 - agreement, scale 2 - asynchrony, scale 1 - strong asynchrony).

And the second round of Delphi will be conducted for the purpose of reaching consensus on unadjacent evaluation indicators in the Delphi Round 1. To do this, we provide the average value of the responses of the entire panel in each round of each evaluation index and the score of the panel to the Delphi panel, and at the same time, provide the opinions of all panels gathered in the first round.

As described above, the evaluation indexes are agreed through Delphi 1 round and 2 round (S208). In this case, the expert consensus standard is different according to the researcher and the specific topic to be agreed, and in some fields, 50% or more of the panel agrees with the consent of the panel. In this embodiment, And the sum of the fractions of the panels responding to "1" and "2" is greater than the sum of the fractions of the panels whose responses to "3" and "4" Of the total number of participants in the first round, and the evaluation indicators that achieved the agreement in the first round are excluded from the second round questionnaire.

On the other hand, if there is an unsettled evaluation index as a result of the Delphi 2 round (S209), the conceptual mapping between the unadjacent evaluation indexes and the agreed evaluation indices between the expert panels is performed (S210). The conceptual mapping is aimed at understanding whether the rejection evaluation index overlaps with the acceptance evaluation index, and maps the concept between acceptance / rejection evaluation index and acceptance / debate evaluation index.

After the validity evaluation as described above, in the third step, the evaluation items of the quality assessment tool of the detailed clinical model and the reliability of the evaluation indexes, which are a substructure thereof, are evaluated.

The reliability of the quality evaluation tool of the detailed clinical model is evaluated by the degree of agreement of the evaluation results of the evaluator (e.g., two or more experts) by the first expert group (S211). Wherein the inter-rater agreement is calculated by a kappa coefficient, and the kappa coefficient provides an actuality measure of the rater which excludes the likelihood of coincidence by the rater when the rater is measured by the rater. In this embodiment, the kappa coefficient is set to a threshold value of 0.60 or more, and this value belongs to the "good agreement level" according to the kappa coefficient analysis criterion (S212). Here, the evaluation object sets the openEHR archetype, (S213). When the reliability is evaluated as described above, the evaluation index is checked (S213). If the evaluation index is not correct, the evaluation index is corrected (S214, S215).

If the evaluation index is revised as described above, the fourth step is to determine the evaluation items of the quality assessment tool of the detailed clinical model and the evaluation indexes as its sub-structure. For example, the modified evaluation index again performs the validity evaluation of the first expert group (S216), modifies the evaluation index of the evaluation tool (S217), and again, through the Levy (S218) And is finally confirmed as an evaluation index of the quality evaluation tool of the clinical model (S219, S220). For reference, the improvement of the evaluation index of the evaluation tool improves the evaluation index of the evaluation tool according to the type of inconsistency detected after conducting an e-mail survey to the evaluator.

As described above, the evaluation indexes modified or confirmed through the first to fourth steps are stored in the database of the quality evaluation tool of the detailed clinical model including the evaluation criteria.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. I will understand. Accordingly, the technical scope of the present invention should be defined by the following claims.

101: central processing unit 102: input unit
103: Output DCM: Detailed clinical model
DB1: Database

Claims (14)

An input unit for receiving information related to at least one detailed clinical model (DCM) to be evaluated;
A database for storing information on at least one evaluation item for evaluating the quality of the detailed clinical model (DCM) and at least one or more detailed evaluation indicators included therein, evaluation criteria, and evaluation results;
A central processing unit for evaluating the information of the detailed clinical model (DCM) using the evaluation item, the detailed evaluation index and the evaluation standard, giving a score corresponding thereto, and evaluating the quality based on the given score;
And an output unit for outputting the evaluation result of the central processing unit through a specific device or medium,
In the evaluation item,
Purpose and scope as the first evaluation item; Participation of stakeholders as a second evaluation item; Strictness in development as a third evaluation item; Clarity as the fourth evaluation item; Compliance with standards as a fifth evaluation item; General methodology as the sixth evaluation item; Metadata as the seventh evaluation item; Management and maintenance as an eighth evaluation item; Or a combination of these. ≪ RTI ID = 0.0 > 8. < / RTI > The method according to claim 1,
Wherein the first evaluation item includes:
A 1-1 evaluation index for assigning a score according to an evaluation criterion for a description of purpose for developing and using the detailed clinical model content;
A first 1-2 evaluation index for assigning a score according to an evaluation criterion for describing information related to an application target and target populations to which DCM contents can be applied; Or a combination thereof;
Wherein the second evaluation item includes:
A 2-1 evaluation index that assesses how many clinicians participated in the development or design of DCM and helped to reflect end-user requirements, and scored according to the evaluation criteria;
Evaluation of the use pattern definition, and evaluation of the number of clinicians participating in the approval. Or a combination of these. ≪ RTI ID = 0.0 > 8. < / RTI > The method of claim 2,
The third evaluation item may include:
3-1 evaluation index that evaluates the detailed clinical model data elements by referring to the identified sources and scoring according to the evaluation criteria;
A 3-2 evaluation index for assigning a score according to an evaluation criterion for a clinical evaluation of a preferred use case of a usage pattern (or a medical care pattern) in data elements of the DCM contents; Or a combination thereof;
Wherein the fourth evaluation item includes:
Evaluating whether or not the DCM contents are expressed unambiguously by the formalized syntax, by checking whether the DCM contents are expressed unambiguously in the formulated syntax, and assigning scores according to the evaluation criteria and evaluating them;
A 4-2 evaluation index for evaluating whether the DCM text data elements are coded or not by checking whether the code of each data element is present and giving a score according to the evaluation criterion;
An index for evaluating cardinality, wherein the cardinality used for the data elements of the DCM contents is checked, and a score is given according to the evaluation criteria;
As an index for evaluating whether or not the DCM domain is accurately referred to, it is checked whether the DCM content belongs to a domain, and it is checked whether the domain and the instances of the domain match well. 4 Evaluation indicators; Or a combination of these. ≪ RTI ID = 0.0 > 8. < / RTI > The method of claim 3,
Wherein the fifth evaluation item includes:
An indicator for evaluating whether or not the use of the international standard terminology is used, a combination of the terms in the DCM instances are mapped to the international standard terminology, and a score is given according to the evaluation standard to evaluate the 5-1 evaluation index;
An indicator for evaluating whether or not an international standard data type is used, a rating index for evaluating whether an international standard data type is used for DCM contents by assigning a score according to the evaluation standard;
An indicator for evaluating the use of the international standard measurement unit, which checks whether the code for each data element exists and evaluates it by assigning a score according to the evaluation standard for the use of the international standard measurement unit. Indicators;
An indicator for evaluating the use test of the data type, wherein the data type is used in the DCM contents to check whether the data type is well specified and used;
An indicator for evaluating the exception handling of the use of international terms, wherein the score is given according to an evaluation criterion for the extent to which the use exception has been processed;
As an index for evaluating the quality test of the mapping for the international standard terminology, it is checked whether the mapping quality control process is specified to the evaluation subjects, and a score is given according to the evaluation standard for the degree to which the mapping quality control process is specified 5-6 evaluation index to be evaluated by the evaluation; Or a combination thereof;
In the sixth evaluation item,
An indicator for assessing whether a modifier is used, which is evaluated by assigning a score according to an evaluation criterion for the degree to which the modifier is used;
6-2 evaluation index that evaluates the source of information by assigning a score according to the evaluation criteria for the degree to which the information source attribute exists (or is used) or applied;
An indicator for assessing whether or not support for negation is supported by evaluating the degree to which the negation mechanism is supported in the detailed clinical model (DCM) ;
6-4 evaluation index that evaluates whether or not support for past history is given by rating according to evaluation criteria for the degree of presence of past history;
Evaluation as to whether or not the descriptive text is provided is made by checking whether the permitted values and attributes are present and assigning a score according to the evaluation criteria for the extent to which the descriptive text for the data element is allowed. Indicators; Or a combination of these. ≪ RTI ID = 0.0 > 8. < / RTI > The method according to any one of claims 1 to 4,
In the seventh evaluation item,
7-1 evaluation that evaluates whether the developer (or organization) of the DCM contents is listed with the score in accordance with the evaluation criteria for the degree of writing to the developer (or organization) Indicators;
7-2 evaluation index that evaluates whether there is a description of the training of the DCM user as a score according to the evaluation criteria for the description of the user and the qualification;
A seventh to eighth evaluation index for evaluating whether or not version information (DCM version) for the DCM contents is included, the seventh to eighth evaluation indexes being scored according to the evaluation criteria for the existence of the version information and its actual meaning;
An index for evaluating the accessibility of the metadata, wherein the score is assigned according to an evaluation criterion for whether the metadata is publicly accessible in at least one of the home page or the DCM repository;
As an indicator for evaluating the extent of the metadata, the metadata element (s), such as development date, expiration date, last update date, approval authority, or clinical keyword, are generated and evaluated based on evaluation criteria 7-5 Evaluation index given and evaluated;
An indicator for assessing whether DCM provides a public access to the results of the review, the outcome of the endeavor, or a combination of these, as an indication of whether the outcome is accessible and whether the endorsement date, content for the endorsement Or an approval expiration date; and (7) a rating indicator for evaluating whether or not there is information accessible from among the approval expiration dates by assigning a score according to the evaluation criteria; Or a combination thereof;
The eighth evaluation item may include:
An indicator for assessing the public accessibility of the DCM maintenance agency; an 8-1 evaluation index for rating the information access based on evaluation criteria;
An indicator for evaluating whether or not a user's feedback mechanism exists for the DCM, and evaluating whether the feedback is provided through the DCM homepage or the repository; Or a combination of these. ≪ RTI ID = 0.0 > 8. < / RTI > In a detailed clinical model quality evaluation method using a detailed clinical model quality evaluation tool,
A first step of receiving information related to at least one detailed clinical model (DCM) to be evaluated by the evaluation tool;
The evaluation tool evaluates the information of the detailed clinical model using the evaluation items stored in the database, the detailed evaluation indexes and the evaluation criteria, and gives a score corresponding thereto. Based on the given score, the quality of the detailed clinical model A second step of evaluating;
And a third step of the evaluation tool outputting the evaluated result through a specific apparatus or a medium,
In the evaluation item,
Purpose and scope as the first evaluation item; Participation of stakeholders as a second evaluation item; Strictness in development as a third evaluation item; Clarity as the fourth evaluation item; Compliance with standards as a fifth evaluation item; General methodology as the sixth evaluation item; Metadata as the seventh evaluation item; Management and maintenance as an eighth evaluation item; Or a combination thereof. ≪ Desc / Clms Page number 19 > The method of claim 6,
Wherein the first evaluation item includes:
A 1-1 evaluation index for assigning a score according to an evaluation criterion for a description of purpose for developing and using the detailed clinical model content;
A first 1-2 evaluation index for assigning a score according to an evaluation criterion for describing information related to an application target and target populations to which DCM contents can be applied; Or a combination thereof;
Wherein the second evaluation item includes:
A 2-1 evaluation index that assesses how many clinicians participated in the development or design of DCM and helped to reflect end-user requirements, and scored according to the evaluation criteria;
Evaluation of the use pattern definition, and evaluation of the number of clinicians participating in the approval. Or a combination thereof. ≪ Desc / Clms Page number 19 > The method of claim 7,
The third evaluation item may include:
3-1 evaluation index that evaluates the detailed clinical model data elements by referring to the identified sources and scoring according to the evaluation criteria;
A 3-2 evaluation index for assigning a score according to an evaluation criterion for a clinical evaluation of a preferred use case of a usage pattern (or a medical care pattern) in data elements of the DCM contents; Or a combination thereof;
Wherein the fourth evaluation item includes:
Evaluating whether or not the DCM contents are expressed unambiguously by the formalized syntax, by checking whether the DCM contents are expressed unambiguously in the formulated syntax, and assigning scores according to the evaluation criteria and evaluating them;
A 4-2 evaluation index for evaluating whether the DCM text data elements are coded or not by checking whether the code of each data element is present and giving a score according to the evaluation criterion;
A 4-3 evaluation index for checking the cardinality used for the data elements of the DCM contents as an index for evaluating the cardinality, assigning a score according to the evaluation criteria, and evaluating the cardinality;
As an index for evaluating whether or not the DCM domain is accurately referred to, it is checked whether the DCM content belongs to a domain, and it is checked whether the domain and the instances of the domain match well. 4 Evaluation indicators; Or a combination thereof. ≪ Desc / Clms Page number 19 > The method of claim 8,
Wherein the fifth evaluation item includes:
An indicator for evaluating whether or not the use of the international standard terminology is used, a combination of the terms in the DCM instances are mapped to the international standard terminology, and a score is given according to the evaluation standard to evaluate the 5-1 evaluation index;
An indicator for evaluating whether or not an international standard data type is used, a rating index for evaluating whether an international standard data type is used for DCM contents by assigning a score according to the evaluation standard;
An indicator for evaluating the use of the international standard measurement unit, which checks whether the code for each data element exists and evaluates it by assigning a score according to the evaluation standard for the use of the international standard measurement unit. Indicators;
An indicator for evaluating the use test of the data type, wherein the data type is used in the DCM contents to check whether the data type is well specified and used;
An indicator for evaluating the exception handling of the use of international terms, wherein the score is given according to an evaluation criterion for the extent to which the use exception has been processed;
As an index for evaluating the quality test of the mapping for the international standard terminology, it is checked whether the mapping quality control process is specified to the evaluation subjects, and a score is given according to the evaluation standard for the degree to which the mapping quality control process is specified 5-6 evaluation index to be evaluated by the evaluation; Or a combination thereof;
In the sixth evaluation item,
An indicator for assessing whether a modifier is used, which is evaluated by assigning a score according to an evaluation criterion for the degree to which the modifier is used;
6-2 evaluation index that evaluates the source of information by assigning a score according to the evaluation criteria for the degree to which the information source attribute exists (or is used) or applied;
An indicator for assessing whether or not support for negation is supported by evaluating the degree to which the negation mechanism is supported in the detailed clinical model (DCM) ;
(6) An indicator for assessing support for past history, which evaluates the score according to an evaluation criterion for the degree of the presence of a past history;
A sixth to eve evaluation index for evaluating the descriptive text by checking whether or not permitted values and attributes are present and assigning a score according to an evaluation criterion for the extent to which the descriptive text for the data element is allowed; Or a combination thereof. ≪ Desc / Clms Page number 19 > The method according to any one of claims 6 to 9,
In the seventh evaluation item,
7-1 evaluation that evaluates whether the developer (or organization) of the DCM contents is listed with the score in accordance with the evaluation criteria for the degree of writing to the developer (or organization) Indicators;
An indicator for evaluating the description of the training of the DCM user, the score being given according to the evaluation criteria of the user and the description of the qualification;
A seventh to eighth evaluation index for evaluating whether or not version information (DCM version) for the DCM contents is included, the seventh to eighth evaluation indexes being scored according to the evaluation criteria for the existence of the version information and its actual meaning;
An index for evaluating the accessibility of the metadata, wherein the score is assigned according to an evaluation criterion for whether the metadata is publicly accessible in at least one of the home page or the DCM repository;
As an indicator for evaluating the extent of the metadata, the metadata element (s), such as development date, expiration date, last update date, approval authority, or clinical keyword, are generated and evaluated based on evaluation criteria 7-5 Evaluation index given and evaluated;
An indicator for assessing whether DCM provides a public access to the results of the review, the outcome of the endeavor, or a combination of these, as an indication of whether the outcome is accessible and whether the endorsement date, content for the endorsement Or an approval expiration date; and (7) a rating indicator for evaluating whether or not there is information accessible from among the approval expiration dates by assigning a score according to the evaluation criteria; Or a combination thereof;
The eighth evaluation item may include:
An indicator for assessing the public accessibility of the DCM maintenance agency; an 8-1 evaluation index for rating the information access based on evaluation criteria;
An indicator for evaluating whether or not a user's feedback mechanism exists for the DCM, and evaluating whether the feedback is provided through the DCM homepage or the repository; Or a combination thereof. ≪ Desc / Clms Page number 19 > A method for developing a detailed clinical model quality evaluation tool through a device for developing a quality evaluation tool of a detailed clinical model,
A first step in which the quality evaluation tool development apparatus of the detailed clinical model receives a predefined evaluation item and its evaluation index;
A second step of providing the evaluation items and the evaluation indexes to a plurality of evaluators and receiving the validity evaluation results of the evaluation items and the evaluation indexes from the plurality of evaluators;
A third step of providing the evaluation items and the evaluation indexes to a plurality of evaluators and receiving the reliability evaluation results of the evaluation items and the evaluation indexes from the plurality of evaluators;
When the quality evaluation tool development apparatus of the detailed clinical model does not match the evaluation results between the evaluators on the basis of the result of the validity evaluation inputted, the improved evaluation items, the evaluation indexes, or a combination thereof are input again, A fourth step of determining a final evaluation item and an evaluation index when the evaluation results between the evaluators are identical, by repeatedly performing a validity evaluation on the evaluation items, the evaluation indexes, or a combination thereof; And
The quality evaluation tool development apparatus of the detailed clinical model stores the information on the finally determined evaluation items and the evaluation indexes in the database. The reliability evaluation is performed by calculating a kappa coefficient indicating the degree of agreement between the evaluators, And the evaluation information is provided to the evaluator so that the evaluator judges that the evaluation index is not in agreement with the " good agreement level ", and below the evaluation result is inconsistent. The method of claim 11,
Wherein the information is converted into an evaluation index measurable in an abstract existing quality requirement and item in order to specify a predefined evaluation index at a measurable level in the first step,
The conversion should be based on the detailed clinical criteria, which include both quality requirements and items related to clinical use as criteria for inclusion and exclusion of existing quality requirements and items, and excluding quality requirements and items not related to clinical use How to Develop a Model Quality Assessment Tool. The method of claim 11,
After establishing two expert groups (the first expert group and the second expert group), the first expert group performs the surface validity evaluation and the second expert group performs the content validity evaluation using the Delphi technique ;
The content validity was evaluated by using the open questionnaire (1) evaluation index, (2) descriptive description about the evaluation index, (3) appropriateness of the evaluation index, (4) possibility of improving the quality of the evaluation index, (6) delphi 2 round to derive an agreement between the evaluation indexes not agreed upon in the Delphi 1 round, and How to develop tools. 14. The method of claim 13,
When the agreement standard is 70% or more,
The percentage of consensus on each assessment index is the sum of the percentages of the panels that responded with a specific response scale (consent, strong consent) of more than 70%, and the percentage of the panel that responded with different response measures (strong asynchronous, asynchronous) Support the determination that an agreement has been reached if the sum exceeds 70% of the total panel;
If there is an unsettled evaluation indicator as a result of the Delphi 2 round,
A method for developing a detailed clinical model quality assessment tool characterized in that concept mapping is performed between the unadjacent evaluation indexes and the evaluation indexes agreed upon between the expert panels.

Images