CA3071197A1 - Data quality categorization and utilization system, device, method, and computer-readable medium - Google Patents

Abstract

The present invention is directed to a system, method and/or computer readable medium for classifying and/or applying the quality of data elements received from a source to a third party application using the data elements. The system, method and/or computer readable medium include a QC engine analyzer to receive data sets, generate a weighted quality factor for the data elements making up the data set, and apply a quality algorithm to the data elements to generate QC stampes. Also included in the system, method and/or computer readable medium is a data engine analyzer associated with the third party application to receive the QC stamps and generate a result based on the associated data sets with reference to the QC engine analyzer, apply the quality algorithm to each of the QC stamps to generate a QC data certificate that is associated with the third party application result. The present system, method and/or computer readable medium increases the quality of the third party application results by reducing the contribute of low quality data elements.

Description

DATA QUALITY CATEGORIZATION AND UTILIZATION SYSTEM, DEVICE, METHOD, AND COMPUTER-READABLE MEDIUM
RELATED APPLICATIONS
[0001] The present application claims the benefit of the earlier filed United States Patent Provisional Application No. 62/367,035 filed on July 26, 2016.
FIELD OF THE INVENTION

[0002] The present invention relates generally to a data quality system, device, method, computer-readable medium, and/or cooperating environment, and more particularly to a data quality categorization and utilization system, device, method computer-readable medium, and/or cooperating environment for use in association with data analysis, data mining and conflict resolutions in a medical context.
BACKGROUND OF THE INVENTION

[0003] As the connectivity between devices increases due to popularity and/or infrastructure improvements (e.g., the "Internet of Things"), the number and variety of data sources also increases. This is especially apparent in the healthcare industry, where there is an ever growing set of data sources including, but not limited to:
= Patient health record (patient visit history, patient blood work results);
= Medical facility records (master facility list in Kenya, aggregate facility statistics from the World Health Organization ("WHO"));
= Health worker databases (e.g., hospital staff such as St. Michael's Hospital in Toronto, Canada; Ethiopia Health Extension Workers, etc.);

= Various application specific technology systems (e.g., disease contact tracing, Malaria rapid testing);
= Connected devices, such as environmental data (Global Positioning Systems, temperature, pressure, velocity) and patient data (body temperature, blood pressure, diagnostic results);
= Regional healthcare status (WHO reports on pandemics, Centers for Disease Control report on disease risk);
= Non-connected devices (classical tools, such as thermometers, stethoscopes, etc.);
= Manual (i.e., manual data entry);
= Operating system used to collect the information;
= Online maintenance service subscribed for the devices; and = Wearable devices (e.g., smart watches).

[0004] As the foregoing non-comprehensive list demonstrates, the number of data sources continues to increase and in many cases provides similar or related data points (e.g., body temperature). These data points are similar in that they convey the same information, such as, body temperature from a connected thermometer and body temperature from a classical digital thermometer. These data points are also related in that they can be used to infer similar information, such as inferring that a patient may have malaria due to patient health data collected manually versus a rapid diagnostic test using a connected device.

[0005] Currently, there are many models of patient care and clinical decision making that are based on collected data as described in the background. Over time, as the amount of data collected continues to grow (e.g., due to an increasing number of collection sources), new

6 models of patient care and clinical decision making may be required to leverage that data for improved patient care.
[0006] Since it appears that the outcomes of clinical decision making models are closely related to the quality of the data that is being utilized in the model, it may be important to determine and apply the quality of the data to such models.

[0007] The association of the collected data and the quality of the data will provide the data users a level of confidence to provide accurate healthcare decision outcomes.

[0008] In the future the data analysis and data mining applications used by health organisations, hospitals and/or governments will be developed by various software providers, the data used by these applications will need to be formatted or at minimum verified in order to be use correctly by these applications.

[0009] As a result, there may be a need for, or it may be desirable to provide, a system, device, method and/or computer readable medium that provides for data quality categorization and utilization of the increasing volume of data to improve healthcare decision outcomes.
.. [0010] It may be an object of the present invention to obviate or mitigate one or more disadvantages and/or shortcomings associated with the prior art, to meet or provide for one or more needs and/or advantages, and/or to achieve one or more objects of the invention -- one or more of which may preferably be readily appreciable by and/or suggested to those skilled in the art in view of the teachings and/or disclosures hereof.
SUMMARY OF THE INVENTION
[0011] The present disclosure provides a device, method, computer-readable medium and system for categorizing and utilizing data quality factors in association with data from multiple sources. More specifically, embodiments of the present invention are directed to a device, method, computer-readable medium and system for categorizing and utilizing data quality in association with data analysis, data mining and conflict resolutions in a healthcare context, comprising: (a) a QC engine analyzer to automatically (i) receive one or more data sets, each data set comprising a plurality of data elements, (ii) generate a weighted quality factor for each of the data elements, (iii) apply a quality algorithm to each of the data elements to generate one or more QC stamps for, and associate the one or more QC stamp to, each of the one or more data sets; (b) a data engine analyzer associated with the third party application to receive the one or more data sets having the one or more QC
stamps and (i) generate a third party application result based on the one or more data sets having the one or more QC stamps and (ii) with reference to the QC engine analyzer, apply the quality algorithm to each of the one or more data sets having the one or more QC
stamps to generate a QC data certificate, and associate the QC data certificate with the third party application result; and (c) one or more databases to store the data elements; whereby the system is operative to increase the quality of the third party application results by reducing the contribution of low quality data elements to the third party application results.
[0012] In another embodiment of the present invention, there is provided a method for classifying and applying the quality of data elements received from a source to a third party application using the data element, wherein the method comprises: (a) providing a QC engine analyzer to automatically (i) receive one or more data sets, each data set comprising a plurality of data elements, (ii) generate a weighted quality factor for each of the data elements, (iii) apply a quality algorithm to each of the data elements to generate one or more QC stamps for, and associate the one or more QC stamp to, each of the one or more data sets;
(b) providing a data engine analyzer associated with the third party application to receive the one or more data sets having the one or more QC stamps and (i) generate a third party application result based on the one or more data sets having the one or more QC stamps and (ii) with reference to the QC engine analyzer, apply the quality algorithm to each of the one or more data sets having the one or more QC stamps to generate a QC data certificate, and associate the QC data certificate with the third party application result; and (c) providing one or more databases to store the data elements; whereby the method increases the quality of the third party application results by reducing the contribution of the low quality data elements to the third party application results.
[0013] In yet another embodiment of the present invention, there is provided a non-transitory computer readable medium encoded with executable instructions for classifying and applying the quality of data elements received from a source to a third party application using the data elements, comprising: (a) providing a QC engine analyzer to automatically (i) receive one or more data sets, each data set comprising a plurality of data elements, (ii) generate a weighted quality factor for each of the data elements, (iii) apply a quality algorithm to each of the data elements to generate one or more QC stamps for, and associate the one or more QC stamp to, each of the one or more data sets; (b) providing a data engine analyzer associated with the third party application to receive the one or more data sets having the one or more QC stamps and (i) generate a third party application result based on the one or more data sets having the one or more QC stamps and (ii) with reference to the QC engine analyzer, apply the quality algorithm to each of the one or more data sets having the one or more QC
stamps to generate a QC data certificate, and associate the QC data certificate with the third party application result; and (c) providing one or more databases to store the data elements.
[0014] In a preferred embodiment of the present invention, the above-noted system, method and/or computer readable medium further provides that the data elements comprise: a data element name; a data element type; a data element value; a data element unit;
a data element value type; a data element source; a data element source type; a data element calibration; a data element quality score; and/or a data element timestamp.
[0015] In yet a further a preferred embodiment of the present invention, the above-noted system, method and/or computer readable medium further provides that the source comprises:
a manual entry of observed data; a manual entry of data; a trusted database; a controlled database; an uncontrolled database; a non-connected digital device; a non-connected non-digital device; a connected device; a connected in-vitro diagnostic device;
and/or a non-connected in-vitro diagnostic device.
[0016] In yet a further a preferred embodiment of the present invention, the above-noted system, method and/or computer readable medium further provides that the weighted quality factor comprises a predetermined range of values.
[0017] In yet a further a preferred embodiment of the present invention, the above-noted system, method and/or computer readable medium further provides that the weight quality factor for the data elements changes over time based on a feedback algorithm.
[0018] In yet a further a preferred embodiment of the present invention, the above-noted system, method and/or computer readable medium further provides that the data engine analyzer is embedded or non-embedded with the third party application.
[0019] In yet a further a preferred embodiment of the present invention, the above-noted system, method and/or computer readable medium further provides that the database comprises: a healthcare inputs database; a QC device database; a QC user database; a clinical data formats database; an other clinical data database; a QC stamp for dataset or data format database; and/or a healthcare inputs database verified.

[0020] Other advantages, features and characteristics of the present invention, as well as methods of operation and functions of the related elements of the system, method and computer readable medium, and the combination of steps, parts and economies of manufacture, will become more apparent upon consideration of the following detailed description and the appended claims with reference to the accompanying drawings, the latter of which are briefly described hereinbelow.
BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The novel features which are believed to be characteristic of the system, device and methods according to the present invention, as to their structure, organization, use, and method of operation, together with further objectives and advantages thereof, may be better understood from the following drawings in which presently preferred embodiments of the invention may now be illustrated by way of example. It is expressly understood, however, that the drawings are for the purpose of illustration and description only, and are not intended as a definition of the limits of the invention. In the accompanying drawings:
[0022] FIG. 1 is a schematic diagram of an embodiment of the present invention;
[0023] FIG. 2 is a schematic diagram of a further embodiment of the present invention; and [0024] FIG. 3 is a schematic diagram of yet a further embodiment of the present invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0025] The description that follows, and the embodiments described therein, may be provided by way of illustration of an example, or examples, of particular embodiments of the principles of the present invention. These examples are provided for the purposes of explanation, and not of limitation, of those principles and of the invention.
In the description, like parts are marked throughout the specification and the drawings with the same respective reference numerals. The drawings are not necessarily to scale and in some instances proportions may have been exaggerated in order to more clearly depict certain embodiments and features of the invention.
[0026] The present disclosure may be described herein with reference to system architecture, block diagrams and flowchart illustrations of methods, and computer program products according to various aspects of the present disclosure. It may be understood that each functional block of the block diagrams and the flowchart illustrations, and combinations of functional blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by computer program instructions.
[0027] These computer program instructions may be loaded onto a general purpose computer, special purpose computer (e.g., a server), or other programmable data processing apparatus to produce a machine, such that the instructions that execute on the computer or other programmable data processing apparatus create means for implementing the functions specified in the flowchart block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart block or blocks.
The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.
[0028] Accordingly, functional blocks of the block diagrams and flow diagram illustrations support combinations of means for performing the specified functions, combinations of steps for performing the specified functions, and program instruction means for performing the specified functions. It may also be understood that each functional block of the block diagrams and flowchart illustrations, and combinations of functional blocks in the block diagrams and flowchart illustrations, can be implemented by either special purpose hardware-based computer systems which perform the specified functions or steps, or suitable combinations of special purpose hardware and computer instructions.
[0029] The present disclosure may be now described in terms of an exemplary system in which the present disclosure, in various embodiments, would be implemented.
This may be for convenience only and may be not intended to limit the application of the present disclosure. It may be apparent to one skilled in the relevant art(s) how to implement the present disclosure in alternative embodiments.
[0030] In this disclosure, a number of terms and abbreviations may be used.
The following definitions and descriptions of such terms and abbreviations are provided in greater detail.
[0031] As used herein, a person skilled in the relevant art may generally understand the term "comprising" to generally mean the presence of the stated features, integers, steps, or components as referred to in the claims, but that it does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.
[0032] It should also be appreciated that the present invention can be implemented in numerous ways, including as a process, method, an apparatus, a system, a device, a method, or a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over a network (e.g., optical or electronic communication links). In this specification, these implementations, or any other form that the invention may take, may be referred to as processes. In general, the order of the steps of the disclosed processes may be altered within the scope of the invention.

[0033] Preferred embodiments of the present invention can be implemented in numerous configurations depending on implementation choices based upon the principles described herein. Various specific aspects are disclosed, which are illustrative embodiments not to be construed as limiting the scope of the disclosure. One aspect of the disclosure is a method, computer program product, apparatus, and system for categorizing and utilizing data quality of data obtained from multiple sources. Although the present specification describes components and functions implemented in the embodiments with reference to standards and protocols known to a person skilled in the art, the present disclosures as well as the embodiments of the present invention are not limited to any specific standard or protocol.
Each of the standards for non-mobile and mobile computing, including the Internet and other forms of computer network transmission (e.g., TCP/IP, UDP/IP, HTML, and HTTP) represent examples of the state of the art. Such standards are periodically superseded by faster or more efficient equivalents having essentially the same functions.
Accordingly, replacement standards and protocols having the same functions are considered equivalents.
[0034] As those of ordinary skill in the art would generally understand, the Internet is a global computer network which comprises a vast number of computers and computer networks which are interconnected through communication links. A person skilled in the relevant art may understand that an electronic communications network of the present invention, may include, but is not limited to, one or more of the following: a local area network, a wide area network, peer to peer communication (e.g., bluetooth, zigbee, z-wave, etc.), an intranet, or the Internet. The interconnected computers exchange information using various services, including, but not limited to, electronic mail, Gopher, web-services, application programming interface (API), File Transfer Protocol (FTP). This network allows a server computer system (a Web server) to send graphical Web pages of information to a remote client computer system. The remote client computer system can then display the Web pages via its web browser. Each Web page (or link) of the "world wide web"
("WWW") is uniquely identifiable by a Uniform Resource Locator (URL). To view a specific Web page, a client computer system specifies the URL for that Web page in a request (e.g., a HyperText Transfer Protocol ("HTTP") request). The request is forwarded to the Web server that supports the Web page. When the Web server receives the request, it sends the Web page to the client computer system. When the client computer system receives the Web page, it typically displays the Web page using a browser. A web browser or a browser is a special-purpose application program that effects the requesting of web pages and the displaying of web pages and the use of web-based applications. Commercially available browsers include Microsoft Internet Explorer and Firefox, Google Chrome among others. It may be understood that with embodiments of the present invention, any browser would be suitable.
[0035] A person skilled in the relevant art would generally understand that a reference to "Internet of Things (IoT)" or "IoT device" refers to networked or interconnected objects, typically, but not limited to, everyday objects (e.g., watch, speaker, refrigerator, television, etc.), more technically purposed objects (i.e., medical devices) and devices.
It is described as a self-configuring wireless network of sensors whose purpose would be to interconnect all such connected devices. The concept may be attributed to the former Auto-ID
Center, founded in 1999, based at the time at the Massachusetts Institute of Technology (MIT).
[0036] Web pages are typically defined using HTML. HTML provides a standard set of tags that define how a Web page is to be displayed. When a provider indicates to the browser to display a Web page, the browser sends a request to the server computer system to transfer to the client computer system an HTML document that defines the Web page. When the requested HTML document is received by the client computer system, the browser displays the Web page as defined by the HTML document. The HTML document contains various tags that control the displaying of text, graphics, controls, and other features. The HTML

document may contain URLs of other Web pages available on that server computer system or other server computer systems.
[0037] A person skilled in the relevant art may generally understand a web-based application refers to any program that is accessed over a network connection using HTTP, rather than existing within a device's memory. Web-based applications often run inside a web browser or web portal. Web-based applications also may be client-based, where a portion of the program is downloaded to a user's desktop, but processing is done over the Internet on an external server. Web-based applications may also be dedicated programs installed on an internet-ready device, such as a smart phone or tablet. A person skilled in the relevant art may understand that a web site may also act as a web portal. A web portal may be a web site that provides a variety of services to users via a collection of web sites or web based applications. A portal is most often one specially designed site or application that brings information together from diverse sources in a uniform way. Usually, each information source gets its dedicated area on the page for displaying information (a portlet); often, the user can configure which ones to display. Portals typically provide an opportunity for users to input information into a system. Variants of portals include "dashboards".
The extent to which content is displayed in a "uniform way" may depend on the intended user and the intended purpose, as well as the diversity of the content. Very often design emphasis is on a certain "metaphor" for configuring and customizing the presentation of the content and the chosen implementation framework and/or code libraries. In addition, the role of the user in an organization may determine which content can be added to the portal or deleted from the portal configuration.
[0038] It may be generally understood by a person skilled in the relevant art that the term "mobile device" or "portable device" refers to any portable electronic device that can be used to access a computer network such as, for example, the internet. Typically a portable electronic device comprises a display screen, at least one input/output device, a processor, memory, a power module and a tactile man-machine interface as well as other components (e.g., camera, speaker, microphone, etc.) that are common to portable electronic devices individuals or members carry with them on a daily basis. Examples of portable devices suitable for use with the present invention include, but are not limited to, smart phones, cell phones, wireless data/email devices, tablets, PDAs and MP3 players, test devices, etc.
[0039] It may be generally understood by a person skilled in the relevant art that the term "network ready device" or "internet ready device" refers to devices that are capable of connecting to and accessing a computer network, such as, for example, the Internet, including but not limited to an IoT device. A network ready device may assess the computer network through well-known methods, including, for example, a web-browser. Examples of internet-ready devices include, but are not limited to, mobile devices (including smart-phones, tablets, PDAs, etc.), gaming consoles, and smart-TVs. It may be understood by a person skilled in the relevant art that embodiment of the present invention may be expanded to include applications for use on a network ready device (e.g. cellphone). In a preferred embodiment, the network ready device version of the applicable software may have a similar look and feel as a browser version but that may be optimized to the device. It may be understood that other "smart" devices (devices that are capable of connecting to and accessing a computer network, such as, for example, the interne such as medical or test devices, including but not limited to smart blood pressure monitors, smart glucometers, IoT devices, etc.
[0040] It may be further generally understood by a person skilled in the relevant art that the term "downloading" refers to receiving datum or data to a local system (e.g.
mobile device) from a remote system (e.g. a client) or to initiate such a datum or data transfer. Examples of a remote systems or clients from which a download might be performed include, but are not limited to, web servers, FTP servers, email servers, or other similar systems.
A download can mean either any file that may be offered for downloading or that has been downloaded, or the process of receiving such a file. A person skilled in the relevant art may understand the inverse operation, namely sending of data from a local system (e.g. mobile device) to a remote system (e.g. a database) may be referred to as "uploading". The data and/or information used according to the present invention may be updated constantly, hourly, daily, weekly, monthly, yearly, etc. depending on the type of data and/or the level of importance inherent in, and/or assigned to, each type of data. Some of the data may preferably be downloaded from the Internet, by satellite networks or other wired or wireless networks.
[0041] Elements of the present invention may be implemented with computer systems which are well known in the art. Generally speaking, computers include a central processor, system memory, and a system bus that couples various system components including the system memory to the central processor. A system bus may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The structure of a system memory may be well known to those skilled in the art and may include a basic input/output system ("BIOS") stored in a read only memory ("ROM") and one or more program modules such as operating systems, application programs and program data stored in random access memory ("RAM"). Computers may also include a variety of interface units and drives for reading and writing data.
A user of the system can interact with the computer using a variety of input devices, all of which are known to a person skilled in the relevant art.
[0042] One skilled in the relevant art would appreciate that the device connections mentioned herein are for illustration purposes only and that any number of possible configurations and selection of peripheral devices could be coupled to the computer system.
[0043] Computers can operate in a networked environment using logical connections to one or more remote computers or other devices, such as a server, a router, a network personal computer, a peer device or other common network node, a wireless telephone or wireless personal digital assistant. The computer of the present invention may include a network interface that couples the system bus to a local area network ("LAN").
Networking environments are commonplace in offices, enterprise-wide computer networks and home computer systems. A wide area network ("WAN"), such as the Internet, can also be accessed by the computer or mobile device.
[0044] It may be appreciated that the type of connections contemplated herein are exemplary and other ways of establishing a communications link between computers may be used in accordance with the present invention, including, for example, mobile devices and networks.
The existence of any of various well-known protocols, such as TCP/IP, Frame Relay, Ethernet, FTP, HTTP and the like, may be presumed, and computer can be operated in a client-server configuration to permit a user to retrieve and send data to and from a web-based server. Furthermore, any of various conventional web browsers can be used to display and manipulate data in association with a web based application.
[0045] The operation of the network ready device (i.e., a mobile device) may be controlled by a variety of different program modules, engines, etc. Examples of program modules are routines, algorithms, programs, objects, components, data structures, etc.
that perform particular tasks or implement particular abstract data types. It may be understood that the present invention may also be practiced with other computer system configurations, including multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCS, personal computers, minicomputers, mainframe computers, and the like.
Furthermore, the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0046] Embodiments of the present invention can be implemented by a software program for processing data through a computer system. It may be understood by a person skilled in the relevant art that the computer system can be a personal computer, mobile device, notebook computer, server computer, mainframe, networked computer (e.g., router), workstation, and the like. In one embodiment, the computer system includes a processor coupled to a bus and memory storage coupled to the bus. The memory storage can be volatile or non-volatile (i.e.
transitory or non-transitory) and can include removable storage media. The computer can also include a display, provision for data input and output, etc. as may be understood by a person skilled in the relevant art.
[0047] Some portion of the detailed descriptions that follow are presented in terms of procedures, steps, logic block, processing, and other symbolic representations of operations on data bits that can be performed on computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A
procedure, computer executed step, logic block, process, etc. is here, and generally, conceived to be a self-consistent sequence of operations or instructions leading to a desired result. The operations are those requiring physical manipulations of physical quantities.
Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers or the like.
[0048] It may be generally understood that in establishing a user interface, a task bar may be preferably positioned at the top of a screen to provide a user interface.
Preferably, a textual representation of a task's name is presented in this user interface, preferably as a button, and the task names may be shortened as necessary if display space of the button is constrained.

The labelled button having the task's name preferably operate as a type of hyperlink, whereby the user/viewer can immediately switch to the activity, view, etc. of each of the tasks by selecting the button containing the applicable name from the task bar. In other words, the user or viewer is redirected by the application to the function represented by the task button by selecting the labelled hyperlink. Preferably, the task entry associated with the currently-displayed work unit view may be shown in a different graphical representation (e.g., using a different color, font, or highlighting). In preferred embodiments, there may be provided a display having a selectable "X" in the task bar entry for each task: if the user clicks on the "X", then its associated task may be ended and the view of its work unit may be removed. A user interface may be web-based, application based, or a combination.
[0049] In accordance with a preferred aspect of the present invention, a person skilled in the relevant art would generally understand the term "application" or "application software" to refer to a program or group of programs designed for end users. While there are system software, typically but not limited to, lower level programs (e.g., interact with computers at a basic level), application software resides above system software and may include, but is not limited to database programs, word processors, spreadsheets, etc. Application software may be grouped along with system software or published alone. Application software may simply be referred to as an "application".
[0050] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present invention, discussions utilizing terms such as "receiving", "sending", "creating", "providing", "communicating" or the like refer to the actions and processes of a computer system, or similar electronic computing device, including an embedded system, that manipulates and transfers data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. As used herein, reference to the "transmission", "processing", "interpretation" or the like of data associated with a cloud may refer to advancing through logic contained in the guideline. This may be accomplished, among other methods, by running on a processor one or more computer programs representative of the careflows, workflow, clinical guidelines, etc.
[0051] According to the invention, one or more data quality classification and utilization systems, platforms, devices, methods, computer-readable media, and/or cooperating environments may be disclosed.
[0052] The invention is contemplated for use in association with one or more cooperating environments, to afford increased functionality and/or advantageous utilities in association with same. The invention, however, is not so limited.
[0053] Certain novel features which are believed to be characteristic of the data quality classification and utilization system, platform, device, method, computer readable medium, and/or certain features of the engine, platform, device, system, method, computer readable medium which are novel in conjunction with the cooperating environment, according to the present invention, as to their organization, use, and/or method of operation, together with further objectives and/or advantages thereof, may be better understood from the accompanying disclosure in which presently preferred embodiments of the invention are disclosed by way of example. It is expressly understood, however, that the accompanying disclosure is for the purpose of illustration and/or description only, and is not intended as a definition of the limits of the invention.

[0054] Naturally, in view of the teachings and disclosures herein, persons having ordinary skill in the art may appreciate that alternate designs and/or embodiments of the invention may be possible (e.g., with substitution of one or more steps, algorithms, processes, features, structures, parts, components, modules, utilities, etc. for others, with alternate relations and/or configurations of steps, algorithms, processes, features, structures, parts, components, modules, utilities, etc).
[0055] Although some of the steps, algorithms, processes, features, structures, parts, components, modules, utilities, relations, configurations, etc. according to the invention are not specifically referenced in association with one another, they may be used, and/or adapted for use, in association therewith.
[0056] One or more of the disclosed steps, algorithms, processes, features, structures, parts, components, modules, utilities, relations, configurations, and the like may be implemented in and/or by the invention, on their own, and/or without reference, regard or likewise implementation of one or more of the other disclosed steps, algorithms, processes, features, structures, parts, components, modules, utilities, relations, configurations, and the like, in various permutations and combinations, as may be readily apparent to those skilled in the art, without departing from the pith, marrow, and spirit of the disclosed invention.
[0057] The present invention is directed to two issues in the organization of healthcare data management: (i) the categorization of collected healthcare information and data used in the clinical decision making process into levels of confidence; and (ii) formatting healthcare data sets based on the requirements of a healthcare application and the expected outcome.
[0058] Preferably, the present invention is adapted to provide a user with the option of choosing one or more categories of data to be included for use with a clinical application and present a level of confidence of the result provided by the clinical application based on the levels of confidence associated with each category of data (or data set).
[0059] Metadata Model of Data [0060] Persons skilled in the art may understand that metadata modeling is a type of metamodeling applicable to software engineering and/or systems engineering for the analysis, construction and development of the frames, rules, constraints, models and theories applicable and useful for modeling a predefined class of problems.
[0061] In a preferable embodiment of the invention, a metadata model may be established by a user for all of the collected information data elements including, but not limited to, the following elements (using body temperature as an example):
= a data element name (e.g., the body temperature of a patient);
= a data element type (e.g., patient health: body temperature);
= a data element value (e.g., 37.5);
= a data element units (e.g., degrees Celsius);
= a data element value type (e.g., numeric);
= a data element source (e.g., Laboratory No. 1 thermometer);
= a data element source type (e.g., connected device);
= a data element calibration (e.g., calibrated device [Y/N]);
= a data element quality score; and/or = a data element timestamp.

[0062] A data element quality score is preferably automatically calculated by a server, computer and/or mobile device (local and/or remote to the user) based on the foregoing data elements.
[0063] Data element name [0064] The data element name is the name of the data being collected. This is preferably an unrestricted name to allow each application or user of the metadata model to use a desired name.
[0065] Data element type [0066] The system preferably comprises a set of known data element types. In the healthcare context, data element types include, but are not limited to, the types set out in Table 1. The system is also preferably adapted to include an "other" data element type to allow a user to collect a desired data element type that may not be included in Table 1. This "other" data element type is preferably referenced by a code (string) to provide naming flexibility.
[0067] As shown in Table 1, the data element type is preferably restricted to the set of known data element type codes and the code for "other".
Table 1 - Sample list of Element Types Expected Expected Expected Element Type Code Description Element Value Element Value Element Units Type Patient Health: Ox0001 The body 30 < value < 50 Celsius Numeric Body temperature of the Temperature patient Patient Ox0101 The age of the 0 < value < 150 Years Numeric Demographics: patient Age Expected Expected Expected Element Type Code Description Element Value Element Value Element Units Type Patient 0x0102 The sex of the 0x001 - Male n/a String Demographics: patient 0x002- Female Sex 0x003 - Other Patient Health: 0x0004 Diastolic Blood 30 < value < mm Hg Numeric Diastolic Pressure reading 120 Blood Pressure for patient Other 0x9999 Generic value n/a n/a n/a .. [0068] In preferable embodiments, the data element type also includes expected ranges for the following categories, as exemplified in Table 1, including:
= "Data Element Value";
= "Data Element Units"; and = "Data Element Value Type".
.. [0069] Data element value, units and type [0070] The "value", "units" and "value type" categories preferably provide the expected value, units and value type for the respective data element type. The "value"
and "units" are preferably both stored in text format allowing them to be unrestricted if desired. In certain embodiments, the "units" may be restricted if the implementer or user of the system desires.
.. The "value" data element type is preferably based on a set of predetermined accepted types as defined in the system, including:
= Numeric;
= String;

= Comma-separated values Numeric ("CSV-Numeric");
= Comma-separated values String ("CSV-String");
= Extensible Markup Language ("XML"); and/or = JavaScript Object Notation ("JSON").
[0071] Data Element Source [0072] Preferably, the data element source is a name associated with the source of the data.
This field is preferably an unrestricted string field and can be any value.
[0073] Data Element Source Type [0074] The data element source type preferably comprises the value of a code indicating the type of source that the data was obtained from. The system may comprise a known set of source types as outlined in Table 2.
Table 2 - Data element source types Source Types Descriptions Direct entry into the system of data observed by the person Manual Entry of Observed Data entering into the system Manual Entry of data Entry of data that was not observed by operator A well governed and maintained data source. [Note that this is Trusted Database not based on brand but on trust is governing organization]
A controlled data source whose governance and maintenance is Controlled Database not well known. [Example, a master facility list for Kenya]
Uncontrolled Database A public data source that allows for community data collection Non-connected Digital Device Devices that collect data digitally but does not share the data Source Types Descriptions electronically. The operator would then directly enter the data from the device.
Devices that provide non-digital reading and do not share the Non-connected Non-Digital data electronically. The operator would then read the data from Device the device and enter into the system.
Device that collects data and can share it electronically (either Connected Device automatically or with user intervention).
The same as the connected device type, except the device is a registered IVD device.
Connected IVD Device IVD Devices have a higher quality burden so they will be tracked separately The same as Non-connected Device except that the device is a registered IVD device.
Non-connected IVD Device IVD Devices have a higher quality burden so they will be tracked separately Other Another source of data.
[0075] Data Element Calibration [0076] The data element calibration is preferably associated to a device to provide an indication of whether the device is calibrated or not. This calibration information may be included in the Quality score or may be provided as a standalone value. For example, a calibrated device may contribute to a higher quality score and an uncalibrated device (or a device that has not been calibrated for more than the recommended time) may contribute to a lower quality score.
[0077] Data Element Timestamp [0078] The data element timestamp is preferably associated with the collection of the data element value. For example, a timestamp will be associated by a server, computer and/or mobile device (local and/or remote to the user) with a data element value to indicate the date and time at which the data was collected.
[0079] Data Element Quality Score [0080] The data element quality score is preferably determined by the system (i.e., a server, computer and/or mobile device (local and/or remote to the user)) and described below. This data element quality score is assigned to the data element by the system taking into consideration one or more of the other data element values.
[0081] Quality Classification [0082] In a preferred embodiment of the invention, each data element comprises a quality score. The quality score is preferably based on a quality algorithm that utilizes the other data quality element attributes and is executed by a server, computer and/or mobile device (local and/or remote to the user). One example of the quality algorithm is the Simple Additive Weighting ("SAW") method for multi-attribute scoring which may consist of assigning to each alternative a sum of values, each one associated to the corresponding evaluation criterion, and weighted according to the relative importance of the corresponding criterion.
The present system preferably utilizes a similar method or a derivative of the SAW method, which may include for example, the weighted product method and/or the technique for order preference by similarity to ideal solution.
[0083] In the present invention, each data source type provided (see, for example, set out in Table 2) is preferably automatically assigned a quality factor by the server, computer and/or mobile device (local and/or remote to the user). The weight of the quality factor contributes to the quality of the data element obtained from all sources of that type.
This factor is initially set at a specific level by an administrator of the system (e.g., based on a quality assessment of the sources in the target market). Advantageously, using feedback on the reliability of the data sources over time, the quality factor of a given data source may preferably be automatically adjusted by the server, computer and/or mobile device (local and/or remote to the user) to more closely reflect the actual and/or current quality (and/or reliability) of the data.
[0084] Additionally, each named data source that is added to the system is preferably assigned a predetermined weighted quality factor by the user or administrator.
The initial quality factor of each added named source is preferably based on the type of source.
Advantageously, using feedback on the reliability of the added named data source over time, the system (i.e., the server, computer and/or mobile device (local and/or remote to the user)) may preferably automatically adjust the quality factor associated with the named data source.
Persons skilled in the art will understand that the weighted quality factor may be any range of values that provides a relative weight for a data source compared to other data sources (e.g., zero to one, zero to one-hundred, etc.) [0085] This automatic adjustment of quality factors allows the system to have a more accurate indication of the quality of all of its sources of data.
[0086] The quality score can be represented by a QC stamp. For example, in preferable embodiments, a data set may be associated with a QC stamp. The associated QC
stamp may preferably be recognized by a third party database or application or can be used by a data user as a parameter to validate or in a clinical algorithm (as a data set).
[0087] In preferable embodiments, the two potential outcomes from the QC data monitoring applications are:
(i) a QC stamp, which is preferably directly associated with a set of clinical data; and/or (ii) a QC data certificate, which is preferably directly associated with a result of a clinical application.
[0088] Utilizing Quality of Data [0089] As each data element is preferably associated with a quality, users of the data may preferably make decisions based on the data element values and quality.
[0090] (i) Conflict resolution [0091] In preferable embodiments, when a situation occurs where the same data point has multiple values from different sources, the system is adapted to provide users with an option to determine the data element with greater quality and/or reliability. Options may preferably be presented to the user using the graphical user interface of a mobile device or computer local to the user.
[0092] For example, if a patient body temperature is incorrectly entered during a triage process shortly (e.g., ten minutes) before a connected thermometer measures the patient's body temperature during a physician's examination, the system will preferably identify a conflict with respect to body temperature and determine that the temperature measured by the connected thermometer is more likely to be the correct value on the basis that it is obtained from a higher quality source and due to the short time proximity to the previous lower quality measurement. The body temperature from the triage process may be assigned a low weighted quality factor (e.g., zero if it is not to be considered).
[0093] (ii) Quality based decision systems [0094] In certain embodiments, data-based decision systems may be designed to be quality-sensitive, to provide users with an option to apply high quality and/or low quality decisions based on the context / situation.

[0095] For example, a decision-based system can determine whether a patient referral should be conducted based preferably on system rules implemented by the user or administrator and executed by the server, computer and/or mobile device (local and/or remote to the user). In certain scenarios, when the context is a possible cancer diagnosis, the system can automatically decide to utilize only the high quality data to ensure a higher quality decision point (e.g., the low quality data may be assigned a weight factor of zero).
The system, however, can decide to also apply lower quality data for a less severe context, such as the flu virus.
[0096] (iii) Identification of degrading systems [0097] As systems degrade, the quality of those systems may degrade as well.
Regardless of whether the degradation is technical in nature or whether it is the degradation of the processes and governance of the system, the degradation is typically associated with a reduction in quality. The present system preferably contains a feedback mechanism (e.g., a feedback algorithm) that updates data source quality factors over time. The feedback algorithm is preferably executed by the server, computer and/or mobile device (local and/or remote to the user). The trending of the quality factor of sources over time (e.g., a relative comparison of the quality factor of similar or different sources to identify a quality factor outlier ¨ that is, a data source that appears to have a significantly lower quality than similar or different data sources) can provide an indication of system degradation to generate an opportunity for remedial action to be taken (e.g., assign a lower weight factor).
[0098] Schema 1 : System for classification and certified medical data set [0099] As shown in FIG. 1, a QC data monitoring application 100, via a QC
engine analyzer 110 comprising a QC engine algorithm (not shown) executed by the server, computer and/or mobile device (local and/or remote to the user), preferably generates and/or associates a QC

stamp based on various parameters linked or attached to data sets 130. The QC
engine analyzer 110 is preferably in communication with a healthcare database input 120, a QC
device database 122, a QC user database 124, a clinical data formats database 126, and/or an others clinical database 128. The QC stamps are preferably subsequently used in various clinical applications in association with the QC data monitoring application 100 to certify the clinical result and/or to select a list of data sets which may be used by a third party clinical application (not shown).
[00100] Schema 2: Utilization of data quality for the purposes of data analysis and conflict resolutions in a medical context with embedded QC monitoring function [00101]As shown in FIG. 2, preferable embodiments of the invention is adapted to allow a clinical data application provider to embed a QC application into the provider's own application.
[00102] A clinical decision application 200 uses various medical data sets, including the , healthcare database inputs 120. The data sets received from the QC stamp for dataset or data format database 130 preferably comprises a QC stamp. The QC data monitoring application 100 is preferably integrated as an application programming interface ("API") or a function into the clinical decision application 200.
[00103] When the clinical decision application 200 retrieves a data set, for example from the QC stamp for dataset or data format database 130, the QC data monitoring application 100 processes the QC stamp attached to the data set. The data engine analyser 210, comprising a data engine algorithm (not shown) executed by the server, computer and/or mobile device (local and/or remote to the user), analyzes the data set taking into consideration the QC stamp and generates the QC monitored healthcare application results 220 (i.e., a final result and a global QC score) which becomes a Result "QC monitored".

[00104] Schema 3: Utilization of data quality for the purposes of data analysis and conflict resolutions in a medical context with external QC monitoring and data set validation [00105] In preferable embodiments, as shown in FIG. 3, the QC data monitoring application 100 may be used to provide the right clinical data format to the clinical application 200 and to generate a QC stamp at the end of the data analysis in order to certify the quality of the information provided. In FIG. 3, the user of the clinical application 200 may not be required to include the QC data monitoring application 100 in the user's own application; the data engine analyser 210 comprising the data engine algorithm (not shown) executed by the server, computer and/or mobile device (local and/or remote to the user) may provide QC
monitoring function that is non-embedded with the clinical application 200.
The clinical decision application 200 may request the use of a Healthcare database with QC
stamps (i.e., "healthcare database inputs verified 320"), the QC data monitoring application 100 preferably reviews and analyzes the data, the data format and the type of clinical application (i.e., the "data set list data format 330") and generates a QC data certificate 340, at the end of the clinical application processing the QC data certificate 340 will be associated with the final result 350.
[00106] As shown in FIG. 3, the QC data monitoring application 100 is used by a trusted third party to validate or certify the quality of the clinical data used.
[00107] Encryption [00108] Preferably, the foregoing data is encrypted or de-encrypted (or decrypted) for secured transmission by the devices executing the application using an encryption algorithm which may include encryption methods known to persons skilled in the art. Following transmission, the encrypted data is preferably automatically de-encrypted by the processors of the devices executing the application and using the encryption algorithm.

[00109] Databases [00110] A preferred embodiment of the present invention provides a system comprising data storage (e.g. databases 120, 122, 124, 126, 128, 130, 320 in FIGS. 1-3) that may be used to store all necessary data required for the operation of the system. A person skilled in the relevant art may understand that a "data store" refers to a repository for temporarily or persistently storing and managing collections of data which include not just repositories like databases (a series of bytes that may be managed by a database management system (DBMS)), but also simpler store types such as simple files, emails, etc. A
data store in accordance with the present invention may be one or more databases, co-located or distributed geographically. The data being stored may be in any format that may be applicable to the data itself, but may also be in a format that also encapsulates the data qua! ity.
[00111] As shown in FIGS. 1 to 3, various data stores or databases may interface with the system of the present invention, preferably including, without limitation, a healthcare inputs database 120, a QC device database 122, a QC user database 124, a clinical data formats database 126, an other clinical data database 128, a QC stamp for dataset or data format database 130, and a healthcare inputs database verified 320.
[00112] Although not shown, some additional databases may be associated with the present system for data quality classification and utilization, including: proprietary databases, epidemiologic databases, medical records databases, UN and major/international healthcare institution databases, healthcare and emergency infrastructure databases, education and economic databases, news databases, demographic databases, communication and military infrastructure databases, and weather, travel, topographic databases.

[00113] A clinical and healthcare database may preferably contain, among other things, diagnostic and medical data (clinical information), such as, for example, one or more of the following, which may or may not be related to medical events: (a) test results from diagnostic devices equipped with remote data transfer systems and/or global positioning or localization features; (b) information from UN databases and major healthcare international institutions;
and/or (c) scenarios and knowledge data.
[00114] A sociological database may preferably contain, among other things, sociological data (human information), such as, for example, one or more of the following: (a) population information from local and/or international demographic databases; (b) political and/or organization systems in the area and/or from international databases; (c) education and/or economic systems in the area and/or from international databases; and/or (d) information from news and/or newspapers, drawn from the Internet or elsewhere.
[00115] An infrastructure database may preferably contain, among other things infrastructure data or information, such as, for example, one or more of the following: (a) information concerning healthcare infrastructure; (b) information concerning communication infrastructures; and/or (c) information concerning emergency and/or military infrastructure;
all preferably drawn from local and/or international databases.
[00116] A geophysics database may preferably contain, among other things, geophysics data or information, such as, for example, one or more of the following: (a) weather and/or climatic information from local databases; and/or (b) topographic information from local and/or international databases.
[00117] As used herein, a person skilled in the relevant art may generally understand the term "cloud" to refer to a set of software services, a database, hardware to host the database and services.

[00118] The foregoing description has been presented for the purpose of illustration and maybe not intended to be exhaustive or to limit the invention to the precise form disclosed.
Other modifications, variations and alterations are possible in light of the above teaching and may be apparent to those skilled in the art, and may be used in the design and manufacture of other embodiments according to the present invention without departing from the spirit and scope of the invention. It may be intended the scope of the invention be limited not by this description but only by the claims forming a part of this application and/or any patent issuing herefrom.

Claims (20)

WHAT IS CLAIMED IS:

1. A system for classifying and applying the quality of data elements received from a source to a third party application using the data elements, wherein the system comprises:
(a) a QC engine analyzer to automatically (i) receive one or more data sets, each data set comprising a plurality of data elements, (ii) generate a weighted quality factor for each of the data elements, (iii) apply a quality algorithm to each of the data elements to generate one or more QC stamps for, and associate the one or more QC stamp to, each of the one or more data sets;
(b) a data engine analyzer associated with the third party application to receive the one or more data sets having the one or more QC stamps and (i) generate a third party application result based on the one or more data sets having the one or more QC stamps and (ii) with reference to the QC engine analyzer, apply the quality algorithm to each of the one or more data sets having the one or more QC stamps to generate a QC data certificate, and associate the QC data certificate with the third party application result; and (c) one or more databases to store the data elements;
whereby the system is operative to increase the quality of the third party application results by reducing the contribution of low quality data elements to the third party application results.

2. A system according to claim 1, wherein the data elements comprise: a data element name; a data element type; a data element value; a data element unit; a data element value type; a data element source; a data element source type; a data element calibration; a data element quality score; and/or a data element timestamp.

3. A system according to any one of claims 1-2, wherein the source comprises: a manual entry of observed data; a manual entry of data; a trusted database; a controlled database; an uncontrolled database; a non-connected digital device; a non-connected non-digital device; a connected device; a connected in-vitro diagnostic device; and/or a non-connected in-vitro diagnostic device.

4. A system according to any one of claims 1-3, wherein the weighted quality factor comprises a predetermined range of values.

5. A system according to claim 4, wherein the weight quality factor for the data elements changes over time based on a feedback algorithm.

6. A system according to any one of claims 1-5, wherein the data engine analyzer is embedded or non-embedded with the third party application.

7. A system according to any one of claims 1-6, wherein the database comprises: a healthcare inputs database; a QC device database; a QC user database; a clinical data formats database; an other clinical data database; a QC stamp for dataset or data format database;
and/or a healthcare inputs database verified.

8. A method for classifying and applying the quality of data elements received from a source to a third party application using the data element, wherein the method comprises:
(a) providing a QC engine analyzer to automatically (i) receive one or more data sets, each data set comprising a plurality of data elements , (ii) generate a weighted quality factor for each of the data elements, (iii) apply a quality algorithm to each of the data elements to generate one or more QC stamps for, and associate the one or more QC stamp to, each of the one or more data sets;
(b) providing a data engine analyzer associated with the third party application to receive the one or more data sets having the one or more QC stamps and (i) generate a third party application result based on the one or more data sets having the one or more QC stamps and (ii) with reference to the QC engine analyzer, apply the quality algorithm to each of the one or more data sets having the one or more QC stamps to generate a QC data certificate, and associate the QC data certificate with the third party application result; and (c) providing one or more databases to store the data elements;
whereby the method increases the quality of the third party application results by reducing the contribution of the low quality data elements to the third party application results.

9. The method of claim 8, wherein the data elements comprise: a data element name; a data element type; a data element value; a data element unit; a data element value type; a data element source; a data element source type; a data element calibration; a data element quality score; and/or a data element timestamp.

10. The method of any one of claims 8-9, wherein the source comprises: a manual entry of observed data; a manual entry of data; a trusted database; a controlled database; an uncontrolled database; a non-connected digital device; a non-connected non-digital device; a connected device; a connected in-vitro diagnostic device; and/or a non-connected in-vitro diagnostic device.

11. The method of any one of claims 8-10, wherein the weighted quality factor comprises a predetermined range of values.

12. The method of claim 11, wherein the weight quality factor for the data elements changes over time based on a feedback algorithm.

13. The method of any one of claims 8-12, wherein the data engine analyzer is embedded or non-embedded with the third party application.

14. The method of any one of claims 8-13, wherein the database comprises: a healthcare inputs database; a QC device database; a QC user database; a clinical data formats database;
an other clinical data database; a QC stamp for dataset or data format database; and/or a healthcare inputs database verified.

15. A non-transitory computer readable medium encoded with executable instructions for classifying and applying the quality of data elements received from a source to a third party application using the data elements, comprising:
(a) providing a QC engine analyzer to automatically (i) receive one or more data sets, each data set comprising a plurality of data elements, (ii) generate a weighted quality factor for each of the data elements, (iii) apply a quality algorithm to each of the data elements to generate one or more QC stamps for, and associate the one or more QC stamp to, each of the one or more data sets;
(b) providing a data engine analyzer associated with the third party application to receive the one or more data sets having the one or more QC stamps and (i) generate a third party application result based on the one or more data sets having the one or more QC stamps and (ii) with reference to the QC engine analyzer, apply the quality algorithm to each of the one or more data sets having the one or more QC stamps to generate a QC data certificate, and associate the QC data certificate with the third party application result; and (c) providing one or more databases to store the data elements.

16. The computer readable medium of claim 15, wherein the data elements comprise: a data element name; a data element type; a data element value; a data element unit; a data element value type; a data element source; a data element source type; a data element calibration; a data element quality score; and/or a data element timestamp.

17. The computer readable medium of any one of claims 15-16, wherein the source comprises: a manual entry of observed data; a manual entry of data; a trusted database; a controlled database; an uncontrolled database; a non-connected digital device;
a non-connected non-digital device; a connected device; a connected in-vitro diagnostic device;
and/or a non-connected in-vitro diagnostic device.

18. The computer readable medium of any one of claims 15-17, wherein the weighted quality factor comprises a predetermined range of values.

19. The computer readable medium of claim 18, wherein the weight quality factor for the data elements changes over time based on a feedback algorithm.

20. The computer readable medium of any one of claims 15-19, wherein the database comprises: a healthcare inputs database; a QC device database; a QC user database; a clinical data formats database; an other clinical data database; a QC stamp for dataset or data format database; and/or a healthcare inputs database verified.