JP7439125B2 - Decentralized privacy-preserving computing for protected data - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a cross-reference to U.S. Provisional Application No. 62/948,556, entitled "DISTRIBUTED PRIVACY-PRESERVING COMPUTING ON PROTECTED DATA," filed on December 16, 2019, and filed on March 26, 2019. 62/824,183, entitled ``FEDERATED MACHINE LEARNING TECHNIQUES FOR HIGHLY CURATED HEALTH-CARE DATA SETS,'' the entire contents of which are incorporated herein by reference for all purposes. Can be incorporated.

FIELD The present invention relates to privacy-preserving computing, and more particularly to techniques (e.g., systems, methods, etc.) for developing artificial intelligence applications and/or algorithms using distributed analysis on multiple sources of privacy-preserving harmonized data. , a computer program product containing code or instructions executable by one or more processors). The present invention is particularly useful for developing artificial intelligence algorithms for regulated medical applications using privacy-protected harmonized clinical and health data.

Background Modern computational paradigms, including cloud computing, data-parallel cluster computing, and high-performance computing, combine with a variety of widely available machine learning and deep learning algorithm architectures to properly optimize the underlying algorithms. The availability of sufficient data creates an environment in which a vast number of artificial intelligence (AI) applications can be developed to solve problems in almost any industry. It is now clear that access to data is a major barrier to the development of AI applications. In fact, many industries require the use of data from a variety of sources to create robust and generalizable AI. The challenge is that the owners of the data are typically unable or unwilling to share the data or do not allow the data to leave their control. This is understandable, as data is an asset, often containing sensitive private and/or personal data, and can be regulated in ways that make sharing difficult or impossible. These challenges are particularly difficult to overcome in the development of medical AI.

Approximately 30% of the world's stored data is in healthcare, which is driving the development and funding of AI algorithms. Insights created by AI and machine learning (“ML”) enhance clinical judgment, enable treatment accuracy, and improve the ability to draw connections (i.e., correlations) within complex data needed to create digital therapeutics. ) has the potential to learn. Medical AI is relevant to the fields of biotechnology, pharmaceuticals, medical information technology, analysis and genetic testing, and medical devices. Artificial intelligence approaches, including machine learning systems, can identify patterns within complex data. Generally, the higher the fidelity and diversity of data utilized to create algorithms and models, the more accurately and consistently the algorithms and models will perform across diverse environments and populations. These AI approaches therefore require access to diverse high-fidelity data to develop, optimize, and validate models. However, most AI algorithm developers lack sufficient fidelity and diversity to train, test, and validate their algorithms and models without breaking significant barriers to accessing the data. There are no medical data assets. Furthermore, even when AI algorithm developers have sufficient data assets, few perform third-party validation, resulting in essentially proof-of-concept studies that cannot be applied in production or clinical environments. You will end up with algorithms and models that are not possible solutions. Further development of AI models and algorithms for production or clinical use appears to be severely hampered by major obstacles: timely access to high fidelity, diversity, and privacy-preserving data.

Healthcare has regulatory, legal, and ethical requirements to maintain the privacy of patient information. Privacy objectives include protecting data from unauthorized access, providing transparency of use in accordance with individual privacy consent, and minimizing the use of personally identifiable data wherever possible. . Therefore, data privacy and protection is a barrier for AI, which requires timely access to high-fidelity, real-time, and diverse data. The opportunity in medical AI is to use privacy-preserving computing to eliminate the risk of exposing identifiable information. These considerations are relevant to AI in many industries where the sensitivity of the data (e.g., whether it contains trade secrets or private data about an individual) prevents its sharing outside the boundaries of the organization responsible for protecting the data. This applies to the development of Therefore, a need exists to establish privacy-preserving computing across multiple organizations to facilitate timely access to high-fidelity, diverse data.

brief overview
A system of one or more computers that causes the system to perform certain operations or actions by having software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform certain operations or actions during operation. Can be configured. One or more computer programs may be configured to perform certain operations or actions by containing instructions that, when executed by a data processing device, cause the device to perform certain operations or actions. One general aspect is, in a data processing system, receiving an algorithm and input data requirements associated with the algorithm, the input data requirements being used to optimize and/or validate data assets for operation with the algorithm. The method includes the step of receiving, including selection criteria. The method also includes identifying, by the data processing system, the data asset as available from the data host based on optimization and/or validation selection criteria for the data asset. The method also includes curating, by the data processing system, data assets within a data storage structure within an infrastructure of the data host. The method also includes preparing a data asset in a data storage structure for processing by the algorithm by the data processing system. The method also includes, by the data processing system, integrating the algorithm into a secure capsule computation framework, wherein the secure capsule computation framework integrates the algorithm into a data asset within a data storage structure. It also includes providing and integrating the information in a secure manner that preserves privacy. The method also includes operating the data asset through an algorithm by the data processing system. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method.

Implementations may include one or more of the following features. The algorithm and input data requirements are received from the algorithm developer, an entity distinct from the data host, and how the optimization and/or validation selection criteria define the characteristics, format, and requirements for the data assets to work with the algorithm. . The characteristics and requirements of the data assets include (i) the environment of the algorithm, (ii) the distribution of examples within the input data, (iii) the parameters and type of the device producing the input data, (iv) the variance versus bias, and (v) tasks implemented by algorithms, or (vi) methods defined based on any combination thereof. A method further comprising the step of hosting a data host by a data processing system, the hosting step comprising verifying that use of the data asset in the algorithm is compliant with data privacy requirements. The method in which preparing the data asset includes applying one or more transformations to the data asset, annotating the data asset, aligning the data asset, or a combination thereof. Running the data asset through the algorithm includes creating multiple instances of the model, splitting the data asset into a training dataset and one or more test datasets, and running the multiple instances of the model. 1. training on a training dataset; integrating results from training each of the plurality of instances of the model into a fully federated model; and operating one or more test datasets with the fully federated model; and computing performance of a fully federated model based on operation of one or more test datasets. Implementations of the described techniques may include hardware, methods or processes, or computer software on computer-accessible media.

Implementations may include one or more of the following features. A secure encapsulated computational framework is provisioned within a computational infrastructure configured to accept the cryptographic code necessary to operate the algorithm, and the process of provisioning the computational infrastructure is performed using a instantiating the capsule computation framework; placing cryptographic code inside the secure capsule computation framework by an algorithm developer; and, after the secure capsule computation framework has been instantiated, encrypting the decoding the code. Running the data asset through the algorithm includes splitting the data asset into one or more validation datasets, running the one or more validation datasets through the algorithm, and one or more validation data sets. and calculating performance of the algorithm based on the set of operations. identifying is performed using differential privacy for sharing information within the data asset by describing patterns of groups within the data asset while keeping private information about individuals within the data asset hidden; The curating step includes selecting a data storage structure from among a plurality of data storage structures, and provisioning the data storage structure within an infrastructure of the data host, and the selection of the data storage structure is based on a type of algorithm. , the type of data within the data asset, the system requirements of the data processing system, or a combination thereof. Implementations of the described techniques may include hardware, methods or processes, or computer software on computer-accessible media.

One general aspect is identifying multiple instances of an algorithm, where each instance of the algorithm is integrated into one or more secure capsule computation frameworks, and where each instance of the algorithm is integrated into one or more secure capsule computation frameworks. The framework transfers each instance of the algorithm to a training data asset within one or more data storage structures of one or more data hosts in a secure manner that preserves the privacy of the training data asset and each instance of the algorithm. The method includes the step of providing and identifying. The method also includes executing, by the data processing system, a federated training workflow on each instance of the algorithm, the federated training workflow taking the training data asset as input and using the parameters to characterize the training data asset. map to a target inference, compute a loss or error function, update parameters to learned parameters to minimize the loss or error function, and output one or more trained instances of the algorithm , also includes the steps of executing. The method also includes the step of integrating, by the data processing system, the learned parameters of each trained instance of the algorithm into a fully federated algorithm, the step of integrating comprising aggregating the learned parameters to aggregate the learned parameters. and updating the learned parameters of the fully federated algorithm with the aggregated parameters. The method also includes executing, by the data processing system, a test workflow with a fully federated algorithm, the test workflow taking the test data as input and using the updated learned parameters to It also includes the steps of finding patterns, outputting inferences, and executing them. The method also includes calculating, by the data processing system, the performance of the fully federated algorithm in providing the inference. The method also includes determining, by the data processing system, whether performance of the fully federated algorithm meets algorithm termination criteria. The method also includes the step of, by the data processing system, replacing each instance of the algorithm with a fully federated algorithm and rerunning the federated training workflow with each instance of the fully federated algorithm if the performance of the fully federated algorithm does not meet algorithm termination criteria. Also included. The method also includes providing, by the data processing system, the performance of the fully federated algorithm and the aggregated parameters to an algorithm developer of the algorithm if the fully federated algorithm's performance satisfies the algorithm termination criteria. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method.

Implementations may include one or more of the following features. Identifying a plurality of instances of the algorithm is a step of receiving, in a data processing system, an algorithm and input data requirements associated with the algorithm, wherein the input data requirements include optimizing and optimizing the data assets for operation with the algorithm. and/or a method comprising receiving, including verification selection criteria. The method may also include identifying, by the data processing system, the data asset as available from one or more data hosts based on optimization and/or validation selection criteria for the data asset. The method may also include curating, by the data processing system, data assets within a data storage structure within each data host's infrastructure of the one or more data hosts. The method may also include dividing at least a portion of the data assets into training data assets within a data storage structure within each data host's infrastructure of the one or more data hosts. The algorithm and input data requirements are received from an algorithm developer, an entity distinct from one or more data hosts, and the optimization and/or validation selection criteria are based on the characteristics, format, and How to define requirements. The method wherein the federated training workflow further includes encrypting the training gradients, and the step of integrating includes decoding the training gradients. Implementations of the described techniques may include hardware, methods or processes, or computer software on computer-accessible media.

Implementations may include one or more of the following features. If performance of the fully federated algorithm meets algorithm termination criteria, the method further comprises, by the data processing system, sending the aggregated parameters to each instance of the algorithm. The method also includes, by the data processing system, performing an update training workflow on each instance of the algorithm, the update training workflow updating the learned parameters with the aggregated parameters of one or more of the algorithms. It may also include performing, updating and outputting the trained instances. The method further comprises operating the remaining data assets with each instance of the algorithm by the data processing system. operating the data asset by each instance of the algorithm further partitioning at least a portion of the data asset into one or more validation datasets; and operating the one or more validation datasets by each instance of the algorithm. and calculating the performance of each instance of the algorithm based on behavior on one or more validation data sets. Implementations of the described techniques may include hardware, methods or processes, or computer software on computer-accessible media.

One general aspect includes a method that includes identifying, by a data processing system, a data asset available from a data host based on data asset selection criteria. The method also includes curating, by the data processing system, data assets in a data storage structure within an infrastructure of the data host. The method also includes the step of preparing a transformer prototype data set to be used by the data processing system as a guide for developing algorithms for data transformation, the transformer prototype data set being subjected to a harmonization process. It also includes the step of importing and preparing the key attributes of. The method also includes creating, at the data processing system, a first harmonized transformer set for transformation of the data asset based on a current format of data in the transformer prototype data set. The method also includes applying, by the data processing system, a first harmonization transformer set to the data asset to generate a transformed data asset. The method also includes preparing a harmonized prototype data set to be used by the data processing system as a guide for developing algorithms for data transformation, the harmonized prototype data set being a key to the harmonization process. It also includes the process of importing and preparing attributes. The method also includes creating, by the data processing system, a second set of harmonized transformers for transformation of the transformed data asset based on a current format of data in the harmonized prototype data set. The method also includes applying, by the data processing system, a second set of harmonization transformers to the transformed data asset to produce a harmonized data asset. The method also includes operating the harmonized data asset through an algorithm by the data processing system, the algorithm operating the harmonized data asset within the data storage structure. It also includes operating within a secure encapsulated computational framework that provides assets and algorithms in a secure manner that preserves their privacy. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method.

Implementations may include one or more of the following features. How the selection criteria are received from the algorithm developer, which is a different entity than the data host, and where the selection criteria define the characteristics, formats, and requirements for the data assets to work with the algorithm. The characteristics and requirements of the data assets include (i) the environment of the algorithm, (ii) the distribution of examples within the input data, (iii) the parameters and type of the device producing the input data, (iv) the variance versus bias, and (v) tasks implemented by algorithms, or (vi) methods defined based on any combination thereof. further comprising anonymizing the transformer prototype dataset and providing the anonymized transformer prototype dataset to an algorithm developer for the purpose of creating a first harmonized transformer set for transformation of the data asset. Method. A method in which applying a first set of harmonized transformers to a data asset is performed within a data structure. In a data processing system, annotating a transformed data asset according to a defined annotation protocol to produce an annotated dataset, the step of annotating the transformed data occurring within a data structure. , a second harmonization transformer set is applied to the annotated dataset to generate a harmonized data asset. The method by which applying a second harmonization transformer set to an annotated data asset occurs within a data structure. The method further comprises determining whether the first harmonizing transformer set, the annotation, and the second harmonizing transformer set are applied successfully and without violating data privacy requirements. Implementations of the described techniques may include hardware, methods or processes, or computer software on computer-accessible media.

One general aspect is the process of identifying an algorithm or model, where the algorithm or model is integrated into a secure capsule computation framework, and where the secure capsule computation framework integrates the algorithm or model with data in a data host. A method includes identifying, providing training data assets in a storage structure in a privacy-preserving, secure manner of the training data assets and algorithms or modes. The method also includes performing a federated training workflow on the algorithm or model by the data processing system, the federated training workflow taking a training data asset as input and using the parameters to target features of the training data asset. Also includes the steps of mapping the inference, computing a loss or error function, updating parameters to the learned parameters to minimize the loss or error function, and outputting and executing the trained algorithm or model. . The method also includes the step of integrating, by the data processing system, the learned parameters of the algorithm or model into a fully federated algorithm or model, the step of integrating comprising The method also includes a step of integrating, including obtaining the parameters and updating the learned parameters of the fully federated algorithm or model with the aggregated parameters. The method also includes executing, by the data processing system, a test workflow with a fully federated algorithm or model, the test workflow taking test data as input and using the updated learned parameters to It also includes the steps of finding patterns in the data, outputting inferences, and executing them. The method also includes calculating, by the data processing system, the performance of the fully federated algorithm in providing the inference. The method also includes determining, by the data processing system, whether performance of the fully federated algorithm or model satisfies algorithm termination criteria. The method also allows the data processing system to replace the algorithm or model with the fully federated algorithm or model and rerun the federated training workflow with the fully federated algorithm or model if the performance of the fully federated algorithm or model does not meet the algorithm termination criteria. It also includes the process of The method also includes providing, by the data processing system, the performance of the fully federated algorithm or model and the aggregated parameters to an algorithm developer of the algorithm or model if the performance of the fully federated algorithm or model satisfies algorithm termination criteria. include. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method.

Implementations may include one or more of the following features. identifying an algorithm includes receiving, in a data processing system, an algorithm and input data requirements associated with the algorithm, the input data requirements including validation selection criteria for data assets to operate on the algorithm; A method that involves a process. The method may also include identifying, by the data processing system, the data asset as available from the data host based on validation selection criteria for the data asset. The method may also include curating, by the data processing system, the data assets in a data storage structure within an infrastructure of the data host. The method may also include partitioning at least a portion of the data asset into a verification data asset within a data storage structure within the data host's infrastructure. A method further comprising the step of hosting a data host by a data processing system, the hosting step comprising verifying that use of the data asset in the algorithm is compliant with data privacy requirements. The method may also include completing governance and compliance requirements, including approval from an institutional review board for use of data assets from the data host for purposes of validating the algorithm. The method also includes selecting the data storage structure from among the plurality of data storage structures and provisioning the data storage structure within the infrastructure of the data host. may include being based on the type of algorithms within the algorithm, the type of data within the data asset, the system requirements of the data processing system, or a combination thereof. If performance of the algorithm satisfies verification criteria, the method further comprises maintaining the algorithm and the verification data asset by the data processing system in a secure manner that preserves the privacy of the verification data asset and the algorithm. The validation data assets are multiple independent sets of data assets, the encryption code is signed by the data processing system and stored in the data storage archive, and the performance of the algorithm is verified against multiple independent sets of data assets. A method that provides a single validation report for the validation of an algorithm that is aggregated from multiple validations performed. Implementations of the described techniques may include hardware, methods or processes, or computer software on computer-accessible media.

One common aspect is the process of identifying an algorithm, where the algorithm is integrated into a secure capsule computation framework provided by the algorithm developer, and where the secure capsule computation framework integrates the algorithm with data storage. The method includes the step of identifying, providing a verification data asset within a structure in a secure manner that preserves the privacy of the verification data asset and the algorithm. The method also includes executing a validation workflow algorithmically by the data processing system, the validation workflow taking a validation data asset as input and applying the algorithm to the validation data asset using the learned parameters. , it also includes the steps of outputting and executing the inference. The method also includes calculating, by the data processing system, the performance of the algorithm in providing the inference, the performance being calculated based on the gold standard label. The method also includes determining, by the data processing system, whether performance of the algorithm meets validation criteria defined by the algorithm developer. The method also includes optimizing one or more hyperparameters of the algorithm in the data processing system and reprocessing the algorithm using the optimized one or more hyperparameters if the performance of the algorithm does not meet validation criteria. It also includes the step of re-running the verification workflow. The method also includes providing, by the data processing system, the performance of the algorithm and the one or more hyperparameters to the algorithm developer if the performance of the algorithm meets validation criteria. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method.

Implementations may include one or more of the following features. The validation selection criteria include clinical cohort criteria, demographic criteria, and/or data set class balance, and the clinical cohort criteria include the group of people for which the data assets should be acquired for the cohort study, the cohort study the risk factors to which a group of people may be exposed over a period of time, the question or hypothesis to be answered and the associated disease or condition, other parameters defining the criteria for a cohort study, or any combination thereof. How to define. A secure encapsulated computational framework is provisioned within a computational infrastructure configured to accept the cryptographic code necessary to operate the algorithm, and the process of provisioning the computational infrastructure is performed using a instantiating a capsule computation framework; placing cryptographic code inside the secure capsule computation framework by an algorithm developer; and, after the secure capsule computation framework has been instantiated, encrypting decoding the code. Implementations of the described techniques may include hardware, methods or processes, or computer software on computer-accessible media.

One general aspect is, in a data processing system, receiving an algorithm and input data requirements associated with the algorithm, the input data requirements including validation selection criteria for data assets to operate on the algorithm. The method includes the step of receiving. The method also includes identifying, by the data processing system, the data asset as available from the data host based on validation selection criteria for the data asset. The method also includes curating, by the data processing system, data assets within a data storage structure within an infrastructure of the data host. The method also includes preparing a data asset in a data storage structure for processing by the algorithm by the data processing system. The method also includes, by the data processing system, integrating the algorithm into a secure capsule computation framework, wherein the secure capsule computation framework integrates the algorithm into a data asset within a data storage structure. It also includes providing and integrating the information in a secure manner that preserves privacy. The method also includes algorithmically executing a validation workflow by the data processing system, the validation workflow taking a data asset as input, finding patterns in the data asset using the learned parameters, and making inferences. It also includes the process of outputting and executing. The method also includes calculating, by the data processing system, the performance of the algorithm in providing the inference, the performance being calculated based on the gold standard label. The method also includes providing performance of the algorithm to an algorithm developer by the data processing system. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method.

Implementations may include one or more of the following features. The validation selection criteria include clinical cohort criteria, demographic criteria, and/or data set class balance, and the clinical cohort criteria include the group of people for which the data assets should be acquired for the cohort study, the cohort study the risk factors to which a group of people may be exposed over a period of time, the question or hypothesis to be answered and the associated disease or condition, other parameters defining the criteria for a cohort study, or any combination thereof. How to define. A method further comprising the step of hosting a data host by a data processing system, the hosting step comprising verifying that use of the data asset in the algorithm is compliant with data privacy requirements. The method may also include completing governance and compliance requirements, including approval from an institutional review board for use of data assets from the data host for purposes of validating the algorithm. The method also includes selecting the data storage structure from among the plurality of data storage structures and provisioning the data storage structure within the infrastructure of the data host. may include being based on the type of algorithms within the algorithm, the type of data within the data asset, the system requirements of the data processing system, or a combination thereof. The method further comprises maintaining the algorithms and data assets by the data processing system in a secure manner that preserves the privacy of the data assets and algorithms. Implementations of the described techniques may include hardware, methods or processes, or computer software on computer-accessible media.

Implementations may include one or more of the following features. A secure encapsulated computational framework is provisioned within a computational infrastructure configured to accept the cryptographic code necessary to operate the algorithm, and the process of provisioning the computational infrastructure is performed using a instantiating a capsule computation framework; placing cryptographic code inside the secure capsule computation framework by an algorithm developer; and, after the secure capsule computation framework has been instantiated, encrypting decoding the code. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the operations of the method. the data assets are multiple independent sets of data assets, the encryption code is signed by the data processing system and stored in the data storage archive, and the performance of the algorithm is A method that provides a single validation report for the validation of an algorithm that is aggregated from multiple validations performed. Implementations of the described techniques may include hardware, methods or processes, or computer software on computer-accessible media.

In some aspects, one or more data processors and, when executed on the one or more data processors, the one or more data processors have one or more data processors disclosed herein. A non-transitory computer readable storage medium containing instructions for performing some or all of the methods is provided.

In some embodiments, one or more data processors tangibly embodied in a non-transitory machine-readable storage medium perform some or all of the one or more methods disclosed herein. A computer program product is provided that includes instructions configured to perform.

Some aspects of the present disclosure include a system that includes one or more data processors. In some embodiments, the system, when executed on one or more data processors, causes the one or more data processors to perform part or all of one or more of the methods disclosed herein. A non-transitory computer-readable storage medium containing instructions for causing all and/or some or all of one or more processes to be performed. Some aspects of the present disclosure require one or more data processors to perform some or all of one or more methods and/or some or all of one or more processes disclosed herein. a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause the computer program product to perform the following steps:

[Invention 1001]
A method including the following steps:
in a data processing system, receiving an algorithm and input data requirements associated with the algorithm, the input data requirements including optimization and/or validation selection criteria for data assets to operate with the algorithm; The process of receiving;
identifying, by the data processing system, the data asset as available from a data host based on the optimization and/or validation selection criteria for the data asset;
curating, by the data processing system, the data assets in a data storage structure within an infrastructure of the data host;
preparing the data asset in the data storage structure for processing by the algorithm by the data processing system;
integrating, by the data processing system, the algorithm into a secure capsule computation framework, the secure capsule computation framework integrating the algorithm into the data asset within the data storage structure; providing and integrating said algorithm in a privacy-preserving and secure manner; and
operating the data asset through the algorithm by the data processing system;
[Present invention 1002]
The algorithm and the input data requirements are received from an algorithm developer, which is a different entity than the data host, and the optimization and/or validation selection criteria are based on the characteristics, format, etc. for which the data asset is to be operated with the algorithm. , and the method of the present invention 1001 for defining requirements.
[Present invention 1003]
the characteristics of the data asset and the requirements,
(i) the environment of said algorithm; (ii) the distribution of examples within said input data; (iii) the parameters and type of devices producing said input data; (iv) the variance versus bias; or (vi) any combination thereof.
The method of the invention 1002, defined based on.
[Present invention 1004]
The identifying step uses differential privacy to share information within the data asset by describing patterns of groups within the data asset while keeping private information about individuals within the data asset hidden. carried out,
the curating step includes selecting the data storage structure from among a plurality of data storage structures; and provisioning the data storage structure within the infrastructure of the data host;
the selection of the data storage structure is based on the type of algorithm, the type of data within the data asset, system requirements of the data processing system, or a combination thereof;
Method of the invention 1002.
[Present invention 1005]
further comprising hosting the data host by the data processing system;
the step of welcoming comprises ensuring that the use of the data asset in the algorithm is compliant with data privacy requirements;
The method of the invention 1001, 1002, or 1003.
[Present invention 1006]
The present invention, wherein preparing the data asset includes applying one or more transformations to the data asset, annotating the data asset, harmonizing the data asset, or a combination thereof. Any method of inventions 1001 to 1005.
[Present invention 1007]
said operating said data asset through said algorithm;
creating a plurality of instances of the model; splitting the data asset into a training dataset and one or more test datasets; and training the plurality of instances of the model with the training dataset. integrating results from the training of each of the plurality of instances of the model into a fully federated model; operating the one or more test datasets through the fully federated model; and calculating performance of the fully federated model based on the operation of one or more test datasets.
The method of any of the inventions 1001-1006, comprising performing.
[Present invention 1008]
said operating said data asset through said algorithm;
dividing the data asset in one or more validation datasets; operating the one or more validation datasets through the algorithm; and based on the operation of the one or more validation datasets. and calculating the performance of said algorithm using
The method of any of the inventions 1001-1006, comprising performing.
[Present invention 1009]
the secure encapsulated computational framework is provisioned within a computational infrastructure configured to accept cryptographic code necessary to operate the algorithm;
The provisioning of the computational infrastructure includes instantiating the secure encapsulated computational framework on the computational infrastructure, and adding the cryptographic code to the secure encapsulated computational framework by the algorithm developer. and decrypting the encrypted code after the secure capsule computation framework is instantiated.
The method according to any one of the inventions 1001 to 1008.
[Present invention 1010]
one or more data processors;
when executed on the one or more data processors, the one or more data processors:
an act of receiving an algorithm and input data requirements associated with the algorithm, wherein the input data requirements include optimization and/or validation selection criteria for data assets to operate with the algorithm;
an act of identifying the data asset as available from a data host based on the optimization and/or validation selection criteria for the data asset;
an act of curating the data asset within a data storage structure within an infrastructure of the data host;
an act of preparing the data asset in the data storage structure for processing by the algorithm;
an act of integrating the algorithm into a secure capsule computation framework, the secure capsule computation framework integrating the algorithm into the data asset within the data storage structure, preserving the privacy of the data asset and the machine learning model; providing and integrating operations in a secure manner;
an act of running the data asset through the algorithm;
a non-transitory computer-readable storage medium containing instructions for performing operations including;
system, including.
[Present invention 1011]
The algorithm and the input data requirements are received from an algorithm developer that is a different entity than the data host, and the optimization and/or validation selection criteria are based on the characteristics, format, and a system of the present invention 1010 for defining requirements.
[Invention 1012]
the characteristics of the data asset and the requirements,
(i) the environment of said algorithm; (ii) the distribution of examples within said input data; (iii) the parameters and type of devices producing said input data; (iv) the variance versus bias; or (vi) any combination thereof.
The system of the present invention 1011 is defined based on.
[Present invention 1013]
The act of identifying uses differential privacy to share information within the data asset by describing patterns of groups within the data asset while concealing private information about individuals within the data asset. carried out,
The act of curating includes selecting the data storage structure from among a plurality of data storage structures and provisioning the data storage structure within the infrastructure of the data host;
the selection of the data storage structure is based on the type of algorithm, the type of data within the data asset, the requirements of the system, or a combination thereof;
System of the present invention 1010.
[Present invention 1014]
the operations further include welcoming the data host;
the welcoming includes ensuring that use of the data asset in the algorithm is compliant with data privacy requirements;
The system of the invention 1010, 1011, or 1012.
[Present invention 1015]
The act of preparing the data asset includes applying one or more transformations to the data asset, annotating the data asset, harmonizing the data asset, or a combination thereof. Any system from Inventions 1010 to 1014.
[Invention 1016]
the act of operating the data asset through the algorithm;
creating a plurality of instances of the model; splitting the data asset into a training dataset and one or more test datasets; and training the plurality of instances of the model with the training dataset. integrating results from the training of each of the plurality of instances of the model into a fully federated model; operating the one or more test datasets with the fully federated model; and calculating performance of the fully federated model based on the operation of one or more test datasets.
The system of any of the inventions 1010-1015, comprising performing.
[Invention 1017]
the act of operating the data asset through the algorithm;
dividing the data asset in one or more validation datasets; operating the one or more validation datasets through the algorithm; and based on the operation of the one or more validation datasets. and calculating the performance of said algorithm using
The system of any of the inventions 1010-1015, comprising performing.
[Invention 1018]
the secure encapsulated computational framework is provisioned within a computational infrastructure configured to accept cryptographic code necessary to operate the algorithm;
The provisioning of the computing infrastructure includes instantiating the secure encapsulated computing framework on the computing infrastructure and transmitting the cryptographic code to the secure encapsulated computing framework through the algorithm developer. and decrypting the encrypted code after the secure capsule computation framework is instantiated.
The system according to any one of the present inventions 1010 to 1017.
[Invention 1019]
one or more data processors,
an act of receiving an algorithm and input data requirements associated with the algorithm, wherein the input data requirements include optimization and/or validation selection criteria for data assets to operate with the algorithm;
an act of identifying the data asset as available from a data host based on the optimization and/or validation selection criteria for the data asset;
an act of curating the data asset within a data storage structure within an infrastructure of the data host;
an act of preparing the data asset in the data storage structure for processing by the algorithm;
an act of integrating the algorithm into a secure capsule computation framework, the secure capsule computation framework integrating the algorithm into the data asset in the data storage structure, the data asset and the machine learning model; providing and integrating operations in a secure manner that preserves privacy;
an act of running the data asset through the algorithm;
an instruction configured to perform an action involving
A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising:
[Invention 1020]
The algorithm and the input data requirements are received from an algorithm developer that is a different entity than the data host, and the optimization and/or validation selection criteria are based on the characteristics, format, and the computer program product of the invention 1019 defining the requirements.
[Invention 1021]
the characteristics of the data asset and the requirements,
(i) the environment of said algorithm; (ii) the distribution of examples within said input data; (iii) the parameters and type of devices producing said input data; (iv) the variance versus bias; or (vi) any combination thereof.
The computer program product of the present invention 1020 as defined in accordance with .
[Invention 1022]
The act of identifying uses differential privacy to share information within the data asset by describing patterns of groups within the data asset while concealing private information about individuals within the data asset. carried out,
The act of curating includes selecting the data storage structure from among a plurality of data storage structures and provisioning the data storage structure within the infrastructure of the data host;
the selection of the data storage structure is based on the type of algorithm, the type of data within the data asset, the requirements of the system, or a combination thereof;
Computer program product of the present invention 1020.
[Invention 1023]
the act further includes an act of welcoming the data host;
the act of accepting comprises ensuring that use of the data asset in the algorithm is compliant with data privacy requirements;
The computer program product of the invention 1019, 1020, or 1021.
[Invention 1024]
the act of operating the data asset through the algorithm;
creating a plurality of instances of the model; splitting the data asset into a training dataset and one or more test datasets; and training the plurality of instances of the model with the training dataset. integrating results from the training of each of the plurality of instances of the model into a fully federated model; operating the one or more test datasets through the fully federated model; and calculating performance of the fully federated model based on the operation of one or more test datasets.
The computer program product of any of inventions 1019-1023, comprising: executing.
[Invention 1025]
the act of operating the data asset through the algorithm;
dividing the data asset in one or more validation datasets; operating the one or more validation datasets through the algorithm; and based on the operation of the one or more validation datasets. and calculating the performance of said algorithm using
The computer program product of any of inventions 1019-1023, comprising: executing.
[Invention 1026]
the secure encapsulated computational framework is provisioned within a computational infrastructure configured to accept cryptographic code necessary to operate the algorithm;
The provisioning of the computational infrastructure includes instantiating the secure encapsulated computational framework on the computational infrastructure, and adding the cryptographic code to the secure encapsulated computational framework by the algorithm developer. and decrypting the encrypted code after the secure capsule computation framework is instantiated.
The computer program product according to any of the inventions 1019 to 1025.
[Invention 1027]
A method including the following steps:
identifying a plurality of instances of an algorithm, each instance of the algorithm being integrated into one or more secure capsule computation frameworks, wherein the one or more secure capsule computation frameworks are integrated with the providing each instance of the algorithm to a training data asset within one or more data storage structures of one or more data hosts in a secure manner that preserves the privacy of the training data asset and each instance of the algorithm; , identifying process;
executing, by a data processing system, a federated training workflow with each instance of the algorithm, the federated training workflow taking the training data asset as input and targeting characteristics of the training data asset using parameters; one or more trained instances of the algorithm; The process of outputting and executing;
integrating, by the data processing system, the learned parameters for each trained instance of the algorithm into a fully federated algorithm, the step of integrating comprising: aggregating the learned parameters; updating the learned parameters of the fully federated algorithm with the aggregated parameters;
executing, by the data processing system, a test workflow with the fully federated algorithm, the test workflow taking test data as input and using the updated learned parameters to The process of finding patterns, outputting and executing inferences;
calculating, by the data processing system, the performance of the fully federated algorithm in providing the inference;
determining, by the data processing system, whether the performance of the fully federated algorithm satisfies an algorithm termination criterion;
If the performance of the fully federated algorithm does not meet the algorithm termination criteria, the data processing system replaces each instance of the algorithm with the fully federated algorithm and reruns the federated training workflow with each instance of the fully federated algorithm. the steps to be carried out; and
providing, by the data processing system, the performance of the fully federated algorithm and the aggregated parameters to an algorithm developer of the algorithm if the performance of the fully federated algorithm satisfies the algorithm termination criteria;
[Invention 1028]
the step of identifying the plurality of instances of the algorithm;
receiving, in the data processing system, the algorithm and input data requirements associated with the algorithm, the input data requirements providing optimization and/or validation selection criteria for data assets to operate with the algorithm; including, receiving,
identifying, by the data processing system, the data asset as available from the one or more data hosts based on optimization and/or validation selection criteria for the data asset;
curating, by the data processing system, the data assets in a data storage structure within an infrastructure of each data host of the one or more data hosts;
dividing at least a portion of the data assets into the training data assets within the data storage structure within the infrastructure of each data host of the one or more data hosts;
1027. The method of the invention 1027, comprising:
[Invention 1029]
said algorithm and said input data requirements are received from an algorithm developer that is a different entity than said one or more data hosts, and said optimization and/or validation selection criteria are for data assets to be operated on said algorithm; The method of the invention 1028 for defining the characteristics, format, and requirements of.
[Invention 1030]
The method of the present invention 1027, 1028, or 1029, wherein the federated training workflow further includes encrypting training gradients, and wherein the integrating includes decoding the training gradients.
[Present invention 1031]
sending, by the data processing system, aggregated parameters to each instance of the algorithm if the performance of the fully federated algorithm satisfies the algorithm termination criteria;
executing, by the data processing system, an update training workflow on each instance of the algorithm, the update training workflow updating the learned parameters with the aggregated parameters and updating the learned parameters with the aggregated parameters; Steps to execute that output multiple updated and trained instances
The method of invention 1027, 1028, 1029, or 1030, further comprising:
[Invention 1032]
1032. The method of the invention 1031 further comprising operating the remaining data assets through each instance of the algorithm by the data processing system.
[Present invention 1033]
said operating said data asset through each instance of said algorithm;
further dividing at least a portion of said data asset into one or more validation datasets; and operating said one or more validation datasets through each instance of said algorithm; and calculating performance of each instance of the algorithm based on the behavior of a validation dataset.
1032. The method of the invention 1031, comprising performing.
[Present invention 1034]
one or more data processors;
when executed on the one or more data processors, the one or more data processors:
an act of identifying multiple instances of an algorithm, each instance of the algorithm being integrated into one or more secure capsule computation frameworks, wherein the one or more secure capsule computation frameworks are integrated with the algorithm; providing each instance of the algorithm to a training data asset within one or more data storage structures of one or more data hosts in a secure manner that preserves the privacy of the training data asset and each instance of the algorithm; The action of identifying;
an act of executing a federated training workflow on each instance of the algorithm, the federated training workflow taking the training data asset as input and using parameters to map features of the training data asset to a target inference; calculating a loss function or an error function, updating the parameters to learned parameters to minimize the loss function or the error function, and outputting one or more trained instances of the algorithm; The action of
an act of integrating, by the data processing system, the learned parameters for each trained instance of the algorithm into a fully federated algorithm, wherein the act of integrating aggregates the learned parameters; and updating learned parameters of the fully federated algorithm with the aggregated parameters;
an act of executing, by the data processing system, a test workflow with the fully federated algorithm, the test workflow taking test data as input and using the updated learned parameters to perform a test workflow with the fully federated algorithm; Find patterns, output inferences, and the actions to perform.
an act of calculating, by the data processing system, the performance of the fully federated algorithm in providing the inference;
an act of determining, by the data processing system, whether the performance of the fully federated algorithm satisfies an algorithm termination criterion;
If the performance of the fully federated algorithm does not meet the algorithm termination criteria, the data processing system replaces each instance of the algorithm with the fully federated algorithm and reruns the federated training workflow with each instance of the fully federated algorithm. The action to be performed and
an act of providing, by the data processing system, the performance of the fully federated algorithm and the aggregated parameters to an algorithm developer of the algorithm if the performance of the fully federated algorithm satisfies the algorithm termination criteria;
a non-transitory computer-readable storage medium containing instructions for performing operations including;
system, including.
[Invention 1035]
A method including the following steps:
identifying, by the data processing system, data assets available from the data host based on data asset selection criteria;
curating, by the data processing system, the data assets in a data storage structure within an infrastructure of the data host;
preparing a transformer prototype data set to be used by the data processing system as a guide for developing algorithms for data transformation, the transformer prototype data set including key attributes of a harmonization process; The process of capturing and preparing;
creating, at the data processing system, a first harmonized transformer set for transformation of the data asset based on a current format of data in the transformer prototype dataset;
applying, by the data processing system, the first set of harmonized transformers to the data asset to generate a transformed data asset;
preparing, by the data processing system, a harmonized prototype data set to be used as a guide for developing algorithms for data transformation, the harmonized prototype data set including key attributes of the harmonization process; The process of taking in and preparing;
creating, by the data processing system, a second set of harmonized transformers for transformation of the transformed data asset based on a current format of data in the harmonized prototype data set;
applying, by the data processing system, the second harmonization transformer set to the transformed data asset to produce a harmonized data asset; and
operating the reconciled data asset through an algorithm by the data processing system, the algorithm operating the reconciled data asset in the data storage structure; operating within a secure encapsulated computational framework that provides in a secure manner that preserves the privacy of the acquired data assets and said algorithms;
[Invention 1036]
The selection criteria of the invention 1035 is received from an algorithm developer that is a different entity than the data host, and the selection criteria defines characteristics, formats, and requirements for the data asset to operate with the algorithm. Method.
[Present invention 1037]
the characteristics of the data asset and the requirements,
(i) the environment of said algorithm; (ii) the distribution of examples within said input data; (iii) the parameters and type of devices producing said input data; (iv) the variance versus bias; or (vi) any combination thereof.
The method of the invention 1036, defined based on.
[Invention 1038]
the algorithm developer for the purpose of anonymizing the transformer prototype dataset and creating the first harmonized transformer set for transformation of the data asset; 1037. The method of the invention 1037, further comprising the step of subjecting to.
[Invention 1039]
The method of the invention 1035, 1036, 1037, or 1038, wherein applying the first set of harmonized transformers to the data asset occurs within the data structure.
[Invention 1040]
further comprising, at the data processing system, annotating the transformed data asset according to a defined annotation protocol to generate an annotated dataset;
the annotating the transformed data is performed within the data structure, and the second harmonization transformer set is applied to the annotated dataset to generate a harmonized data asset. Ru,
The method according to any one of the present invention 1035 to 1039.
[Present invention 1041]
The method of any of the inventions 1035-1040, wherein the applying the second set of harmonized transformers to the annotated data asset occurs within the data structure.
[Present invention 1042]
determining whether the first harmonized transformer set, the annotation, and the second harmonized transformer set are applied successfully and without violating data privacy requirements;
The method of invention 1040 or 1041, further comprising:
[Invention 1043]
one or more data processors;
when executed on the one or more data processors, the one or more data processors:
an act of identifying the data asset available from a data host based on data asset selection criteria;
an act of curating the data asset within a data storage structure within an infrastructure of the data host;
an act of preparing a transformer prototype data set to be used as an algorithm for data transformation, the transformer prototype data set capturing key attributes for a harmonization process;
an act of creating a first harmonized transformer set for transformation of the data asset based on a current format of data in the transformer prototype dataset;
an act of applying the first harmonization transformer set to the data asset to generate a transformed data asset;
an act of preparing a harmonized prototype data set to be used to develop an algorithm for data transformation, the harmonized prototype data set capturing key attributes of the harmonization process;
an act of creating a second harmonized transformer set for transformation of the transformed data asset based on a current format of data in the harmonized prototype data set;
applying the second harmonization transformer set to the transformed data asset to generate a harmonized data asset;
an act of operating the harmonized data asset through an algorithm, the algorithm operating the harmonized data asset in the data storage structure, the harmonized data asset and the algorithm; The operating and
a non-transitory computer-readable storage medium containing instructions for performing operations including;
system, including.
[Present invention 1044]
A method including the following steps:
identifying an algorithm or model, the algorithm or model being integrated into a secure capsule computation framework, the secure capsule computation framework integrating the algorithm or model within a data storage structure of a data host; identifying, providing a training data asset in a secure manner that preserves the privacy of the training data asset and the algorithm or mode;
executing, by a data processing system, a federated training workflow with the algorithm or model, the federated training workflow taking the training data asset as input and using parameters to target inferences about features of the training data asset; calculating a loss function or an error function, updating parameters to the learned parameters to minimize the loss function or the error function, and outputting a trained algorithm or model;
integrating, by the data processing system, the learned parameters of the algorithm or model into a fully federated algorithm or model, the integrating step comprising aggregating the learned parameters to create an aggregated integrating, the step comprising: obtaining parameters; and updating learned parameters of the fully federated algorithm or model with the aggregated parameters;
executing a test workflow with the fully federated algorithm or model by the data processing system, the test workflow taking test data as input and using the updated learned parameters to execute a test workflow with the fully federated algorithm or model; The process of finding patterns within and outputting and executing inferences;
calculating, by the data processing system, the performance of the fully federated algorithm in providing the inference;
determining, by the data processing system, whether the performance of the fully federated algorithm or model satisfies algorithm termination criteria;
If the performance of the fully federated algorithm or model does not meet the algorithm termination criteria, the data processing system replaces the algorithm or model with the fully federated algorithm or model and executes the federated training workflow with the fully federated algorithm or model. a step of re-executing; and
If the performance of the fully federated algorithm or model satisfies the algorithm exit criteria, the data processing system transmits the performance of the fully federated algorithm or model and the aggregated parameters to an algorithm developer of the algorithm or model. Process of providing.
[Invention 1045]
A method including the following steps:
identifying an algorithm, the algorithm being integrated into a secure capsule computation framework provided by an algorithm developer, the secure capsule computation framework integrating the algorithm with validation data in a data storage structure; identifying, providing an asset in a secure manner that preserves the privacy of the verification data asset and the algorithm;
executing, by a data processing system, a validation workflow with the algorithm, the validation workflow taking the validation data asset as input and applying the algorithm to the validation data asset using the learned parameters; , the process of outputting and executing the inference;
calculating, by the data processing system, a performance of the algorithm in providing the inference, the performance being calculated based on a gold standard label;
determining, by the data processing system, whether the performance of the algorithm meets validation criteria defined by an algorithm developer;
if the performance of the algorithm does not meet the validation criteria, optimizing one or more hyperparameters of the algorithm at the data processing system and using the optimized one or more hyperparameters; re-running the validation workflow with the algorithm; and
If the performance of the algorithm meets the validation criteria, providing the performance of the algorithm and the one or more hyperparameters to the algorithm developer by the data processing system.
[Invention 1046]
the step of identifying the algorithm;
receiving, in the data processing system, the algorithm and input data requirements associated with the algorithm, the input data requirements including validation selection criteria for data assets to operate with the algorithm; ,
identifying, by the data processing system, the data asset as available from a data host based on the validation selection criteria for the data asset;
curating, by the data processing system, the data assets in a data storage structure within an infrastructure of the data host;
dividing at least a portion of the data asset into the verification data asset within the data storage structure within the infrastructure of the data host;
1044. The method of the invention 1044, comprising:
[Invention 1047]
The validation selection criteria include clinical cohort criteria, demographic criteria, and/or data set class balance, and the clinical cohort criteria includes a group of people for which the data asset is to be acquired for a cohort study. , the type of said cohort study, the risk factors to which said group of people may be exposed over a period of time, the question or hypothesis to be answered and the associated disease or condition, and other parameters defining the criteria for said cohort study; or any combination thereof.
[Invention 1048]
accommodating the data host by the data processing system, the accommodating step comprising verifying that use of the data asset in the algorithm is compliant with data privacy requirements; ; and
completing governance and compliance requirements, including approval from an institutional review board for use of said data assets from said data host for the purpose of validating said algorithms;
further including;
The curating step includes selecting the data storage structure from among a plurality of data storage structures and provisioning the data storage structure within the infrastructure of the data host, the selection is based on a type of algorithm within the algorithm, a type of data within the data asset, system requirements of the data processing system, or a combination thereof;
The method of the invention 1045 or 1046.
[Invention 1049]
if the performance of the algorithm meets the verification criteria, maintaining by the data processing system the algorithm and the verification data asset in a secure manner that preserves the privacy of the verification data asset and the algorithm;
The method of any of inventions 1044-1047, further comprising.
[Invention 1050]
the secure encapsulated computational framework is provisioned within a computational infrastructure configured to accept cryptographic code necessary to operate the algorithm;
The provisioning of the computational infrastructure includes instantiating the secure encapsulated computational framework on the computational infrastructure, and adding the cryptographic code to the secure encapsulated computational framework by the algorithm developer. and decrypting the encrypted code after the secure capsule computation framework is instantiated.
The method according to any one of the inventions 1044 to 1048.
[Present invention 1051]
the verification data asset is a plurality of independent sets of data assets, the encryption code is signed by the data processing system and stored in a data storage archive, and the performance of the algorithm is a plurality of independent sets of data assets; The method of the invention 1049 is provided as a single validation report for validation of said algorithm aggregated from multiple validations performed on a set of data assets.
[Invention 1052]
one or more data processors;
when executed on the one or more data processors, the one or more data processors:
an act of identifying an algorithm, the algorithm being integrated into a secure capsule computation framework provided by an algorithm developer, the secure capsule computation framework integrating the algorithm with validation data in a data storage structure; an act of identifying, providing an asset in a secure manner that preserves the privacy of the validation data asset and the algorithm;
an act of executing a validation workflow with the algorithm, the validation workflow taking the validation data asset as input, finding patterns in the validation data asset using learned parameters, and outputting inferences; The action to be performed and
an act of calculating performance of the algorithm in providing the inference, wherein the performance is calculated based on a gold standard label;
an act of determining whether the performance of the algorithm meets validation criteria defined by an algorithm developer;
If the performance of the algorithm does not meet the validation criteria, optimize one or more hyperparameters of the algorithm and use the optimized one or more hyperparameters to perform the validation workflow with the algorithm. and the action of re-running the
an act of providing the performance of the algorithm and the one or more hyperparameters to the algorithm developer if the performance of the algorithm meets the verification criteria;
a non-transitory computer-readable storage medium containing instructions for performing operations including;
system, including.
[Present invention 1053]
A method including the following steps:
receiving, in a data processing system, an algorithm and input data requirements associated with the algorithm, the input data requirements including validation selection criteria for data assets to operate with the algorithm;
identifying, by the data processing system, the data asset as available from a data host based on the validation selection criteria for the data asset;
curating, by the data processing system, the data assets in a data storage structure within an infrastructure of the data host;
preparing, by the data processing system, the data asset in the data storage structure for processing by the algorithm;
integrating, by the data processing system, the algorithm into a secure capsule computation framework, the secure capsule computation framework integrating the algorithm into the data asset within the data storage structure; providing and integrating said algorithm in a privacy-preserving and secure manner; and
executing, by the data processing system, a validation workflow with the algorithm, the validation workflow taking the data asset as input and using the learned parameters to find patterns in the data asset and to make inferences; The process of outputting and executing;
calculating, by the data processing system, the performance of the algorithm in providing the inference, the performance being calculated based on a gold standard label; and
providing the performance of the algorithm to the algorithm developer by the data processing system;
[Invention 1054]
The validation selection criteria include clinical cohort criteria, demographic criteria, and/or data set class balance, and the clinical cohort criteria includes a group of people for which the data asset is to be acquired for a cohort study. , the type of said cohort study, the risk factors to which said group of people may be exposed over a period of time, the question or hypothesis to be answered and the associated disease or condition, and other parameters defining the criteria for said cohort study; or any combination thereof.
[Present invention 1055]
accommodating the data host by the data processing system, the accommodating step comprising verifying that use of the data asset in the algorithm is compliant with data privacy requirements; ; and
completing governance and compliance requirements, including approval from an institutional review board for use of said data assets from said data host for the purpose of validating said algorithms;
further including;
The curating step includes selecting the data storage structure from among a plurality of data storage structures and provisioning the data storage structure within the infrastructure of the data host, the selection is based on a type of algorithm within the algorithm, a type of data within the data asset, system requirements of the data processing system, or a combination thereof;
The method according to the invention 1053 or 1054.
[Invention 1056]
1056. The method of any of the inventions 1053-1055, further comprising maintaining, by the data processing system, the algorithm and the data asset in a secure manner that preserves the privacy of the data asset and the algorithm.
[Present invention 1057]
the secure encapsulated computational framework is provisioned within a computational infrastructure configured to accept cryptographic code necessary to operate the algorithm;
The provisioning of the computational infrastructure includes instantiating the secure encapsulated computational framework on the computational infrastructure, and adding the cryptographic code to the secure encapsulated computational framework by the algorithm developer. and decrypting the encrypted code after the secure capsule computation framework is instantiated.
The method according to any one of the inventions 1053 to 1056.
[Invention 1058]
the data asset is a plurality of independent sets of data assets, the encryption code is signed by the data processing system and stored in a data storage archive, and the performance of the algorithm is a plurality of independent sets of data assets; 1057. The method of the invention 1057, provided as a single validation report for validation of said algorithm aggregated from multiple validations performed on a set of data assets.
The terms and expressions used are used as terms of description rather than limitation, and the use of such terms and expressions excludes any equivalent of that portion of the features illustrated and described. It is to be understood that this is not intended and that various modifications may be made within the scope of the claimed invention. Thus, while the claimed invention has been particularly disclosed in aspects and optional features, those skilled in the art will recognize that modifications and variations of the concepts disclosed herein may be used and that such It is to be understood that such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

The invention will be better understood upon consideration of the following non-limiting figures.

FIG. 2 is a diagram illustrating an AI ecosystem in accordance with various aspects. 1 is a diagram illustrating an AI algorithm development platform in accordance with various aspects. FIG. FIG. 3 illustrates core processes for optimizing and/or validating models on data (e.g., clinical and health data) in accordance with various aspects. FIG. 2 illustrates a process of hosting one or more data hosts on an AI algorithm development platform in accordance with various aspects. FIG. 2 illustrates a process for identifying, acquiring, and curating data assets to be used with an AI algorithm development platform in accordance with various aspects. FIG. 2 illustrates a process for converting data assets for use with an AI algorithm development platform in accordance with various aspects. FIG. 2 illustrates a process for annotating data assets for use with an AI algorithm development platform in accordance with various aspects. FIG. 2 illustrates a process for harmonizing data assets to be used with an AI algorithm development platform in accordance with various aspects. FIG. 2 illustrates a process for optimizing one or more models using an AI algorithm development platform in accordance with various aspects. FIG. 2 illustrates a process for validating one or more models using an AI algorithm development platform in accordance with various aspects. FIG. 3 illustrates an example flow for optimizing and/or validating models for clinical and health data in accordance with various aspects. 1 is an illustration of an example computing device as part of a data processing system in accordance with various aspects. FIG.

detailed description
I. Introduction
This disclosure describes techniques for developing AI applications and/or algorithms by distributing analysis across multiple sources of privacy-protected harmonized data (e.g., clinical and health data). More specifically, some aspects of the present disclosure improve application and/or algorithm development ( provides an AI algorithm development platform that accelerates algorithm development (which may individually or collectively be referred to herein as algorithm development). Although various aspects of machine learning and algorithmic architectures are disclosed herein in which AI algorithms are developed to solve problems in the healthcare industry, these architectures and techniques may also be used in other types of systems and settings. It should be understood that this can also be implemented. For example, use these architectures and technologies to ensure that the sensitivity of the data (e.g., whether the data contains trade secrets or private data about individuals) prevents the sharing of the data outside the boundaries of the organization responsible for protecting the data. , can be implemented in the development of AI algorithms in many industries (finance, life sciences, supply chain, national security, law enforcement, public safety, etc.).

Various aspects disclosed herein describe techniques for algorithm and model development, including training, testing, optimization, and validation. The terms "algorithm(s)" and "model(s)" are used interchangeably in the detailed description (but not in the claims) for readability and brevity; Thus, where the term "algorithm(s)" is used herein, the term "model(s)" may be substituted and "(one or more) Where the term "model(s)" is used, it may be replaced by the term "algorithm(s)". However, it should be understood that these terms have separate meanings as follows. An "algorithm(s)" is a function, method, or procedure that is implemented to complete a task or solve a problem, whereas a "model(s)" is a A well-defined calculation involving one or more algorithms that takes as input a value or set of values and produces as output a value or set of values. For example, to find the value of y for a particular value of x when the variables are linearly related, an algorithm such as y=mx+b is used to describe the linear relationship between the x and y variables. obtain. A model with slope (m) and intercept (c) values may be trained, tested, optimized, or validated, and then the model uses the algorithm y=mx+b and slope (m) and intercept (c) values. can be used to find the value of y for different points of x. Typically, regulated AI and ML algorithms (e.g., those intended for use in medical applications) are generalizable and therefore independent of the environment in which they are used, regardless of ethnicity, application environment and It should be performed in a manner that is independent of details such as workflow (e.g., specific clinical environment) and geography. To achieve consistent performance, algorithms need timely access to highly diverse data to enable discovery, optimization, and validation.

One of the biggest barriers to the discovery, optimization, and validation of regulated AI and ML algorithms is the lack of timely access to highly diverse data, often privacy-protected data. . Privacy-protected data is legally protected and is considered organizational property by institutions, even if it resides in an isolated repository with different "owners" and algorithm developers have access to the data. is likely not in a computable form. Within the medical industry, this issue was addressed by Francis Collins, MD, PhD, Director of the National Institutes of Health (NIH), in a July 2018 presentation to the NIH Workshop on Harnessing Artificial Intelligence and Machine Learning to Advance Biomedical Research. ``Many researchers spend considerable time making their data accessible and analyzable, and this aspect needs to be improved.'' ing. In addition, in its Data Science Strategic Plan, the NIH states that ``currently... there is a general public base that transforms or enhances innovative algorithms and tools into enterprise-ready resources that meet industry standards for ease of use and operational efficiency.'' He admits that there is no universal system. These challenges apply broadly across industries where data assets are distributed across different locations and include sensitive or private data.

Key roles in the development of AI solutions include algorithm developers, data providers, third-party annotators, third-party verifier(s), algorithm regulator(s), and end users. or customers (e.g., healthcare organizations). As an example of the complexity of the process of developing algorithms in regulated environments (as is the case in healthcare, finance, law enforcement, education, and many other cases), This paper examines the company's current process of developing algorithms. Such companies typically endure lengthy processes to create algorithms that meet industry standards. This process includes securing data donors, establishing the technical environment, establishing security and privacy reviews, preparing the data (e.g., curation and anonymization), and transferring the anonymized data to algorithm developers. applying AI or ML to anonymized data for algorithm discovery and development; documenting the results of algorithm discovery and development; and transmitting algorithms or models to regulatory bodies for review and approval. (e.g., the Food and Drug Administration (FDA)). This process includes reconstruction attacks (reconstructing raw private data from public features extracted from the raw data), model inversion attacks (utilizing responses received from the model to create feature vectors), Includes several privacy threats, including membership inference attacks (determining whether a sample was a member of the training set used to build the ML model), and re-identification of private data.

Many traditional privacy-enhancing techniques allow data providers and algorithm developers to utilize secure communications, cryptographic techniques, or differentially private data release (perturbation techniques) to overcome or minimize these privacy threats. I'm concentrating on what I want to do. For example, differential privacy is effective in preventing membership inference attacks. Additionally, limiting the model inference output (e.g., only class labels) can reduce the success of model inversion and membership inference attacks. Despite the aforementioned techniques to protect private data while discovering, optimizing, and validating, many algorithm developers are still struggling with increased time-to-market, cost, and complexity of algorithm development. Avoiding the use of privacy-protected data (including clinical data from healthcare institutions) that incorporates inherent biases due to lack of fidelity (including clinical) and used in privacy-sensitive industries An algorithm is generated that is essentially proof of concept research, rather than real-world research that would result in creating an AI to do something. For example, the timeline for completing the steps necessary to secure privacy-protected data exceeds typical venture capital return expectations. What capital markets are experiencing is the timeliness for industries that utilize less sensitive data to generate algorithms and expect those algorithms in privacy-sensitive industries (including healthcare). For example, this lack of perseverance to achieve profits may lead to early-stage companies running out of funds to continue the multi-site training required for clinically deployed algorithms, leading to a shift toward medical AI. would mean that most investments would not reach their goals.

To address these issues, various aspects target platforms and technologies for developing artificial intelligence applications by distributing analysis across multiple sources of privacy-protected, harmonized clinical and health data. do. For example, in the context of medical AI algorithm development, a platform may include a group of participating medical data provider organizations from around the world that collectively represent ethnically and device diverse data. This group may participate in a centralized contract and pricing model that allows algorithm developers to establish one contract to access multiple data sources. Establish within a platform of shared technology infrastructure for data harmonization, annotation, validation, and federated training, allowing algorithm developers to access multiple data sources in a single development and deployment environment It can be. Multi-organizational collaboration allows algorithm developers to meet a single standardized security and privacy review that can be applied across multiple organizations. It may be possible to adopt. Data preparation should employ standardized, multi-institutional Institutional Review Board-approved (IRB) formats (utilized in the context of life sciences companies) that eliminate redundancy when collaborating with multiple organizations; and data curation, anonymization, transformation, and augmentation for debugging and debugging in a way that protects the underlying assets without exposing the underlying dataset to algorithm developers. (tools developed for one data host may be reused with data from additional data hosts or modified for reuse). Preparation data to be used for algorithm discovery, validation, or training is never migrated from the control of the original data owner. The software components responsible for performing discovery, validation, and/or training operate in an encapsulated format within an infrastructure controlled by the data host. This infrastructure may be on-premises or in the cloud, but is entirely within the domain control of the data host.

The platform provides a secure encapsulated computational services framework for applying AI or ML to data prepared in a privacy-preserving manner. In some instances, technical calibration is performed as part of this process, but first on a small portion of the data prepared in an iterative process to ensure that the end-to-end computational pipeline is complete. obtain. Because these steps operate within the platform's standardized infrastructure, this process can be managed by the platform rather than by the data host's employees. This means that the process is repeatable, scalable, and performed by well-trained individuals who are familiar with the process. Application of AI or ML to prepared data may include algorithm discovery, algorithm optimization, and/or algorithm validation.

In some examples, the operation of algorithmic discovery may be implemented entirely within the technical control of the data host, allowing for some level of protection of the underlying data. Certain types of algorithm development activities may be protected using differential privacy techniques to ensure minimal risk of data leakage. Depending on the level of trust between the parties, new algorithm discovery methods may be enabled through extensive monitoring of algorithm activity and outgoing traffic from encapsulated software running in the data host environment. One example aspect of the present disclosure is, in a data processing system, receiving an algorithm or model and input data requirements associated with the algorithm, the input data requirements for a data asset to operate on the algorithm or model. identifying the data asset as available from the data host based on the optimization and/or validation selection criteria for the data asset by the receiving process and the data processing system, including the optimization and/or validation selection criteria for the data asset; curating, by the data processing system, the data assets in a data storage structure within an infrastructure of the data host; and curing, by the data processing system, the data assets in the data storage structure for processing by an algorithm or model. and integrating the algorithm or model into a secure capsule computation framework by the data processing system, wherein the secure capsule computation framework integrates the algorithm or model into a data asset within a data storage structure. The method includes providing and integrating assets and algorithms or models in a privacy-preserving and secure manner, and operating the data assets through the algorithms or models by a data processing system.

In some examples, optimization may occur entirely within the data host's infrastructure and control, without requiring external sharing of data. Depending on the security level chosen by the data host, it is possible to support algorithm optimization (e.g. using homomorphic encryption and refactoring algorithms) without the possibility of leakage of training data. All tools for cohort development can be managed with strict differential privacy controls to prevent reconfiguration, differential, tracking, and other attacks on private data. One example aspect of the present disclosure is a step of identifying an algorithm or model, the algorithm or model being integrated into a secure capsule computation framework, the secure capsule computation framework integrating the algorithm or model with data. providing training data assets in a host data storage structure in a secure manner that preserves the privacy of the training data assets and the algorithm or mode, and performing a federated training workflow with the algorithm or model by the data processing system; The federated training workflow takes a training data asset as input, maps features of the training data asset to a target inference using parameters, computes a loss function or error function, and computes a loss function or error function. updating the learned parameters of the algorithm or model to minimize the trained algorithm or model; A step of integrating into a model, the step of integrating includes a step of aggregating learned parameters to obtain the aggregated parameters, and updating the learned parameters of the fully federated algorithm or the model with the aggregated parameters. and executing a test workflow with a fully federated algorithm or model by a data processing system, the test workflow taking test data as input and updating learned parameters. to find patterns in the test data and output inferences (e.g., model predictions or algorithm results) based on the input data, learned parameters, and configuration of the algorithm or model. calculating, by the data processing system, the performance of the fully federated algorithm in providing an inference; and determining, by the data processing system, whether the performance of the fully federated algorithm or model satisfies algorithm termination criteria. and, if the performance of the fully federated algorithm or model does not meet algorithm termination criteria, the data processing system replaces the algorithm or model with the fully federated algorithm or model and reruns the federated training workflow with the fully federated algorithm or model. and providing, by the data processing system, the performance of the fully federated algorithm or model and the aggregated parameters to an algorithm developer of the algorithm or model if the performance of the fully federated algorithm or model meets algorithm termination criteria.

In some examples, validation may occur entirely within the data host's infrastructure and control, without requiring external sharing of data. There is no possibility of leakage of validation data, and all tools for cohort development are managed with strict differential privacy controls to prevent reconstruction, differential, tracking, and other attacks on private data. One example aspect of the present disclosure is, in a data processing system, receiving an algorithm and input data requirements associated with the algorithm, wherein the input data requirements are validation selection criteria for data assets to operate on the algorithm. and identifying, by the data processing system, the data asset as available from the data host based on validation selection criteria for the data asset; and, by the data processing system, an infrastructure of the data host. preparing, by the data processing system, the data assets in the data storage structure for processing by the algorithm; and providing the algorithm with secure encapsulated computation by the data processing system. integrating into a framework, the secure capsule computation framework providing the algorithm to the data asset within the data storage structure in a secure manner that preserves the privacy of the data asset and the algorithm; algorithmically executing a validation workflow by the processing system, the validation workflow taking the validation data asset as input, finding patterns in the validation data asset using the learned parameters, and outputting inferences; calculating, by the data processing system, performance of the algorithm in providing the inference, the performance being calculated based on a gold standard label; , providing performance of the algorithm to an algorithm developer. A key requirement for algorithms or models that require documentation is often validation of the algorithm or model against diverse independent datasets from different geographies. The platform performs validation using consistent operating parameters and validation reports on independent datasets, and the platform determines the underlying characteristics of each dataset (demographics, equipment used, protocols used for data collection). Completing the verification required for most document filings is dramatically easier, faster, and cheaper.

Advantageously, these techniques enable the deployment of privacy-preserving decentralized platforms to support multi-site development of algorithms and models. Additionally, these technologies can help algorithm developers accelerate time-to-market for algorithms or models, optimize commercial viability, and reduce risk of investments in algorithms and models that utilize private data. I can help. Additionally, these technologies help data hosts optimize the value of their data assets, protect the privacy of individuals whose data is represented in applicable datasets, and improve patient outcomes (in medical applications). in a way that creates innovative thought leadership in the field.

II. Artificial Intelligence (AI) Ecosystem Figure 1 includes data augmentation from datasets 105a-105n, deployment of algorithms/models 110a-110n, validation of algorithms/models 110a-110n, and optimization of algorithms/models 110a-110n. Secure federated computation on private datasets 105a-105n (where "n" represents any natural number), including (training/testing) and federated training of algorithms/models 110a-110n on multiple datasets 105a-105n. Showing 100 AI ecosystems that enable AI development platform 110 includes an AI system 115 and secure capsule computing services 120a-120n. AI system 115 manages the development and deployment of software assets within AI ecosystem 100. Various software modules, including pre-trained algorithms/models 110a-110n for training, optimization, scoring and validation, and algorithm training code (for training the algorithms/models on data) Developed and deployed via system 115. For example, AI system 115 may deploy or output various software components to one or more secure capsule computing services 120a-120n. Secure capsule computing services 120a-120n are encapsulated or other portable software components that can be computed in a secure capsule computing service environment entirely within the computing domain of the data host.

Secure capsule computing services facilitate the deployment of software containing algorithms to be verified and/or trained to an external environment (in this case, the data host's computing environment), and further protect the privacy of the software deployed within the capsule. and provide security services to protect both the security of the host computing environment. In different aspects, encryption, secure encapsulation computational frameworks, isolation, obfuscation, and other methods may be used to provide security to all parties. For example, in some instances, an organization running software on a secure encapsulated computational module (e.g., a data host) may not be able to inspect, copy, or modify an algorithm developer's proprietary software. , which can be important to algorithm developers. Secure encapsulated computational services provide a portable environment to support such computational models. Additionally, stakeholders operating secure encapsulated compute services often believe that the hosted software is malicious and can harm the host's infrastructure or protect data privacy. We want protection against the possibility of infringement. Additionally, AI system 115 may deploy or output data resources, model parameters, or shared data (eg, a parent-teacher training paradigm) for training. In exchange, the AI system 115 includes computational results, data, and computational monitoring results, trained models, model parameters, or other results of deployed computational components and processes, such as secure capsule computational services 120a-120n. May receive input.

Secure capsule computing services 120a-120n may provide the same or different functionality or operations. For example, secure capsule computing services 120a and 120n may manage data transformation and computing activities, but they may perform different functions or operations in performing computing activities. Secure capsule computing service 120a receives data and software from AI system 100 and may receive uncurated data from data sources 125a-125n (eg, data sources 120a and 120b). Data may be imported by software (eg, algorithms/models 110a-110n) from AI system 100, optionally stored, transformed, otherwise harmonized, and then calculated. In contrast, secure capsule computing service 120n receives data and software from AI system 100, receives uncurated data from data sources 125a through 125n (e.g., data sources 120a and 120b), and receives data and software from one or more Data from third parties 130 may be combined. Data can be imported by software (eg, algorithms/models 110a-110n) from AI system 100, optionally stored, transformed, combined with combined data, otherwise aligned, and then calculated. Secure capsule computing services 120a-120n communicate with data sources 125a-125n and optionally a third party 130 via communication network 135. Examples of communication networks 135 may include mobile networks, wireless networks, cellular networks, area networks (LANs), wide area networks (WANs), other wireless communication networks, or combinations thereof.

Figure 2 shows an AI algorithm development platform 200 for developing artificial intelligence algorithms by distributing analysis across multiple sources of privacy-preserving harmonized data (e.g., implemented within the AI ecosystem 100 described with respect to Figure 1). data processing system). In one example, platform 200 includes an AI system 205 that communicates with a network of secure capsule computing services 210 (for simplicity, only one secure capsule computing service is shown). AI system 205 may include a data store that includes a data science development module 215, a data harmonizer workflow creation module 220, a software deployment module 225, a federated master algorithm training module 230, a system monitoring module 235, and a global combined data 240. AI system 205 may communicate with one or more algorithm developers and is configured to receive one or more algorithms or models to be optimized and/or validated in a new project.

Data science development module 215 may be configured to receive input data requirements from one or more algorithm developers for optimization and/or validation of one or more models. Input data requirements define the objectives of data curation, data transformation, and data harmonization workflows. Input data requirements also provide constraints for identifying data assets that are acceptable for use in one or more models. Data harmonizer workflow creation module 220 may be configured to manage the development and deployment of transformation, harmonization, and annotation protocols. Software deployment module 225, along with data science development module 215 and data harmonizer workflow creation module 220, may be configured to evaluate data assets for use in one or more models. This process can be automated or can be an interactive search/query process. Software deployment module 225, along with data science development module 215, may be further configured to integrate the model, along with the necessary libraries and resources, into a secure capsule computation framework.

In some aspects, robust superior algorithms/models learned from multiple independent private datasets 245, 250 (e.g., clinical and health data) collected by data hosts 255 from sources 260 (e.g., patients) is required to develop. Federated master algorithm training module 230 may be configured to aggregate learning from independent data sets into a single master algorithm. In different aspects, the algorithm methodology for federated training may be different. For example, sharing model parameters, ensemble learning, parent-supervised learning on shared data, and many other methods can be developed to enable federated training. Privacy and security requirements, along with commercial considerations such as determining the price each data system can be paid for access to the data, may determine which federated training methodology is used.

System monitoring module 235 monitors activity in secure capsule computing service 210. Activities monitored range from behavioral tracking such as computational workload, error conditions, and connection status to data volume processed, algorithm convergence status, variation in data characteristics, data errors, algorithm/model performance metrics, and It can extend to data science monitoring such as numerous additional metrics required by each use case and aspect.

In some instances, it may be desirable to augment the private datasets 245, 250 with additional data 240 (combined data). For example, air quality data for a geographic location can be combined with patient geographic location data to ascertain environmental exposure. In particular examples, combined data 240 may be sent to secure encapsulated calculation service 210 to be combined with data 245, 250 during data alignment or computation.

Secure capsule computing service 210 may include harmonizer workflow module 265, harmonized data 270, remote computing module 275, system monitoring module 280, and data management module 285. Transformation, harmonization, and annotation workflows managed by data harmonizer workflow creation module 230 may be deployed and executed in the environment by harmonizer workflow module 265 using transformation and harmonization data 270. In one example, combined data 240 may be sent to harmonizer workflow module 265 to be combined with data 245, 250 during data harmonization. The remote computing module 275 may be configured to run the private data sets 245, 250 through the algorithm/model. In some aspects, operating includes creating multiple instances of the algorithm/model and splitting the private dataset 245, 250 into a training dataset and one or more test datasets. , training multiple instances of the algorithm/model on a training dataset, integrating the results from training each of the multiple instances of the model into a fully federated algorithm/model, and one or more test datasets. and calculating the performance of the fully federated model based on the behavior of one or more test datasets. In other aspects, operating includes splitting, combining and/or transforming the private datasets 245, 250 into one or more validation datasets, and applying the one or more validation datasets to machine learning. and performing a validation workflow that includes operating the algorithm/model and calculating performance of the machine learning algorithm/model based on operation of one or more validation datasets. In one example, combined data 240 may be sent to remote calculation module 275 to be combined with data 245, 250 during calculation.

System monitoring module 280 monitors activity in secure capsule computing service 210. Monitored activities may include behavioral tracking such as algorithm/model acceptance, workflow configuration, and data host hosting as required by each use case and aspect. The data management module 285 may be configured to import data assets, such as private datasets 245, 250, from the data host 255 while maintaining the data assets within the data host 255's existing infrastructure.

III. Techniques for optimizing and/or validating models , in various embodiments, using one or more sources of privacy-protected harmonized data (e.g., harmonized clinical and health data). Alternatively, techniques for optimizing and validating multiple models are provided. The model may be provided by a first entity (eg, an algorithm developer such as a life sciences company) and the dataset may be provided by a second entity (eg, a data host such as an academic medical center). For example, life sciences companies can accelerate the time-to-market of products and services by optimizing or evaluating the performance of their models against one or more sources of privacy-protected, harmonized clinical and health data. There may be an interest in accelerating or optimizing commercial viability. Additionally, academic medical centers may be interested in unlocking the value of their clinical and health data in a manner that maintains data privacy. As shown in Figure 3, to meet the desires of these two entities, a core process 300 for optimizing and/or validating models against clinical and health data is implemented using an artificial intelligence algorithm development platform ( For example, using the platform and system described with respect to FIGS. 1 and 2).

At block 305, a third party algorithm developer (first entity) provides one or more algorithms or models to be optimized and/or validated in a new project. One or more algorithms or models may be developed by an algorithm developer using a proprietary development environment, tools, and seed datasets (eg, training/testing datasets). In some examples, the model includes one or more predictive models. The predictive model may be any algorithm, such as a convolutional neural network (“CNN”), e.g. an inception neural network, a residual neural network (“Resnet”), or a recurrent neural network, e.g. ML models can be included, including but not limited to memory storage ("LSTM") models and gated recurrent unit ("GRU") models. Predictive models are also trained to predict something that cannot be directly measured or will occur in the future, or to make inferences from data (e.g., the model's predictions or the results of an algorithm). It can also be any other suitable ML model. For example, in medical applications, these can be used to determine clinical symptoms, identity, diagnosis, or prognosis from images or video frames, e.g., three-dimensional CNN ("3DCNN"), dynamic time warping ("DTW") techniques, hidden Markov (“HMM”) or a combination of one or more such techniques, such as CNN-HMM or MCNN (Multi-Scale Convolutional Neural Network). Algorithm developers may use the same type of predictive model or different types of predictive models (e.g., ensemble ML techniques) to make predictions or inferences, e.g., clinical symptoms, identity, diagnosis, or prognosis. A seed dataset may be an initial dataset (e.g., private or public clinical or health data) obtained by an algorithm developer for initial training and testing of a model.

At block 310, the algorithm developer provides constraints for optimization and/or validation of one or more models. The constraints may include one or more of (i) training constraints, (ii) data preparation constraints, and (iii) validation constraints. These constraints affect data preparation (e.g., data curation, data transformation, data harmonization, and data annotation), model training, model validation, and reporting, as described in further detail with respect to Figures 4-12. Essentially defining the optimization and/or validation objectives of one or more models, including: As understood, requests that the algorithm developer optimize one or more of the algorithm developer's models using training/testing datasets available from the data host (second entity) If so, the algorithm developer may provide the model, training constraints, and data preparation constraints. In another example, if an algorithm developer requests that one or more of the algorithm developer's models be validated using a validation dataset available from a data host, the algorithm developer , can provide validation constraints and data preparation constraints. In yet other examples, when algorithm developers require to optimize and validate one or more of their models using training/testing/validation datasets available from a data host, Algorithm developers may provide models, training constraints, validation constraints, and data preparation constraints.

In some aspects, training constraints include hyperparameters, regularization criteria, convergence criteria, algorithm termination criteria, training/validation/test data partitions defined for use with one or more algorithms, and training/validation criteria. including, but not limited to, one or more of the test reporting requirements. Model hyper-parameters are constructs that are external to the model and whose values cannot be estimated from the data. Hyperparameters are settings that can be adjusted or optimized to control the behavior of machine learning algorithms and help estimate or learn model parameters. The process of selecting and optimizing hyperparameters is a key aspect of many machine learning solutions. Most machine learning algorithms explicitly define hyperparameters that control different aspects of the model, such as memory and execution cost. Hyperparameters are specified by the algorithm developer and may be set using one or more problem-solving techniques, such as heuristics. However, additional hyperparameters may be defined to adapt the algorithm to specific scenarios. For example, hyperparameters may include the number of hidden units of the model, the learning rate of the model, or the convolution kernel width of the model.

Regularization suppresses or reduces the coefficient estimates toward zero. In other words, this technique prevents learning more complex or flexible models to avoid the risk of overfitting. Regularization significantly reduces model variance without substantially increasing model bias. Therefore, regularization constraints such as the tuning parameter λ used in the regularization technique control the impact on bias and variance. As the value of λ increases, the value of the coefficient decreases and thus the variance decreases. Up to a certain point, this increase in λ is beneficial because it only reduces the variance (thus avoiding overfitting) without losing important properties of the data. However, after a certain value, the model begins to lose important properties, causing the model to become biased and thus undertrained. Accordingly, regularization constraints such as the value of λ may be selected to implement various regularization techniques within the model. In contrast, a convergence criterion is used to check the convergence of a sequence (eg, the convergence of one or more weights after several iterations). The convergence criterion may be implemented in various forms, such as a fixed number of epochs, goal definition, and early stopping, and thus the constraints on the convergence criterion depend on the form or technique that should be used and the number of training iterations that should be performed. , target values, performance values, validation datasets to be used, improvements defined, etc.

Algorithm termination criteria define parameters for determining whether the model has achieved sufficient training. Because algorithm training is an iterative optimization process, the training algorithm may operate multiple times on all or part of the training data, update model parameters, and then re-evaluate model performance. . In general, termination criteria may include an algorithm performance goal, often defined as a minimum performance improvement per iteration or set of iterations, in order to continue processing. In some examples, termination criteria may include a maximum number of iterations of the training model update process, or a maximum number of clock time or computational cycles that should be allocated to training. Other methods for determining when to stop the iterative training process are also envisioned. Training/validation/test data partitioning includes criteria for partitioning data assets into training sets, validation sets, and/or test sets. A training dataset is a set of data used to fit or train a model. A validation dataset is a dataset used to provide an unbiased evaluation of a model fitted or trained on a training dataset while adjusting model parameters. A test dataset is a dataset used to provide a fair evaluation of the final model fitted or trained on the training dataset. The partitioning of these datasets may depend on several factors, including the total number of samples available from the data to be used for training, testing, and/or validating the model. For example, some models require significant data to train, so in this case the algorithm developer may define constraints to optimize for a larger training set. Additionally, models with very few parameters may be easier to verify and tune, so algorithm developers may define constraints that reduce the size of the validation set. However, if the model has many parameters, the algorithm developer may want to define constraints that accommodate a large validation set (although cross-validation may also be considered and included within the constraints).

A training/testing report may include metrics and criteria that an algorithm developer is interested in observing from training, optimizing, and/or testing one or more models. In some examples, metric and criterion constraints are selected to indicate the performance of the model. For example, metrics and metrics such as mean percent error may provide information about bias, variance, and other errors that may occur when completing a model such as vanishing gradients and exploding gradients. Bias is an error in a learning algorithm when the learning algorithm is too weak to learn from the data. In the case of high bias, the learning algorithm is unable to learn relevant details in the data. Therefore, the model performs poorly on training and test data sets. In contrast, variance is the error of a learning algorithm when it tries to overfit from a dataset or tries to fit the training data as closely as possible. In the case of high variance, the algorithm performs poorly on the test dataset, but may perform quite well on the training dataset. Additionally, common error metrics such as average percent error and R2 score do not necessarily indicate model accuracy, so algorithm developers can use additional metrics and You may want to define criteria. For example, if the selected dataset contains time-series data that tends to be temporally correlated and often exhibits significant autocorrelation (if you are evaluating a model in terms of its ability to directly predict values, then the average percentage error Common error metrics such as ^R2 (coefficient of determination) and R2 (coefficient of determination) (both of which indicate falsely high predictive accuracy) require algorithm developers to use one or more additional metrics or criteria. One may wish to monitor autocorrelation.

In some aspects, data preparation constraints include, but are not limited to, one or more of input data requirements and annotation protocol requirements. Input data requirements may include optimization and/or validation selection criteria for data assets to operate on an algorithm or model. Optimization and/or validation selection criteria define the characteristics, data formats, and requirements of input data (e.g., external data) that should be available to the model. Input data characteristics and requirements refer to the characteristics and requirements of the data such that the data can be used to optimize and/or validate the model. For example, an algorithm implemented by a model may require training data that accurately represents the environment in which the model will operate, such as different ethnic or geographic data, in order to create a more generalizable algorithm. In some examples, the characteristics and requirements of the input data include (i) the environment of the model, (ii) the distribution of the example, such as 50% male and 50% female, (iii) data (e.g., image data) and/or measurements. (iv) variance vs. bias, i.e. a model with large variance can easily fit the training data and accommodate complexity, but is sensitive to noise, while bias A model with a high (vi) any combination thereof. For example, the data characteristics and requirements of a model developed to predict the presence of tumors using classification and clustering techniques are may include requirements for three-dimensional imaging data and/or biomarker test data from a mixture of men and women. Format refers not only to the file format of the data (for example, all image data must be in .jpeg file format), but also to the integrity of the record itself. For example, data format constraints may require a standard nomenclature for datasets recognized by the model (e.g., standardized codes, terminology, synonyms covering anatomy, disease, findings, procedures, microorganisms, substances, etc.). terms and definitions).

Annotation protocol requirements may include different types of data annotations to be used in the model. Data annotation is the task of labeling data (e.g., datasets for optimization and/or validation), which can be in any form, such as structured numbers, text, audio, images, or video. . Data annotation is an important stage of data preparation in supervised machine learning. The model learns to recognize repeating patterns within the annotated data. After the algorithm has processed enough annotated data, the algorithm may begin to recognize the same patterns when presented with new unannotated data. There are various types of annotations that can be defined within an annotation protocol based on constraints. For example, semantic annotations may be defined for various concepts within text, such as analytical data, medical notes, or diagnostic codes. Additionally, text categorization and content categorization may be defined to assign predefined categories to documents. For example, sentences or paragraphs can be tagged by topic within a document, or medical publications can be organized by subject matter, such as internal medicine, oncology, hematology, microbiology, and so on. For image and video annotation, bounding boxes, which are virtual boxes drawn on frames of images or videos, may be used. The content of the bounding box may be annotated to help the model recognize the content as a distinct type of object, such as a tumor or a bone fracture. For audio annotation, entity annotation, entity concatenation, and phrase chunking are used to label and define parts of unstructured speech and tag parts of speech with linguistic or grammatical meanings. can be used for.

In some aspects, validation constraints include, but are not limited to, one or more of validation data selection criteria, validation termination criteria, and validation reporting requirements. The validation data selection criteria may include validation data set selection criteria that may include any factors necessary to select an appropriate subset of data for the application under development. For example, in medical applications, cohort selection includes, but is not limited to, clinical cohort criteria, demographic criteria, and data set class balance. In medical algorithm development, a cohort study is a type of medical research used to investigate the causes of disease and establish associations between risk factors and health outcomes in a group of people known as a cohort. Retrospective cohort studies examine existing data and attempt to identify risk factors for a particular condition. In prospective cohort studies, researchers pose questions and form hypotheses about what may cause the disease. Researchers then observe the cohort over a period of time to prove or disprove the hypothesis. Thus, clinical cohort criteria define the group of people for which data should be obtained for a study, the type of study (e.g., retrospective or prospective), the risk factors to which that group may be exposed over a period of time, and the questions/hypotheses to be answered. and associated diseases or conditions and/or other parameters that define the criteria for cohort studies. Demographic criteria define the demographic factors of a group of people for which data should be obtained for a study. Demographic factors may include, for example, age, gender, education level, income level, marital status, occupation, religion, birth rate, mortality rate, average family size, and average age at marriage. Data set class balance defines whether and how data is presented for research. For example, datasets are often unbalanced (eg, many more patients with negative analytical test results compared to patients with positive analytical test results). A simple way to correct unbalanced datasets is to balance them by oversampling instances of the minority class or undersampling instances of the majority class. Therefore, the constraints on data set class balance are: (i) whether the dataset should be balanced at all; and (ii) how the dataset should be balanced, e.g., compared to 80:20. and (iii) how to balance, e.g. oversampling minority classes. Many of these cohort definition considerations in medicine have similarities in other application areas.

Validation exit criteria define whether the model has achieved sufficient validation. A validation report may include metrics and criteria that an algorithm developer is interested in observing from validation of one or more models. In some examples, metric and criterion constraints are selected to indicate the overall performance and/or accuracy of the model. For example, metrics and criteria determine whether there are data errors (e.g., discrepancies between the expected and actual state of the data) within new data batches brought in by the model, and whether there are errors that appear between data batches. whether there is a feature skew between training and testing, whether there is a distribution skew between training and testing, whether there is a mismatch between the expected data and the assumptions made in the training code, the model model accuracy, model precision, model performance, and empirical data to assist in diagnosing model quality problems. In certain aspects, metrics and criteria may provide information about hyperparameters that can be adjusted to enhance model performance.

At block 315, one or more data hosts are welcomed to the platform using the process described in detail with respect to FIG. In one example, a potential data host may be notified of potential opportunities to derive additional value from the data host's data assets (e.g., clinical and health data) in a manner that maintains data privacy. For data hosts interested in serving their data assets through the Platform, the Platform will provide data host compute and storage infrastructure to the data host's infrastructure (e.g., to the data host prior to instantiating the storage infrastructure). provisioning within the existing infrastructure of the Algorithm Developer (the first entity) and completing governance and compliance requirements for publishing data assets through the Platform, and One or more data hosts may be hosted using a process that includes obtaining data assets that are interested in serving. Once adopted, the acquired data assets will be searchable and evaluable by third parties in a manner that maintains data privacy.

At block 320, data assets to be used with the model are identified, obtained, and curated using the process described in detail with respect to FIG. In one example, the data assets defined by the algorithm developer at block 310 are identified, retrieved, and curated by the platform for transformation, annotation, and computation. All data assets remain within the data host's environment and can be organized physically or in cloud-based data structures, or data assets can be logically organized within an existing data storage infrastructure. can. Storage space may be identified to store metadata, intermediate computational steps, configurations and data, and model origins and computational results. In certain examples, datasets, metadata, and provenance data may be stored in persistent storage for future reference and regulatory review.

At block 325, a determination is made as to whether the curated data asset is annotated according to an annotation protocol (eg, the annotation protocol defined at block 310). In one example, the determination may be made by comparing the constraints of the annotation protocol to the annotations currently applied to the curated data asset. If the curated data asset has been annotated according to an annotation protocol, no annotation is necessary and the process proceeds to block 340. If the curated data asset has not been annotated according to the annotation protocol, annotation is required and the process continues at block 330. At block 330, the curated data asset is prepared for annotation as described in detail with respect to FIG. In some examples, an annotation protocol requires that the data asset be converted to a particular format or modified in a particular way for annotation. For example, a video composed of multiple images may need to be separated into individual frames or images for viewing and annotation. The transformation process is typically an intermediate step toward the end goal of harmonization, where the data is in the form required by one or more algorithms of the model(s).

At block 335, once the data asset is prepared for annotation, the data asset is annotated as described in detail with respect to FIG. Each algorithm in the model may require data to be labeled in a particular way. For example, a breast cancer detection/screening system may require that specific lesions be identified and identified. Another example would be gastrointestinal cancer digital pathology where each image may need to be segmented and labeled by the type of tissue present (normal, necrotic, malignant, etc.). In some cases involving text or clinical data, annotation may include applying a label ontology to a selected subset of text and structured data. The annotation is done ly to the data host within the secure capsule computing service. A key principle of the transformation and annotation process is that the platform applies data cleaning and transformation algorithms without requiring the data to be moved outside the technical scope of the data host, while preserving the privacy of the data assets. and to facilitate various processes for improvement.

At block 340, a determination is made as to whether the annotated data asset is harmonized according to the algorithm protocol (eg, training constraints and/or validation constraints defined at block 310). Data harmonization is the process of bringing together datasets of different file formats, naming conventions, and columns into one cohesive dataset. In some examples, the determination may be made by comparing training constraints and/or validation constraints to the alignment of annotated data assets. If the annotated data asset is harmonized according to the algorithmic protocol, no further harmonization is necessary and the process proceeds to block 360. If the annotated data asset has not been harmonized according to the algorithmic protocol, harmonization is required and the process proceeds to block 345. At block 345, the annotated data assets are reconciled as described in detail with respect to FIG. In some examples, algorithmic protocols require data assets to be converted to a particular format or modified in a particular way for the calculation. Data harmonization may be performed to convert data assets to a particular format or to be modified in a particular manner.

At block 350 and block 355, the model is prepared to calculate data assets. The processes performed at blocks 350 and 355 may occur subsequent to, prior to, or in parallel with data asset preparation (eg, transformation and annotation). At optional block 350, the model may be refactored. This process may be done to support special libraries or optimizations to allow the model to work within the platform. For example, special libraries may need to be included in the model code to manipulate homomorphically encrypted data in an efficient manner. These libraries can be incorporated into the model during the refactoring process. However, it should be understood that a refactoring process is not required for all optimization and/or verification algorithms.

At block 355, the model facilitates deployment of software containing algorithms to be validated and/or trained to an external environment (in this case, the computational environment of the data host), and further provides privacy protection for the software deployed within the capsule. be integrated into the secure capsule computing services framework to provide security services to protect both the security of the host computing environment and the security of the host computing environment. The secure capsule computation services framework includes a model serving system provisioned within the platform. The model serving system associates each model with one or more secure deployment capsules, deploys the models in the one or more secure deployment capsules to data hosts, and the model serves one or more secure deployment capsules. may be provided or utilized from within one or more secure capsules for optimization and/or validation via a secure application program interface (API). For example, when a model is deployed to a data host as part of a secure capsule compute service, the platform provides the model with data resources, model parameters, or shared data for training (e.g., in a parent-teacher training paradigm) via an algorithm API. ) can be expanded or output. In exchange, the platform provides computational results, data, and other results of deployed computational components and processes, such as computational monitoring results, trained models, model parameters, or secure capsule computational services, through algorithmic APIs. can receive input containing. The secure deployment capsule allows models to operate in an isolated manner from within the data host's infrastructure while maintaining the privacy of data assets and models (e.g., algorithms). Additionally, data hosts running models within one or more secure deployment capsules may not be able to inspect, copy, or modify the algorithm developer's proprietary software. Additionally, data hosts running models within one or more secure deployment capsules may be aware that the hosted models are malicious and could harm an organization's infrastructure or violate data privacy. be protected from the possibility of

In various aspects, a secure encapsulated computational framework includes computational infrastructure (e.g., one or more servers, one or more computational node, one or more virtual computing devices, etc.). The encryption may be industry standard public/private key encryption (eg, AES 256). This computational infrastructure can operate on the data host's computational infrastructure or can operate on the data host's cloud infrastructure. The cryptographic code is signed by the platform and stored in an archive (eg, a data storage device). Although the platform cannot see the contents of the encrypted code, this establishes a record of the exact algorithms verified by the platform if required by regulatory bodies such as the Food & Drug Association (FDA). be done. A secure capsule computation framework is instantiated on the compute infrastructure, and cryptographic code is placed by an algorithm developer within the secure capsule computation framework. At this point, the encrypted code is decrypted. In one example, the encrypted code may be decrypted by an algorithm developer by passing the correct private key to a secure capsule computing framework. The Secure Capsule Compute Framework is configured to require the server's CPU hardware to be configured such that once started, no one can interact with the contents of the Secure Capsule Compute Framework other than through defined APIs described herein. It may be created by encrypting the contents of memory using a stored private key. Even the operating system itself cannot interact with the contents of the secure capsule computing framework, as it is encrypted with a key that only the CPU hardware can access. This means that even if the operating system is compromised, the contents of the secure capsule computation framework cannot be monitored or read.

At block 360, model optimization and/or validation is performed using the data assets. In some examples, optimization includes initializing an algorithm with predefined or random values of weights and biases and attempting to predict output at those values. In certain examples, the model may have been pre-trained by the algorithm developer, so the algorithm is already initialized with weights and biases. In other examples, predefined or random values for weights and biases may be defined by the algorithm developer at block 310 and loaded into the algorithm at block 360. Data assets and hyperparameters may then be input into the algorithm, inferences or predictions may be calculated, and tests or comparisons may be made to determine how accurately the trained model predicted the output. Optimization is one or more steps of training and/or testing using data assets in an attempt to optimize model performance (e.g., optimizing weights and biases), as described in detail with respect to Figure 9. may include operating an instance. Validation may include running one or more instances of validation with the data asset to attempt to validate the model based on the gold standard label, as described in detail with respect to FIG. 10. At block 365, one or more reports are generated and distributed to the algorithm developer based on the results of block 360. In some examples, the report is generated in accordance with training/test reporting requirements and/or validation reporting requirements defined at block 310.

FIG. 4 illustrates a process 400 for onboarding one or more data hosts to an artificial intelligence algorithm development platform (eg, the platforms and systems described with respect to FIGS. 1 and 2). At block 405, data host compute and storage infrastructure (eg, the secure encapsulated compute services described with respect to FIGS. 1 and 2) is provisioned within the data host's infrastructure. In some instances, this provisioning includes deploying encapsulated algorithms in the infrastructure, deploying physical computing devices with appropriately provisioned hardware and software in the infrastructure, and storing storage (physical data stores or cloud-based storage). ), including deployments on public or private cloud infrastructure accessible through the infrastructure.

At block 410, governance and compliance requirements are completed. In some instances, completion of governance and compliance requirements includes clearance from an Institutional Review Board (IRB). The IRB may be provided as part of the platform or may be a separate entity belonging to the data host. IRBs may be used to review and approve the use of data assets from data hosts for algorithm development, validation, and training purposes, including federated training. In certain instances, certain projects to run on the platform are reviewed as IRB modifications to streamline the review and approval process. Completion of governance and compliance requirements is required for any project and/or completed by the platform under applicable law, such as the Health Insurance Portability and Accountability Act (HIPAA). It may further include compliance review and approval of the platform itself. In some instances, platforms and projects include deployment of encapsulated algorithms on existing infrastructure, deployment on physical equipment with appropriately provisioned hardware and software, deployment on public or private cloud infrastructure, etc. Any or all activities within and associated with the Platform to perform the Platform must comply with applicable law (e.g., be 100% HIPAA compliant). This process step is intended to capture activities by the data host necessary to review and approve the platform's compliance with applicable laws. In some examples, assertion of a predetermined authentication, such as HyTrust authentication, may be sufficient for this process.

Completion of governance and compliance requirements may further include obtaining security certifications. For example, data hosts typically apply a security review process to all newly provisioned hardware and software systems. Although security review details may be defined by the data host and determined separately to obtain security certification, the platform is provisioned to comply with security best practices that can be clearly documented and demonstrated. need to be done. Completion of governance and compliance requirements may further include review and approval of the compliance of any projects being undertaken by the platform and/or the platform itself under additional governance/compliance activities. For example, each data host or the government realm in which the data host is located may have additional governance/compliance activities. platforms, including deployment of encapsulated algorithms on existing infrastructure, deployment of physical equipment with appropriately provisioned hardware and software, deployment on public or dedicated cloud infrastructure, and to execute projects. Some or all activities within and associated with the Platform may be subject to additional governance/compliance activities. This process step is intended to capture activities by data hosts necessary to review and approve platform governance/compliance activities. In some examples, a predetermined authentication assertion may be sufficient for this process.

At block 415, data assets that the data host desires to be subjected to model optimization and/or validation are retrieved. In some examples, data assets are transferred from an existing storage location and format to provisioned storage (physical data stores or cloud-based storage) for use by the platform (one or more datastore (curated). For example, a first dataset of data assets may be identified as being available in a first database of a data host (a second entity), and a second dataset of data assets may be identified as available in a first database of a data host (a second entity), and a second dataset of data assets may be identified as available in a first database of a data host (a second entity). can be identified as available in the database. In this example, retrieving the data asset may include physically izing the first data set and the second data set within the provisioned storage in a manner that maintains data privacy. The provisioned storage may be newly provisioned in step 505 or may be existing storage for which new permissions are defined for the platform. In other examples, data assets are not physically moved to provisioned storage, but instead a collection of logical addresses is moved to existing storage that is accessible from the provisioned storage or platform (the data host's existing infrastructure). physical data stores or cloud-based storage). As data assets are retrieved and stored (physically or logically), data provenance may be documented for use by the platform in project management and data quality assurance. As will be appreciated, data assets are not moved from the data host's infrastructure (which may be newly provisioned storage or existing storage).

Block 415 may be repeated over time to update the platform with new data as the data host collects new data or utilizes the platform for new data assets. These updates may be continuous or bulk, but all changes to the retrieved data assets, including provenance information, must accurately replicate the data used at each stage of each project and trace its provenance. can be recorded to ensure that it can be understood.

Optionally, at block 420, the data asset may be anonymized. In some instances, data assets need to be anonymized before use on the platform. Anonymization is a process used to prevent someone's personal identity from being revealed. For example, data generated during research on human subjects may be anonymized to preserve the privacy of research participants. Strategies for anonymization may include removing or hiding personal identifiers, such as a person's name, and suppressing or generalizing quasi-identifiers, such as date of birth. In certain examples, data assets may be anonymized and appropriate re-identification information may be recorded in a protected form (eg, encrypted).

Optionally, at block 425, the data asset may be obfuscated. In some instances, data assets need to be obfuscated before being used on the platform. Data obfuscation is a process that involves data encryption or tokenization. Data encryption is a security process in which sensitive data is encoded using an algorithm and an encryption key such that only users with the algorithm and correct encryption key can access or decrypt the sensitive data. In some aspects, data is encrypted using either traditional encryption algorithms (eg, RSA) or homomorphic encryption. The choice to encrypt with a particular algorithm may be determined by a combination of regulatory requirements and business needs. For example, a data host may provide extensive access to homomorphically encrypted data for algorithm validation purposes, but highly restricted access to RSA encrypted data for algorithm training. Data tokenization means that sensitive data has no meaningful value if broken and can only be accessed or decrypted by a user with access to a token vault that stores the relationship between the sensitive value and the token. This is a security method in which a random string of characters called a token can be converted. In some aspects, data is tokenized and a token vault is maintained by a data host.

At block 430, data assets may be indexed. Data indexing allows queries to efficiently retrieve data from a database. An index may be associated with a particular table and may consist of one or more keys or values to be searched in the index (eg, the keys may be based on columns or rows of a data table). By comparing query terms to keys in an index, it is possible to find one or more database records with the same value in an efficient manner. In some examples, basic information such as metadata or statistical attributes of data fields is computed as one or more keys and stored in the index. The details of what basic information is collected and how it is exposed for search within the platform will depend on the data type and anticipated use case. In general, this basic information is intended to aid queries by identifying what data may be available on the platform about project and data attributes. The basic information can also be used to inform the platform or end users of data transformation and harmonization methods that may be required to process the data assets. Additionally, the basic information can be used for anomaly detection and general data quality assurance purposes.

FIG. 5 illustrates a platform (e.g., as described with respect to FIG. 1 and FIG. 2) to complete a new project (e.g., optimization and/or validation of the model(s) received in block 305 of FIG. 5 depicts a process 500 for identifying, acquiring, and curating data assets to be used with (enabled platforms and systems). At block 505, a determination is made as to whether the data assets to be used with the model are already available (already hosted) on the platform. The determination may be made by identifying the presence (or lack thereof) of the data asset on the platform. Data assets may be identified by performing queries on data assets retrieved from hosted data hosts (eg, partner academic medical centers) as described with respect to FIG. 4. In some instances, the input data requirements (e.g., input data characteristics, data formats, and requirements for external data available in the model obtained at block 310 of Figure 3) are those that are available to the platform and that meet the stated purpose. May be used as search terms, filters, and/or additional information by the platform to identify data assets that can be used by the model to accomplish the desired results.

The identification process uses the input data requirements as search terms and/or filters to query the data assets (e.g., query a provisioned data store using a data index). Can be done automatically by the platform. Alternatively, this process may be performed using an interactive process, e.g., the algorithm developer provides the platform with search terms and/or filters, and in response, the platform obtains additional information. In order to formulate a problem, the algorithm developer provides additional information, and the platform uses search terms, filters, and/or additional information to query about the data asset (e.g. (e.g., performing queries on a host's database or web crawling to identify data hosts that may have data assets). In both cases, identification uses differential privacy to share information within a data asset by describing patterns of groups within the data asset while keeping private information about individuals within the data asset hidden. It will be done. If the data asset is available on the platform (e.g., the search identifies a data asset that meets the constraints of the query or input data requirements), the process continues to block 510 to create a new project for the data asset. Configure. If the data asset is not available on the platform (e.g., the search does not identify a data asset that satisfies the constraints of the query or input data requirements), the process continues to block 525 where the data asset is available from an existing data host. Determine whether it is possible.

At block 510, a new project is configured for the data asset. In some examples, data host computers and storage infrastructure are provisioned or configured within the data host's infrastructure to process new projects with the identified data assets. In some examples, provisioning or configuration occurs in a manner similar to the process described in block 405 of FIG. For example, provisioning or configuration can include deploying new project-specific encapsulated algorithms in the infrastructure, deploying new project-specific storage (physical data stores or cloud-based storage), and making the infrastructure accessible through the infrastructure. This may include deployment on public or private cloud infrastructure.

At block 515, regulatory approvals (eg, IRB or other data governance processes) are completed and documented. In some instances, regulatory approvals may already exist and simply need to be updated for a new project, or may need to be fully completed for a new project. In one example, regulatory approval is completed in a manner similar to the process described in block 410 of FIG. For example, completing governance and compliance requirements may include establishing an IRB or amending a current IRB, reviewing and approving new projects and/or platforms for compliance under applicable law, obtaining security certifications, and implementing additional governance/compliance requirements. Compliance activities may include reviewing and approving new projects and/or platforms for compliance.

At block 520, data storage is provisioned and data formats are configured for new data assets identified or welcomed on the platform. In some examples, data storage provisioning includes identifying and provisioning new logical data storage locations, as well as creating appropriate data storage and query structures. For example, in some aspects data may be stored in a relational database, no-SQL data store, flat file (eg, JSON), or other structure. Within these access models, data may be organized as a relational data schema (eg, a star schema) or as a set of data frames, among many other possible ways. Storage model decisions can be influenced by data type, algorithm type, and underlying algorithm software, or can be determined by system requirements set by the platform itself. If data collection and aggregation into the platform is ongoing (as in the case of a prospective study), what datasets are used at each step of the training or validation process so that ongoing performance metrics can be properly compared? Additional documents may be stored to accurately identify what has been used. including the allowable total size of the dataset (as it grows) and how ongoing quality assessment will be performed to avoid introducing defective data into the training set; Additional provisioning factors may need to be considered here. Additionally, because this is a new project for an existing data host, the data host compute and storage infrastructure may need to be provisioned or reconfigured, as described with respect to blocks 510 and 515, and regulatory approvals (e.g., IRB or other data governance processes) may need to be completed and documented to explain the intended use of the new data asset.

At block 525, a determination is made as to whether data assets to be used with the model are available from a known or existing data host (not previously hosted). The determination sends a data asset request to a known or existing data host with input data requirements (e.g., input data characteristics, data format, and requirements for external data available in the model obtained in block 310 of Figure 3). This can be done by any platform that supports The data host may be notified of the opportunity and the data asset may be identified by the data host based on the input data requirements. If the data assets are available from known or existing data hosts (e.g., one or more data hosts respond to the request), the process continues to block 510 to configure a new project and Welcoming an asset (in this example, a new data asset is being welcomed, although the known or existing data host may have had other datasets previously welcomed). If the data asset is not available from known or existing data hosts (e.g., one or more data hosts do not respond to the request), the process proceeds to block 530 to search for a new host (e.g., a potential data hosts may be notified of the new project as an opportunity to derive additional value from their data assets in a manner that maintains data privacy (as described with respect to block 315 of FIG. 3).

At block 530, one or more new data hosts are welcomed to the platform. In an example where one or more new data hosts respond to a notification of a new project, one or more new data hosts and their data assets may be welcomed into the platform as described with respect to FIG. 4. If the data assets are available from one or more new data hosts (e.g., one or more data hosts are welcomed), the process continues to block 510 to configure a new project and create new data hosts. welcome assets.

FIG. 6 shows a platform (e.g., as described with respect to FIG. 1 and FIG. 2) for completing a new project (e.g., optimization and/or validation of the model(s) received in block 305 of FIG. 6 shows a process 600 for converting a data asset to be used with an integrated platform and system. The goal of process 600 is to prepare a data asset for annotation, which may require the data to be presented in a particular format or modified for annotation. At block 605, exemplary data may be prepared. In some examples, preparing the exemplar data includes identifying or creating a dataset that captures key attributes for the harmonization process. In certain examples, the preparation includes anonymizing the exemplar data. For example, a data host may identify or create a small representative exemplar dataset (transformer prototype set) for use as a guide for developing data transformation model algorithms for data transformation. The data of the transformer prototype set may be anonymized and provided to algorithm developers for the creation of remote harmonization transformers to perform the harmonization process.

At block 610, a harmonized transformer is created for transforming the data asset based on the current format of the data in the transformer prototype set. A harmonization transformer may be created according to (i) training constraints, (ii) data preparation constraints, and (iii) validation constraints defined in block 310 of FIG. 3. The harmonization transformer is used to finally transform the data host's raw data assets into a format that can be used for annotation. At block 615, the harmonized transformer developed for the transformer prototype set is applied to the raw data assets of the data host submitted to the project. A harmonization transformer is executed on a data asset by a platform within the data host's infrastructure or by the data host to maintain the privacy of the data asset. At block 620, the platform or data host reviews the resulting transformed dataset (prior to annotation) to determine whether the transformations were applied successfully and without violating data privacy requirements. Determine. In some examples, the determining includes identifying gaps in the initial transformer prototype set and/or errors in transformer execution. If there is a failure in the transformation process, the process returns to block 605 to identify new data members of the transformer prototype set. If the data asset is successfully converted, the process proceeds to an annotation process to annotate the converted data asset.

FIG. 7 shows a platform (e.g., described with respect to FIG. 1 and FIG. 2) for completing a new project (e.g., optimization and/or validation of the model(s) received in block 305 of FIG. 7 shows a process 700 for annotating a data asset to be used with a system (platforms and systems). At block 705, the algorithm developer defines an annotation protocol to create gold standard labels for training, testing, and validating the model. An annotation protocol may be defined as described with respect to block 310 of FIG. At block 710, the algorithm developer creates training materials for the annotators. Training materials include videos of annotations made on example data, annotated example data from algorithm developer sources, annotated examples of anonymized data from data host sources, and written flowcharts. , including, but not limited to, automatic scripts and tools available from annotation software. At block 715, annotation infrastructure may be deployed. Deploying the infrastructure may include setting up an annotation interface and backend data storage environment for annotated data. In some examples, the annotation infrastructure is deployed within the data host's infrastructure to maintain privacy of the data assets. This process may include obtaining licenses for sufficient annotator logins, tracking annotator grades, annotations, and other metadata regarding project status and data origin.

At block 720, annotators are welcomed to the platform. For example, annotators can be identified, engaged, and trained. Login credentials may be provided to the annotator along with any computing or access requirements (computer, cloud computing access, display workstation, etc.). Working conditions, contracts, including compensation, HIPAA compliance, etc., can be implemented to ensure that the privacy of data assets is maintained throughout the annotation. At block 725, an annotation project is set up or configured. In one example, a data host may work with an algorithm developer to define project objectives and annotator worklist strategies. The annotation project can then be set up or configured using the project purpose and annotator worklist strategy. For example, project objectives are defined that include key performance indicators (e.g., number of data points to be annotated per day or per session) that are measurable and can be used to evaluate the success of the annotation project. obtain. Additionally, annotation worklist strategies can include splitting data between annotators, including test data (supplied by algorithm developers with known labels), exposing data to multiple annotators (e.g., due to inter-operator variability) and for annotator accuracy monitoring), multiple data exposures to the same annotator (e.g., to monitor intra-operator variability), and other operational considerations. can be defined to account for considerations of

At block 730, annotation of data (eg, transformed data from process 600) is performed according to the project structure defined at block 725 by a defined annotation protocol. At block 735, performance of the annotation process and compliance with data privacy requirements is monitored. The algorithm developer, data host, or a combination thereof may monitor the annotator's performance. In this context, performance may include both work speed and work accuracy (which may be monitored separately at block 740) and may be separated by annotator and data type. For example, the number of annotations per day and the total number of annotations per annotator may be monitored to determine whether annotators are meeting contractual obligations and progressing toward project objectives. Annotation accuracy can be monitored by several means. For example, test data may be provided by the algorithm developer, have known labels, and be presented to the annotator to ensure that the annotator has substantially followed the protocol. In some aspects, the same data can be presented to multiple annotators to evaluate/estimate the annotators' accuracy. As used herein, the terms "substantially," "approximately," and "about" refer to the specified Defined as most, but not necessarily all (and completely inclusive of the specified). In any disclosure aspect, the terms “substantially,” “approximately,” or “about” may be replaced with “within [percentage]” of the specified. , percentages include 0.1 percent, 1 percent, 5 percent, and 10 percent. At block 730, it is envisioned that automatic annotation techniques may be incorporated as well. In some examples, existing automatic annotation techniques can be used for some annotation tasks. In other cases, as data is annotated, models are developed to automate, accelerate, and/or augment the annotation process. Such tools can then be used in future annotation tasks that may be associated with different datasets and different algorithm developer projects.

FIG. 8 shows a platform (e.g., as described with respect to FIG. 1 and FIG. 2) for completing a new project (e.g., optimization and/or validation of the model(s) received in block 305 of FIG. 8 shows a process 800 for harmonizing data assets to be used with a system (platforms and systems). The goal of process 800 is to prepare a data asset for computation by one or more algorithms, which means that the data is presented in a particular format or modified for processing. may be required. At block 805, exemplary data may be prepared. In some examples, preparing the exemplar data includes identifying or creating a dataset that captures key attributes for the harmonization process. In certain examples, the preparation includes anonymizing the exemplar data. For example, a data host may identify or create a small representative exemplar data set (harmonized prototype set) for use as a model for data transformation. The data of the harmonization prototype set may be anonymized and provided to algorithm developers for the creation of remote harmonization transformers to implement the harmonization process. The exemplar data may be the same exemplar dataset (transformer prototype set) used in the transformation process described with respect to Figure 6, and may be a supplemented dataset (e.g., to create a harmonized prototype set). It may be a new data member added to the transformer prototype set) or it may be an entirely new data set for harmonization.

At block 810, a harmonization transformer for harmonization of data assets is created based on the current format of the data in the harmonization prototype set. A harmonization transformer may be created according to (i) training constraints, (ii) data preparation constraints, and (iii) validation constraints defined in block 310 of FIG. 3. The harmonization transformer is used to finally transform the transformed/annotated data assets of the data host into a format that can be used as input to the model. At block 815, the harmonization transformer developed for the harmonization prototype set is applied to the transformed/annotated data assets of the data host submitted to the project. Data transformers are executed on the transformed/annotated data assets by a platform or data host within the data host's infrastructure to maintain the privacy of the data assets. At block 820, the platform or data host reviews the resulting harmonized dataset (prior to model execution) to determine whether the transformations were applied successfully and without violating data privacy requirements. Determine. In some examples, the determining includes identifying gaps in the initial transformer prototype set and/or errors in transformer execution. If there is a failure in the conversion process, the process returns to block 805 to identify new data members of the harmonized prototype set. If the data asset is successfully converted, the process proceeds to an optimization and/or validation process for using the harmonized data asset in the model. Advantageously, these steps to facilitate multi-level harmonization and annotation of data for use in algorithms can be performed without exposing the underlying data to algorithm developers. be.

FIG. 9 shows a process 900 for optimizing one or more models using a platform (eg, the platforms and systems described with respect to FIGS. 1 and 2). At block 905, a federated training workflow is executed to train a dataset (a training dataset split from the data assets) with each instance of the algorithm. In some examples, algorithm training is performed on a training dataset to generate a trained model. The training data set may be hosted by one or more data hosts as described with respect to FIGS. 3 and 4. A federated training workflow takes a training data asset as input, uses parameters to map the features of the training data asset to a target inference, computes a loss or error function, and then uses parameters to minimize the loss or error function. Update the parameters to the learned parameters and output one or more trained instances of the model. For example, a federated training workflow may take a training data asset as input, use model parameters to find patterns in the training data asset that map attributes of the training data asset to target predictions, and use the training gradient of a loss or error function to computes, updates the model parameters to the learned model parameters in response to the training gradient, and outputs a trained instance of the algorithm that uses the learned model parameters to capture the patterns in the training data asset.

Data from the training set is passed by the data host via an API to a secure capsule computation framework, and one or more model responses (e.g., training gradients) are returned to the platform, where the responses are included in the training report. be aggregated. Other than this interaction, no communication from inside the secure capsule computation framework to outside the secure capsule computation framework is allowed, preventing rogue algorithms from communicating private data outside of the secure capsule computation framework. . In some aspects, the federated training workflow includes full model training where algorithm training is fully converged and all hyperparameters are optimized. In other aspects, the federated training workflow performs incremental training. In either case, the results of the training, such as parameters and/or training gradients, are sent to a master algorithm module for integration into a "fully federated model." Results such as parameters and/or training gradients may be encrypted before transmission to ensure privacy of data assets and models.

At block 910, the results, including parameters and/or training gradients for each trained instance of the model, are integrated into a fully federated model. Integrating means aggregating results such as parameters and/or training gradients to obtain aggregated parameters and/or training gradients, and combining learned model parameters of a fully federated model with aggregated parameters and/or training gradients. and updating with a gradient. In some examples, the aggregation is performed using horizontal federated learning, vertical federated learning, federated transfer learning, or any combination thereof.

At block 915, a test workflow is executed in a fully federated model. The test workflow takes test data as input, uses updated learned model parameters to find patterns in the test data, and outputs inferences. Additionally, the performance of the fully federated model in providing inference is computed, and a decision is made regarding whether the fully federated model satisfies the algorithm termination criteria (i.e., the criterion that defines whether the model has achieved sufficient training). It will be done. In some examples, model hyperparameters and model parameters are tested for convergence, error conditions, failure to converge, etc. according to algorithm termination criteria defined by the algorithm developer at block 310 of FIG. If the termination criterion has not been reached, the process returns to block 905 where the updated model (eg, fully federated model) is distributed to each data host for additional training on the training dataset. This iterative process is repeated until the algorithm termination criteria are met. The progress of the algorithm training process may be reported to the algorithm developer during this step, subject to reporting constraints. If the algorithm termination criteria are met, the process continues at block 920.

At block 920, once the iterative algorithm training process meets the algorithm termination criteria, aggregated results, such as aggregated parameters and/or training gradients of the fully federated model, are delivered to the algorithm developer along with reporting of performance metrics according to reporting constraints. obtain. At block 925, it may be determined by the algorithm developer that the optimization process is complete. In certain examples, aggregated results, such as aggregated parameters and/or training gradients, may be sent to each instance of the model when the iterative algorithm training process meets algorithm termination criteria or as requested by the algorithm developer. An update training workflow may be performed on each instance of the model. The update training workflow updates the learned model parameters with aggregated parameters and/or training gradients, and updates the model using the updated learned model parameters to capture patterns in the training and test data assets. Output the instance that was created.

In certain instances, models and datasets used in the training process are maintained for future purposes (eg, regulatory review). For example, once a training activity is complete, the entire secure capsule computation framework and all of its contents may be securely deleted so that there is no risk of continued exposure to transferred data or models. The original encryption code submitted for training will be archived by the platform to ensure that it is always available for inspection by regulatory authorities.

FIG. 10 shows a process 1000 for validating one or more models using a platform (eg, the platforms and systems described with respect to FIGS. 1 and 2). At block 1005, an initial model is obtained. In some examples, the model obtained is an existing trained model obtained from an algorithm developer. An initial model may be obtained as described with respect to FIG. At block 1010, a validation workflow is executed on the validation dataset (validation dataset split from the data asset). In some examples, validation is performed against a validation data set to determine model performance or accuracy. The validation data set may be hosted by one or more data hosts as described with respect to FIGS. 3 and 4. The validation workflow takes the validation dataset as input, uses learned model parameters to find patterns in the validation dataset, and outputs inferences. Data from the validation set is passed by the data host via an API to a secure capsule computation framework, and one or more model responses (e.g., inference) are returned to the platform, where the responses are aggregated into a validation report. be done. Other than this interaction, no communication from inside the secure capsule computation framework to outside the secure capsule computation framework is allowed, preventing rogue algorithms from communicating private data outside of the secure capsule computation framework. .

At block 1015, the performance or accuracy of the model is calculated based on the gold standard label (ie, ground truth) and a determination is made as to whether the model is validated. For example, an algorithm designed to detect breast cancer lesions in mammograms can be validated on a set of mammograms labeled by a medical professional as either containing or not containing lesions. The performance of the algorithm on this set of professionally labeled mammograms is a validation report. In one example, the feature selection, classification, and parameterization of the model is performed according to the validation criteria (i.e., the criteria by which model validation is determined) defined by the algorithm developer in block 310 of Figure 3 (e.g., (using analytics) and ranked. In some examples, determining whether the model has been validated is based on the validation exit criteria defined by the algorithm developer at block 310 of Figure 3 (i.e., defining whether the model has achieved sufficient validation). including determining whether the criteria) have been met.

If the performance of the model has not been verified, the process returns to block 1020 where the model may be fine-tuned (eg, optimizing hyperparameters). Hyperparameter optimization may be performed using any known optimization technique, such as grid search techniques or random search techniques. This iterative process is repeated until the model is validated or until validation termination criteria are met. The progress of the validation process may be reported to the algorithm developer during this step, subject to reporting constraints. If the verification termination criteria are met, the process continues at block 1025. At block 1025, once validation termination criteria are met, the optimized hyperparameters of the validated model may be delivered to the algorithm developer along with reporting of performance metrics according to reporting constraints. In some examples, reporting of performance metrics may be provided as a single validation report of validation of an algorithm or model against a single set of data assets. In other examples, reporting of performance metrics may be provided as a single validation report of algorithm or model validation aggregated from validation against any number of independent data asset sets. Thereafter, the validation process may be determined to be complete by the algorithm developer. In certain instances, models and datasets used in the validation process are maintained for future purposes (e.g., regulatory review). For example, once the validation activity is complete, the entire secure capsule computation framework and all of its contents may be securely deleted with no risk of continued exposure to transferred data or models. The original encryption code submitted for verification will be archived by the platform to ensure that it is always available for inspection by regulatory authorities.

FIG. 11 is a simplified flowchart 1100 illustrating an example of a process for optimizing and/or validating a model using a model development platform and system (e.g., the model development platform and system described with respect to FIGS. 1-10). It is. Process 1100 begins at block 1105, where a model and input data requirements associated with the model are received from an algorithm developer (eg, a first entity). Input data requirements may include optimization and/or validation selection criteria for data assets to operate on the model. Optimization and/or validation selection criteria define the characteristics, format, and requirements for the data asset to work with the model. At block 1110, a data asset is identified as available from a data host based on the data asset's input data requirements (eg, optimization and/or validation selection criteria). Data assets may be identified by performing one or more queries against one or more hosts' data storage structures based on optimization and/or validation selection criteria.

At block 1115, a data host is welcomed (if not previously welcomed). Embracing includes ensuring that the use of data assets in models complies with data privacy requirements. At block 1120, data assets are curated within a data storage structure within the data host's infrastructure. Curating may include selecting a data storage structure from among a plurality of data storage structures and provisioning the data storage structure within an infrastructure of the data host. The choice of data storage structure may be based on the type of algorithm in the model, the type of data in the data asset, the system requirements of the computing device, or a combination thereof. At block 1125, the data asset is prepared in a data storage structure for processing by the model. Preparing the data asset may include applying one or more transformations to the data asset, annotating the data asset, aligning the data asset, or a combination thereof.

At block 1130, the model is integrated into a secure capsule computation framework. A secure capsule computing framework may provide models to data assets within a data storage structure via an application program interface in a secure manner that preserves the privacy of the data assets. At block 1135, the data asset is operated on by the model. In some aspects, operating the data asset with the model includes creating multiple instances of the model and splitting the data asset into a training dataset and one or more test datasets. training multiple instances of the model on a training dataset; integrating results from training each of the multiple instances of the model into a fully federated model; and training one or more test datasets by the fully federated model. and calculating performance of the fully federated model based on the performance of the one or more test datasets. In other aspects, operating the data asset with the model includes: partitioning the data asset in one or more validation datasets; operating the one or more validation datasets with the model; or calculating performance of the model based on operation of a plurality of validation datasets. At block 1140, a report regarding running the model at block 1145 may be provided to the algorithm developer.

FIG. 12 shows an exemplary system and method suitable for use in systems and methods for developing artificial intelligence algorithms by distributing analysis across multiple sources of privacy-preserving harmonized clinical and health data, according to the present disclosure. A computing device 1200 is shown. Exemplary computing device 1200 includes a processor 1205 that communicates with memory 1210 and other components of computing device 1200 using one or more communication buses 1215. Processor 1205 executes processor-executable instructions stored in memory 1210 to develop artificial intelligence algorithms according to different examples, such as some or all of example method 1100 described above with respect to FIG. configured to perform one or more methods. In this example, memory 1210 stores processor-executable instructions that provide inference 1220 using models and data assets and model performance analysis 1225, as described above with respect to FIGS. 1-11. Computing device 1200 also includes one or more user input devices 1230, such as a keyboard, mouse, touch screen, microphone, etc., in this example, for accepting user input. Computing device 1200 also includes a display 1235 for providing visual output to a user, such as a user interface.

Computing device 1000 also includes a communications interface 1240. In some examples, communication interface 1240 includes an area network (“LAN”), a wide area network (“WAN”) such as the Internet, a metropolitan area network (“MAN”), a point-to-point connection, or a peer-to-peer connection. , may enable communication using one or more networks. Communication with other devices may be accomplished using any suitable networking protocol. For example, one suitable networking protocol is Internet Protocol ("IP"), Transmission Control Protocol ("TCP"), User Datagram Protocol ("UDP"), or any of those such as TCP/IP and UDP/IP. may include a combination of.

Although some examples of the methods and systems herein are described in terms of software running on various machines, the methods and systems are also particularly applicable to field applications for performing various methods according to the present disclosure. It may also be implemented as specially configured hardware such as a programmable gate array (FPGA). For example, each example may be implemented as digital electronic circuitry, or as computer hardware, firmware, software, or a combination thereof. In one example, a device may include one or more processors. A processor includes a computer-readable medium such as random access memory (RAM) coupled to the processor. A processor executes computer-executable program instructions stored in memory, such as executing one or more computer programs. Such processors may include microprocessors, digital signal processors (DSPs), algorithm-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and state machines. Such processors may include PLCs, programmable interrupt controllers (PICs), programmable logic devices (PLDs), programmable read-only memories (PROMs), electronic programmable read-only memories (EPROMs or EEPROMs), or other similar devices. may further include a programmable electronic device.

Such a processor may include a medium that may store processor-executable instructions that, when executed by the processor, may cause the processor to perform, as executed or assisted by, a method according to the present disclosure. , may include or be in communication with one or more non-transitory computer-readable media. Examples of non-transitory computer-readable media include electronic, optical, magnetic, or other storage devices that can provide processor-executable instructions to a processor, such as a processor in a web server. but not limited to. Other examples of non-transitory computer-readable media include floppy disks, CD-ROMs, magnetic disks, memory chips, ROMs, RAM, ASICs, preconfigured processors, all optical media, all magnetic tape and other magnetic media; or any other medium that can be read by a computer processor. The described processors and processes may be within one or more structures or distributed across one or more structures. A processor may include code for performing a method (or portion of a method) according to this disclosure.

IV. Further Considerations In the above description, specific details are set forth to provide a thorough understanding of the embodiments. However, it is understood that the embodiments may be practiced without these specific details. For example, circuits may be shown in block diagrams in order to not obscure the aspects in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring aspects.

Implementation of the techniques, blocks, steps and means described above can be done in various ways. For example, these techniques, blocks, steps, and means may be implemented as hardware, software, or a combination thereof. For hardware implementations, the processing unit can be implemented as one or more algorithm-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), or field-programmable gates. - Can be implemented in an array (FPGA), processor, controller, microcontroller, microprocessor, other electronic unit designed to perform the functions described above, and/or combinations thereof.

Note also that aspects can be described as processes that are illustrated as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. A process ends when its operations are complete, but may have additional steps not included in the diagram. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. If the process corresponds to a function, its termination corresponds to the function's return to the calling function or main function.

Further, aspects may be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and/or any combination thereof. When implemented in software, firmware, middleware, scripting languages, and/or microcode, program code or code segments for performing necessary tasks may be stored on a machine-readable medium, such as a storage medium. A code segment or machine-executable instructions may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, script, class, or any combination of instructions, data structures, and/or program statements. can. A code segment can be coupled to another code segment or hardware circuit by passing information, data, arguments, parameters, and/or memory contents. Information, arguments, parameters, data, etc. may be passed, transferred, or transmitted via any suitable means, including memory sharing, message passing, ticket passing, network transmission, and the like.

For a firmware and/or software implementation, the methodologies can be implemented with modules (eg, procedures, functions, and so on) that perform the functions described herein. Any machine-readable medium tangibly embodying instructions may be used in implementing the methodologies described herein. For example, software code may be stored in memory. Memory can be implemented within the processor or external to the processor. As used herein, the term "memory" refers to any type of long-term, short-term, volatile, non-volatile, or other storage medium, regardless of any particular type or number of memories. , or the type of medium on which the storage is stored.

Further, as disclosed herein, the terms "storage medium", "storage" or "memory" refer to read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, May represent one or more memories for storing data, including magnetic disk storage media, optical storage media, flash memory devices, and/or other machine-readable media for storing information. The term "machine-readable medium" refers to a portable or fixed storage device, an optical storage device, a wireless channel, and/or a medium that contains or carries or stores instructions and/or data. including, but not limited to, various other storage media that can be used.

Although the principles of the present disclosure have been described above with reference to particular devices and methods, it should be clearly understood that this description is made by way of example only and not as a limitation on the scope of the disclosure.

Claims (15)

A method including the following steps:
in a data processing system, receiving an algorithm and input data requirements associated with the algorithm, the input data requirements including optimization and/or validation selection criteria for data assets to operate with the algorithm; The optimization and/or validation selection criteria define characteristics, formats, and requirements for the data asset to operate with the algorithm, and the characteristics and requirements of the data asset define the characteristics and requirements for the data asset to operate on the algorithm and/or refers to the characteristics and requirements of data that can be used to validate said algorithm and input data requirements are received from an algorithm developer;
identifying, by the data processing system, the data asset as available from a data host based on the optimization and/or validation selection criteria for the data asset, the data host comprising: said identifying is an entity different from said algorithm developer;
curating, by the data processing system, the data assets in a data storage structure within an infrastructure of the data host;
preparing the data asset in the data storage structure for processing by the algorithm by the data processing system;
provisioning, by the data processing system, a secure capsule computation framework within a computation infrastructure of the data host infrastructure, wherein the secure capsule computation framework deploys the algorithm within the data storage structure; provisioning the data asset in a secure manner that preserves the privacy of the data asset and the algorithm;
integrating, by the data processing system, the algorithm into the secure capsule computing framework, the algorithm developer integrating cryptographic code for operating the algorithm within the secure capsule computing framework; and decrypting, by the algorithm developer, the encrypted code to obtain a decryption code for operating the algorithm; and
operating, by the data processing system, the data asset through the algorithm, the data asset being transferred from the data storage structure to the secure encapsulated computation via one or more secure application program interfaces; passing to said algorithm in a framework; optimizing, validating, or computing inferences by said algorithm using said data asset and said decryption code; and said optimizing, validating, or passing the results of the inference computation to the algorithm developer or the data host via the one or more secure application program interfaces. the characteristics of the data asset and the requirements,
(i) the environment of said algorithm; (ii) the distribution of instances within said data asset ; (iii) the parameters and type of device producing said data asset ; (iv) the variance versus bias; (v) the 2. The method of claim 1 , defined based on tasks implemented by an algorithm, or (vi) any combination thereof. The identifying step uses differential privacy to share information within the data asset by describing patterns of groups within the data asset while keeping private information about individuals within the data asset hidden. carried out,
the curating step includes selecting the data storage structure from among a plurality of data storage structures; and provisioning the data storage structure within the infrastructure of the data host;
the selection of the data storage structure is based on the type of algorithm, the type of data within the data asset, system requirements of the data processing system, or a combination thereof;
The method according to claim 1 . further comprising hosting the data host by the data processing system;
the step of welcoming comprises ensuring that the use of the data asset in the algorithm is compliant with data privacy requirements;
The method according to claim 1. Claims wherein the step of preparing the data asset includes applying one or more transformations to the data asset, annotating the data asset, harmonizing the data asset, or a combination thereof. The method described in Section 1. said operating said data asset through said algorithm;
creating a plurality of instances of the algorithm; splitting the data asset into a training dataset and one or more test datasets; and training the plurality of instances of the algorithm with the training dataset. integrating results from the training of each of the plurality of instances of the algorithm into a fully federated algorithm; running the one or more test data sets through the fully federated algorithm; 2. The method of claim 1, further comprising: calculating performance of the fully federated algorithm based on the operation of one or more test data sets. said operating said data asset through said algorithm;
dividing the data asset in one or more validation datasets; operating the one or more validation datasets through the algorithm; and based on the operation of the one or more validation datasets. 2. The method of claim 1, further comprising: performing a validation workflow comprising: calculating performance of the algorithm using the algorithm. one or more data processors;
when executed on the one or more data processors, the one or more data processors:
an act of receiving an algorithm and input data requirements associated with the algorithm, wherein the input data requirements include optimization and/or validation selection criteria for data assets to operate with the algorithm; or validation selection criteria define characteristics, formats, and requirements for said data asset to operate with said algorithm, and characteristics and requirements of said data asset define said data asset to optimize and/or validate said algorithm. refers to the characteristics and requirements of the data that can be used for the algorithm and input data requirements received from the algorithm developer;
an act of identifying the data asset as available from a data host based on the optimization and/or validation selection criteria for the data asset, the data host being different from the algorithm developer; the identifying action being an entity;
an act of curating the data asset within a data storage structure within an infrastructure of the data host;
an act of preparing the data asset in the data storage structure for processing by the algorithm;
an act of provisioning a secure capsule computing framework within a computing infrastructure of an infrastructure of the data host, wherein the secure capsule computing framework applies the algorithm to the data assets in the data storage structure; the act of provisioning data assets and the algorithm in a secure manner that preserves privacy;
an act of integrating the algorithm into the secure capsule computation framework, the algorithm developer placing cryptographic code within the secure capsule computation framework for operating the algorithm; decrypting the encrypted code to obtain a decryption code for operating the algorithm by the algorithm developer;
an act of operating the data asset through the algorithm within the secure capsule computation framework, the act of operating the data asset from the data storage structure through one or more secure application program interfaces; optimizing, validating, or calculating inferences by the algorithm using the data asset and the decryption code; and , through the one or more secure application program interfaces to the algorithm developer or the data host. and a computer-readable storage medium. the characteristics of the data asset and the requirements,
(i) the environment of said algorithm; (ii) the distribution of instances within said data asset ; (iii) the parameters and type of device producing said data asset ; (iv) the variance versus bias; (v) the 9. The system of claim 8 , defined based on tasks implemented by an algorithm, or (vi) any combination thereof. The act of identifying uses differential privacy to share information within the data asset by describing patterns of groups within the data asset while concealing private information about individuals within the data asset. carried out,
The act of curating includes selecting the data storage structure from among a plurality of data storage structures and provisioning the data storage structure within the infrastructure of the data host;
the selection of the data storage structure is based on the type of algorithm, the type of data within the data asset, the requirements of the system, or a combination thereof;
9. The system of claim 8 . the operations further include welcoming the data host;
the welcoming includes ensuring that use of the data asset in the algorithm is compliant with data privacy requirements;
9. The system of claim 8 . The act of preparing the data asset includes applying one or more transformations to the data asset, annotating the data asset, harmonizing the data asset, or a combination thereof. The system described in Section 8 . the act of operating the data asset through the algorithm;
creating a plurality of instances of the algorithm; splitting the data asset into a training dataset and one or more test datasets; and training the plurality of instances of the algorithm with the training dataset. integrating results from the training of each of the plurality of instances of the algorithm into a fully federated algorithm; operating the one or more test datasets with the fully federated algorithm; 9. The system of claim 8 , further comprising: calculating performance of the fully federated algorithm based on the operation of one or more test data sets. the act of operating the data asset through the algorithm;
dividing the data asset in one or more validation datasets; operating the one or more validation datasets through the algorithm; and based on the operation of the one or more validation datasets. 9. The system of claim 8 , further comprising: performing a validation workflow comprising: calculating the performance of the algorithm using the algorithm. When executed by one or more processors , the system
an act of receiving an algorithm and input data requirements associated with the algorithm, wherein the input data requirements include optimization and/or validation selection criteria for data assets to operate with the algorithm; or validation selection criteria define characteristics, formats, and requirements for said data asset to operate with said algorithm, and characteristics and requirements of said data asset define said data asset to optimize and/or validate said algorithm. refers to the characteristics and requirements of the data that can be used for the algorithm and input data requirements received from the algorithm developer;
an act of identifying the data asset as available from a data host based on the optimization and/or validation selection criteria for the data asset, the data host being different from the algorithm developer; the identifying action being an entity;
an act of curating the data asset within a data storage structure within an infrastructure of the data host;
an act of preparing the data asset in the data storage structure for processing by the algorithm;
an act of provisioning a secure capsule computing framework within a computing infrastructure of an infrastructure of the data host, wherein the secure capsule computing framework applies the algorithm to the data asset within the data storage structure; , the act of provisioning the data asset and the algorithm in a secure manner that preserves privacy;
an act of integrating the algorithm into the secure capsule computation framework, the algorithm developer placing cryptographic code within the secure capsule computation framework for operating the algorithm; decrypting the encrypted code to obtain a decryption code for operating the algorithm by the algorithm developer;
an act of operating the data asset through the algorithm within the secure capsule computation framework, the act of operating the data asset from the data storage structure through one or more secure application program interfaces; optimizing, validating, or calculating inferences by the algorithm using the data asset and the decryption code; and , passing to the algorithm developer or the data host via the one or more secure application program interfaces ; , one or more non-transitory machine-readable storage media .

Images