KR20230005900A - Systems and methods for providing a private multimodal artificial intelligence platform - Google Patents

Abstract

Systems and methods for providing a personal multimodal artificial intelligence platform are disclosed. The method includes dividing the neural network into a first client-side network, a second client-side network and a server-side network and sending the first client-side network to the first client. The first client-side network processes first data from the first client, and the first data has a first type. The method includes forwarding the second client-side network to the second client. The second client-side network processes second data from the second client, and the second data has a second type. The first type and the second type have a common association. Forward and back propagation takes place between client-side networks and heterogeneous data types of other client-side networks and the server-side network to train the neural network.

Description

Systems and methods for providing a private multimodal artificial intelligence platform

priority claim

This application claims priority to U.S. Provisional Application No. 63/020,930, filed on May 6, 2020 (Document No. 213-0104P), the contents of which are incorporated herein by reference. This application is a part of U.S. Application No. 16/828,085, filed on March 24, 2020 (Document No. 213-0100), which claims priority to U.S. Provisional Application No. 62/948,105, filed on December 13, 2019. It is a continuation application, the contents of which are incorporated herein by reference. This application is part of US Application No. 16/828,216, filed on March 24, 2020 (Document No. 213-0101), which claims priority to US Provisional Application No. 62/948,105, filed on December 13, 2019. It is a continuation application, the contents of which are incorporated herein by reference. This application is filed on March 24, 2020, currently filed on February 16, 2021, U.S. Patent No. 10,924,460, which claims priority to U.S. Provisional Application No. 62/948,105, filed on December 13, 2019. It is a continuation-in-part of U.S. Application No. 17/176,530 (213-0102-CON) filed February 16, 2021, which is now a continuation-in-part of filed U.S. Application No. 16/828,354 (213-0102) The content is incorporated herein by reference. [0001] This application claims priority to U.S. Application Serial No. 16/828,420, filed on March 24, 2020 (Document No. 213-0103), which claims priority to U.S. Provisional Application No. 62/948,105, filed on December 13, 2019. It is a continuation-in-part application, the contents of which are incorporated herein by reference.

technical field

This disclosure relates generally to training neural networks and introduces new techniques for training and deploying neural networks or other trained models in a manner that protects training data from various sources from being discoverable.

Existing approaches to training neural networks exist and use either a federated training approach or a centralized training approach. Each of the existing approaches to training neural networks is based on data received from different clients. The process of sharing data in this context can cause data to leak or become discoverable.

For example, deep learning or machine learning in the medical context may require large datasets to produce sufficient accuracy in trained models that are meaningful for diagnoses. Such data may include data for X-rays or MRIs for different patients or categories of patients. One approach to training these models is to pool the raw data, then allow analysts to access a central repository of the pooled data to perform machine learning on large datasets. However, there are ethical and privacy issues with this approach as well as other issues such as single points of failure and archiving requirements.

To illustrate how the foregoing and other advantages and features of the present disclosure may be obtained, a more specific explanation of the principles outlined above will be given with reference to specific embodiments illustrated in the accompanying drawings. With the understanding that these drawings illustrate only exemplary embodiments of the present disclosure and are therefore not to be regarded as limiting of its scope, the principles herein are described in further detail and detail using the accompanying drawings as follows: Explained:
1 illustrates a federated learning model training approach;
Figure 2 illustrates a split learning centralized model training approach;
3 illustrates a split learning peer-to-peer approach;
4 illustrates a federated split learning approach;
5 illustrates an embodiment related to blind learning;
Figure 7 illustrates how blind correlation works across multiple clients;
8 illustrates a method embodiment;
9-9A illustrate a method embodiment;
10-10A illustrate a method embodiment; and
11 illustrates a system embodiment.

introduction

Certain aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently and some of them can be applied in combination as will be apparent to those skilled in the art. In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of embodiments of the application. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting.

The following description provides exemplary embodiments only and is not intended to limit the scope, applicability or configuration of the present disclosure. Rather, the following description of example embodiments will provide those skilled in the art with possible descriptions for implementing the example embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of application as set forth in the appended claims.

brief explanation

What is needed in the art is a method and system combining known approaches to training neural network models that keep the data on which the model was trained private. With the previous approach, training data could be leaked or discovered as part of the training process. The improved approach disclosed herein can also address other issues, such as storage issues, and eliminate single-point of failure issues. This disclosure first discusses known approaches and then introduces new approaches. In one aspect, a particular platform is used to enable federation development or training of neural network models. Use of the disclosed platform to train a model in this manner is disclosed as another embodiment herein. In another embodiment, data is encrypted as it passes between the server and one or more client devices. Various types of federated learning (shown in FIG. 1 ), split learning (shown in FIG. 2 ) and split running peer-to-peer (shown in FIG. 3 ) are disclosed herein. This disclosure provides several new improvements over existing approaches.

Common federated learning involves passing the entire model from the server to the client device for training using client data. This process may involve using multiple clients, each with its own data, for training purposes. This approach is generally performed in a linear and iterative fashion where the entire model is sent along with the data to the first client and then trained on the first client before the entire model is received back to the server for "averaging". The entire updated model is then sent along with the data to the second client for further processing. The updated model is then sent back to the server for further “averaging,” etc. In the split learning approach, the model is split and parts are sent to each client, but there is still an inefficient linear and interactive training process. The segmented learning peer-to-peer approach performs linearly as peer clients share data in a linear process. Improvements are needed to maintain the privacy of data and the efficiency of the training process.

This disclosure describes two major improvements over federated learning and partitioned learning. The first is a federated partitioned learning approach (see FIGS. 4-5) in which client-side processing occurs in parallel and independently of other clients. The second disclosed approach (shown in Figures 6-10) relates to a multimodal artificial intelligence (MMAI) training approach for processing different types of data from different clients.

As mentioned above, the federated partitioned learning approach is disclosed as a variant of the typical federated learning approach above. In this regard, the method includes dividing a neural network into a first portion and a second portion at a server, and forwarding the second portion to a first client and a second client, respectively. Clients can have data (MRI, patient data, bank data for a customer, etc.) and each can receive a portion of a neural network (a certain number of layers of the network up to the truncated layer). The method includes performing the following operations until the threshold is met: (1) concurrently performing forward steps for the second portion at the first client and at the second client to generate data SA1 and SA2; (See Figures 1 to 4); (2) sending SA1 and SA2 from the first client and the second client to the server; (3) calculating loss values for the first client and the second client at the server; (4) calculating an average loss across the first client and the second client at the server; (5) performing backpropagation using the average loss in the server and calculating the gradient; and (6) sending the gradient from the server to the first and second clients. This approach provides an improvement over federated and partitioned learning approaches by allowing client-side (or "data server") processing to operate independently of each other and in parallel. This approach is also different from the split learning peer-to-peer approach. Independent data servers send their activations to the server side, which aggregates, averages or processes the data according to network requirements to obtain the final trained model.

Another aspect of the present disclosure relates to improvements in developing artificial intelligence models in which multiple different modes of data or data types can be used for training. For example, each client may have a different data type. One client may have an X-ray or MRI image and another client may have text describing the patient's health condition. In this regard, the method may include dividing the neural network into a first client-side network, a second client-side network and a server-side network, and forwarding the first client-side network to the first client. The first client-side network is configured to process first data from the first client, the first data having a first type. The first client-side network may include at least one first client-side layer. The method includes forwarding the second client-side network to the second client. The second client-side network is configured to process second data from the second client, the second data having a second type. The second client-side network may include at least one second client-side layer, wherein the first type and the second type have a common association.

The method comprises receiving, at a server-side network, a first activation from training of a first client-side network on first data from a first client; at the server-side network, a second activation on second data from a second client. receiving a second activation from training of the client-side network, training at least one server-side layer of the server-side network based on the first activation and the second activation to generate a gradient, and The step of sending the gradient to the first client-side network and the second client-side network may be further included. In this way, models can be trained using multiple types of data that have a common relationship, such as a single patient or single type or category of patients.

This summary is not intended to identify key or essential features of the claimed subject matter, and is not intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the entire specification of this patent, some or all of the drawings, and appropriate portions of each claim.

The foregoing, along with other features and embodiments, will become more apparent upon reference to the following specification, claims and accompanying drawings.

details

Disclosed herein is a new system, platform, computing environment, cloud environment, marketplace, or any other characterization of a system that will enable an improved approach to training neural networks. In one aspect, the approach is referred to as a federated-split learning approach that combines features of known approaches but provides a training process that maintains privacy for the data used to train the model from the various client devices. This disclosure first discusses the federated learning approach in more detail, and introduces a new federated partitioned learning approach following the partitioned learning approach and the partitioned learning peer-to-peer approach. A multimodal artificial intelligence (MMAI) learning approach on different types of data is also introduced. The new federated segmentation learning approach and the MMAI approach are based on several models, including those mentioned above. The application examines these first approaches in more detail and then introduces two new learning techniques.

united running

1 illustrates a federated learning approach 100 . This is the approach currently used by major companies. The downside to this approach is that it goes "linear", one data provider at a time, rather than parallel. An example of a neural network shown is a fully connected feed forward neural network that is trained using a federated learning approach. The training process in this case involves the server 102 generating the model 104 and sharing the model 106A, 108A and 110A with the respective clients 106, 108, 110 linearly. The clients in turn train individual models 106A, 108A, 110A separately when receiving the models and send the trained model data back to individual server 102 as shown. Server 102 averages the models and creates a new model 104 (aka the trained model) with updated weights. Server 102 sends the new model or weights to individual clients 106, 108, 110 in a linear fashion. The process is repeated several times or until a certain accuracy is achieved.

At each iteration, server 102 averages all participating models to create a trained model B. Thus, the server has a fully trained model 104 at any point in time. The term "global model" refers to a model that is the result of a training process. A global model is a trained entity used for inference tasks. An inference task might be evaluating medical images to classify whether a patient has cancer or a broken bone or other medical condition.

An example of this approach being used, devices such as electronic watches or mobile devices, eg charging at night and connected to a Wi-Fi network, can have their processor used to train neural network models. Thus, client 1 106 could be an Apple watch, client 2 108 could be someone else's iPhone, and so forth. The Siri voice processing service provided by Apple can be used as a model. All devices train the same model, the only difference is that each client trains on local data. The model or data is sent back to server 102 and the server averages the models together. The downside is that individual clients, such as Client 1 (106), can be tricked into sharing something about the data used to train the model. This would be a leak of privacy data and would raise the issues described above. The problem with the federated learning approach is that there is no model privacy because the entire model is passed from client to client. Each client processes the entire model, resulting in high computational costs and high communication overhead since the entire model is sent multiple times. Reconstruction attacks can also make training data vulnerable.

split running

2 illustrates a split learning centralized approach. The model (neural network) 204 is split into two parts: one part (206A, 208A, 210A) resides on a separate client side (206, 208, 210) and is an input layer to the model and optionally a cut layer (cut layer). layer), and another part (B) resides on the server side 202 and often includes an output layer. The split layer S refers to a layer (cutting layer) in which A and B are split. In FIG. 2, SA represents a split layer or data sent from A to B, and SB denotes a split layer sent from B to A.

In one example, the neural network between B 204 and client 1 206 is B portion 204 plus A1 portion 206A with communication of data SB1 206C and SA1 206B to complete the entire neural network. will be. The training process in this model is as follows. Server 202 creates A and B and sends individual models A (206A, 208A, 210A) to individual clients (206, 208, 210). For every client, actions include repeating the following in a linear or iterative fashion in a group of clients until some condition occurs. Individual clients 206, 208 and 210 in turn download the most recent model A from server 202 (note that this step differs from the approach shown in FIGS. 2 and 3). Clients 206, 208, 210, in their individual turns, perform a forward step on model A and pass the output of A (i.e. active only in S or SA1 206B, SA2 208B, SAN 210B) to the required label. are sent to the server 202. Server 202 uses the SAs received from individual clients 206, 208, and 210 to perform a forward step on B. The server 202 calculates a loss function and the server 202 performs backpropagation and computes gradients in the S layer. Server 202 sends S-only gradients (ie, SB1 206C, SB2 208C, SBN 210C) to individual clients 206, 208, 210. This process is performed linearly across the different clients so that actions occur first for client 206, then client 208 for client 210. Clients 206, 208, 210 perform backpropagation using the SB gradients received from server 202, and clients 206, 208, 210 use the updated A (SA1 (206B), SA2 (208B), SAN (210B)) with the server 202.

The horizontal axis of FIG. 2 is the time at which processing occurs in a round-robin manner from client to client.

In one example, network A1 206A on client 1 may include a convolution layer and an activation layer. After processing the data, Client 1 (206) sends the result of that layer (SA1 (206B)) to the next layer of the network at Server 202, which computes backpropagation and the like as described above. The B network iteratively (in a round robin fashion) processes different data from different clients 206, 208, 210. It ultimately arrives at the averaged reflection of the network. It never trains the network on all data from all clients 206, 208, 210 at the same time. It has the advantage that it can process the data faster and when B is built it is averaged over the data. The final algorithm did not check all data. Model B cannot be tricked into revealing its data because it has never been trained on all data.

Split learning in a peer-to-peer environment

3 illustrates a split learning peer-to-peer approach. A model (neural network) is divided into two parts: one part (A) resides on the client side and contains an input layer, and the other part (B) resides on the server side and often contains an output layer. In FIG. 3 , client-side portion A is shown separately as A1 306A in client 306 , A2 308A in client 308 , and AN 310A in client 310 . The split layer S refers to a layer in which A and B are split. In FIG. 3, SA represents a split layer sent from A to B, and SB denotes a split layer sent from B to A.

In one example, the neural network between B and client 1 (306) is part B plus part A1 (306A) with communication of data SB1 (306C) and SA1 (306B) to complete the entire neural network. The training process in this model is as follows. Server 302 creates A and B and sends A to clients 306, 308 and 310. For every client, the process involves repeating the following until some conditions occur. First, the process involves downloading the most recent A from a previous client.

Note that this step differs from the approach shown in other figures. The process then involves performing a forward step on A and sending A's output (i.e., activations only in S) to server 302 with the necessary labels added. Server 302 uses SAs received from individual clients 306, 308, and 310 to perform forward steps for B. Server 302 computes the loss function and performs backpropagation and computes the gradients in S. Server 302 sends S only gradients (ie, SB) to individual clients 306, 308, 310. The client uses the SB gradients received from the server 302 to backpropagate. The client shares the updated A with the server 302.

The peer-to-peer approach usually involves individual clients updating their A models by downloading directly from the last trained client, or more broadly, from a previously trained client. In this regard, the process of training clients may occur in a round robin fashion where clients are trained sequentially. For example, in a peer-to-peer model, if client 1 306 is trained first, then client 2 308 updates the client-side model A2 from server 302 or another trusted server. 308) updates client model A2 by downloading client-side model A1 from client 1 (306). The previously trained model may be the last trained client model or it may be a model from some other client previously trained based on some criterion. For example, client 1 306 and client 2 308 can train separate models. Client 3 (310) may implement an algorithm or process to determine which client-side model update is required and which client-side model to download between client 1 (306) and client 2 (308). Note that the disclosure below implements a multi-model artificial intelligence training process that may be applied herein. Client 1 (306) processes images and its model A1 focuses on image processing, client 2 (308) processes text and its model A2 focuses on text processing, and client 3 (310) processes images. If so, the algorithm or process may, in a peer-to-peer environment, download the client-side model A1 to client 3 (310) as an update.

In one scenario, there is not enough information from split learning to achieve proper training of the neural network. This model assumes that a good training approach might be for A and B to be simply stacked (A and B) and aggregated at server 302 as plain text.

federation split running

4 illustrates an improvement over the training neural networks disclosed herein. This improvement can be characterized as a federated segmentation learning approach and addresses some of the deficiencies of the approaches described above. Figure 4 introduces a parallel processing approach. Model training occurs at a faster rate than the other models described above due to parallel and independent processing.

The federated partitioned learning approach does not perform the round robin processing described above. Server 402 splits the network at “split layers,” which are user parameters inserted into network definition codes. The "top portion" of the network is maintained at server 402 and the "bottom portion" is forwarded to individual data providers or clients 406, 408, 410 (clients and data providers The terms are used interchangeably herein). Training starts at the lowest network layer, which is the layer closest to the data. Each layer reads the data (from the first layer) or the output of the previous layer (all other layers).

Layers are based on any valid network architecture command (convolutions, dropouts, batch normalization, flattened layers, etc.) and an activation function (relu, tanh, etc.) referred to as "activations") can be calculated. When the last layer on the data side (406, 408, 410) computes the appropriate activations (i.e., outputs), that output is routed to the first layer on the "other side of the split" - the first layer on the server side (402). do.

The next approach involves splitting the model as before. The A model is split into two parts: (A) contains an input layer on the client side and (B) often contains an output layer on the server side. (S) is a split layer. Clients or data providers 406, 408, 410 run independently and send back a response if one is available. Code in server 402 processes the data and sends the output back to SBs 406C, 408C, and 410C to all clients identically.

An exemplary training process is as follows. Server 402 creates A and B and sends portion A 406A, 408A, 410A to clients 406, 408, 410. The following steps are repeated until a condition (eg accuracy) is met. All clients 406, 408, 410 simultaneously perform the forward step for A. Up to this point, all calculations for clients 406, 408, 410 are being performed on independent servers and there is no dependency from one data server to another. This approach highlights one of the innovations disclosed herein. All of these calculations by clients/data providers 406, 408, 410 can all operate simultaneously and in parallel. This contrasts with the linear or "round robin" approach discussed above.

Clients 406, 408, 410 each execute portions A (406A, 408A, 410A) of the neural network and generate individual outputs of A (i.e., SAs 406B, 408B, 410B) and send the output to server 402. send to Server 402 receives three different 'versions' of activations (one from each of SA1, SA2 and SA3). At this point, server 402 "properly" handles those activations, which may mean that server 402 performs different actions on a case-by-case basis. For example, server 402 calculates a loss value for each client 406, 408, 410 and server 402 calculates an average loss across all clients. Server 402 performs backpropagation using the average loss and computes the gradients in S. Server 402 sends gradients to all clients 406, 408, 410 at S (ie, SB 406C, 408C, 410C).

In other words, training on the server side 402 proceeds very similarly to that described above. When the first layer on the server side 402 is “complete” (by averaging or aggregating received from data providers 406, 408, 410) forward propagation occurs until it reaches the “top” of the network. A further innovation described in this disclosure is managing activations coming from data providers 406, 408, 410 and how they are averaged, aggregated or otherwise processed. When the system reaches the top of the model, server 402 computes the gradients needed for backpropagation and sends them back down through the split networks as shown in FIG.

As noted above, the handling and management of activations by server 402 may vary depending on different factors. For example, it is assumed that all three data providers 406, 408, and 410 provide the same data (X-rays). In this case, the data is combined horizontally, which can conceptually mean that data is "stacked" on top of one file on top of another. In this case, the activations that occur are likely averaged. The "average of each activation" is then forwarded to the "top half" of the network.

In other cases, the data may be “vertically” stacked, so client 1 (406) has the first 40 columns of data (e.g. blood tests), and client 2 (408) has the data with the next 60 columns of data (eg, electronic health record containing data such as age, weight, etc.) etc.) In this example, the three clients can be considered to set a combined "record" of 200 columns (aggregated vertically across the page). In this case, the activations are "vertically combined". This and other approaches to combining data can be implemented. The multi-model artificial intelligence model described in more detail below is based on the concept just described regarding combining activations vertically. Note that , more details on this concept will be provided below.

As mentioned above, clients 406, 408 and 410 run in parallel in this embodiment. This reduces the time it takes to train the model as all processing is done in parallel. Also, this data is passed through a specific platform. The applications incorporated above provide examples of specific platforms that can be used to deliver data disclosed herein. This will be discussed further below.

The global model of federated partitioned learning can be aggregated as: When training is complete, the system uses the following approach to aggregate a global model to be used for inference tasks. In the first approach, the server selects one of the models, Ai, and integrates it with the corresponding model B to form a global model. Selection of Ai may be achieved using one of the following methods. For example, random selection may be used when the server randomly selects the model Ai of any client 406 , 408 , 410 . This random selection may be influenced by other factors such as the clients currently available online, the types of data each client processes (text data, image data, time data), or the transmission speed or network latency between the two entities. there is. The server then builds a global model by stacking both Ai and B parts.

In another example, weighted client selection may be used. For this selection criterion, server 402 assigns a weight (i.e., numerical value) to each client that reflects their importance based on their data, computational capabilities, and other valuable assets that they own and contribute during the training process. assign For example, a particular set of models (eg, data for a particular language, data related to an image type, data related to a set of patients, or data from a particular country or region) may receive a lot of weight in model development. Thus, if a country is selected, client devices in that country may be given more weight than clients in other countries. For example, Japan-based client devices may be used for 80% of the model data. Australia could be 10% and Canada the other 10%. As another example, more weight may be given to data from specific clinics related to flu or COVID outbreaks. In another example, data types can also be given more weight. 70% of the model may be image data, 20% text data, and 10% time data.

Another model could be accuracy based selection. In this case, server 402 may test the accuracy generated from each client model Ai and then select the model that produces the "best" accuracy. “Best” may be identified by stakeholders, such as through a machine learning approach. These are all models of the first approach.

A second approach may be a case of aggregating a global model by averaging models A i {1, N} of all clients. Each client first encrypts its model using homomorphic encryption and then sends the encrypted Ai′ data to the server 402 . Server 402 adds all the encrypted models, decrypts the additional results, and calculates the average. The averaged A is then stacked with B to create a global model. One approach may be the default approach and optional approaches may be provided. The decryption processes and the averaging process may also be spread between different servers, for example, with one process occurring on the client side and another process performed by the server 402 to achieve a global model.

Approaches may vary through the development of the model. For example, training can be adjusted such that a model is started training using a default approach and then a weighted approach is used to complete model training.

An example of the method is shown in Fig. 5, in which the server divides the neural network into a first part and a second part (502), sends the second part separately to the first client and the second client (504), and It may include performing the following actions until the threshold is met:

(1) Simultaneously performing the forward step for the second part in the first client and the second client to generate data SA1 and SA2;

(2) sending SA1 and SA2 from the first client and the second client to the server;

(3) calculating loss values for the first client and the second client at the server;

(4) calculating an average loss across the first client and the second client at the server;

(5) performing backpropagation using average loss and calculating gradients in the server; and

(6) Sending the gradient from the server to the first client and the second client (506).

A computing device or devices that perform the above operations may also be addressed as a computer readable storage device storing instructions that, when executed, cause a processor to perform these operations. The actions may be performed in any order and a method may include one or more actions.

In another aspect of this disclosure, the platforms described in Patent Applications incorporated above may provide a basis for communicating data back and forth in any federated models. For example, each of the clients and/or server may need to log on to the platform or one of the platform versions referenced in the applications incorporated herein. Accordingly, delivery of this functionality via a platform or exchange configured as disclosed in these applications is also treated as an aspect of the present disclosure.

In another aspect, the customer can select the SA, SB lines (vectors and numbers) representing the weights that need to be propagated. If a client wants to lock data about it without the server knowing about it, it can homomorphically encrypt that data. An encryption process (which may include any encryption process) may be used in any of the approaches described above.

The above integrated patent applications provide example platforms on which client devices and/or servers can log in or require log in to perform the federated partitioned learning approach disclosed herein.

Note that in one aspect, the steps disclosed herein may be performed by a “system”. A system may include a server and one or more clients together, or it may be a function performed by a server. A system may also be clients or groups of clients, such as clients or groups of clients in a particular geographic area, that perform the client-based functions disclosed herein. In one aspect, a “server” can also be a computing device (physical or virtual) on the server side and a computing device (physical or virtual) on the client side. In one example, the server may be on the client side and may receive the backpropagation output of individual client side models Ai and synchronize the client side global model in training rounds.

Accordingly, each of the server-side system and the client-side system may perform any one or more of the operations disclosed herein. Claims may be included reciting steps occurring in the context of any device disclosed herein. For example, the steps of transmitting, computing and receiving data may be claimed from the perspective of a server device, a client device or a group of client devices depending on which embodiment is being addressed. All such communications from the perspective of an individual component or device may fall within the scope of the particular embodiment focused on that device.

In another aspect, the system may include the platform disclosed in the patent applications incorporated by reference and may also perform steps in concert with the concepts disclosed above. Accordingly, the platform used to provide the federated segmentation learning process described herein is also an embodiment of the present disclosure and the use of that platform to train models in a manner that maintains the privacy of data as described herein and Associated steps may be listed.

In general, training of neural networks is performed on similar data types. For example, a neural network trained to identify cancer by receiving patient images or kidneys is trained on images of cancerous and non-cancerous kidneys. The following describes a new approach to training that uses different types of training data together to train a neural network, using the federated partitioned learning approaches disclosed herein.

A multi-model artificial intelligence approach

As mentioned above, the MMAI innovation is based on the idea of “vertical aggregation” described in the example of federated segmentation learning. An example involves all three clients 406, 408, 410 providing the same type of data-images (for stacking) or vertically combined tabular data. When the inventors were considering the vertical aggregation concept, they realized that this could be done with other types of data. For example, client 1 may provide images, client 2 may provide blood tests, and client 3 may provide text notes to doctors. An important difference is that all these data types require different network architectures. In this case, the developers of the system cannot define one network and then have the server "split" it. So, part of the solution is to have users define a "before splitting" network for each data provider, then define the network and aggregation technology at the server. This approach is illustrated in Figures 6-10.

6 illustrates a multimodal artificial intelligence (MMAI) platform or machine learning (ML) platform 600 . The MMAI approach reduces the computational requirements and communication overhead of other approaches. Also, training is much faster and the process maintains a much higher privacy in the data, including the fact that the model is also kept private.

The MMAI platform 600 applies AI/ML techniques to multiple data types in one large AI model. In general, different AI network architectures are required for different types of data to produce accurate results. For example, images usually require special filters (convolutions), while text or speech require other "time-series-like" processing, and tabular data often works best in ML or feed-forward architectures. do. The issue is that images are best understood when looking at all the pixels together and "convolved" in various ways, whereas speech is best understood in the context of before and after a particular sound (i.e. in a manner similar to time series data), etc. to be. Because of these processing differences, today's "state of the art" systems typically process one type of data (ie images, text, audio, tabular, etc.).

Most AI researchers recognize that "next-generation" breakthroughs in accuracy can be achieved by adding more unique data to their models. This is essentially like giving the model more data, giving it more context to discover interesting differences in cases. An example of this concept is a model that diagnoses atrial fibrillation (A-fib) by examining electrocardiogram (ECG) data. The model can reach a certain level of accuracy with ECG data alone, but the model becomes even more accurate when researchers add age, gender, height, and weight to the ECG data. The increase in accuracy is due to four additional data types that can help the model better understand what it might look like, such as "equivalent" ECGs. You can further refine your data by adding four items, or characteristics of your data.

The MMAI platform 600 shown in FIG. 6 introduces a new generation cryptographic toolset to improve training and protection of personal data. The MMAI platform 600 provides more data to the model than is normally used to train AI/ML models and augment the data. This approach adds significant amounts of data by combining various data types (eg, images and tabular data).

6 illustrates a first external source 602 of data depicted as a Wells Fargo bank. The Wells Fargo data 602a is encrypted 602b and the package of encrypted data 602c is sent to the personal AI infrastructure 603. A second external source 604 of data is shown as Citibank. Citibank data 604a is encrypted 604b and the encrypted data 604c package is sent to the personal AI infrastructure 603. A third external source 606 of data is shown as Bank of America. The Bank of America data 606a is encrypted 606b and the encrypted data 606c package is sent to the private AI infrastructure 603. The AI infrastructure 603 includes a first module 608 that personally searches, selects and pre-processes all data 610 from disparate sources 602 , 604 , 606 . In this example, all sources are identified as banks, but have different structures, and individual data may also be heterogeneous. Of course, not all external sources of data 602, 604, 606 need be of the same type, ie banks. The use of banks is just an example. External sources 602, 604, 606 may be, for example, hospitals, clinics, universities, and the like. The basic idea is that data types can be different from various external sources (602, 604, 606).

Personal AI infrastructure 603 may include components that personally search for, select, and pre-process relevant features from all data 602c, 604c, and 606c it receives for training. Feature 612 represents a subset of data 610 that may result from the processing of components in personal AI infrastructure 603 . In operations 614 and 616, the AI infrastructure 603 privately trains new deep and statistical models on the selected data 612 and in operation 618 images, video, text and/or other data types. for any personal and sensitive data that may contain The AI infrastructure 603 may then sell or grant access to the new models presented in operation 620 .

7 illustrates another variation on the split learning technique 700 . This approach provides low computing requirements and low communication overhead to improve training of models by using a blind correlation process for training based on heterogeneous types of data. Based on the A-fib model example above, another source of more data for the model is the inclusion of chest X-rays for each case the model considers. Unfortunately, the general processing of X-ray images does not correspond to the general processing of tabular ECG data. With a bit of minor engineering added, this incompatibility can be fixed using the split-federation learning tool described above. That is, new instructions can be provided to the tool so that different data types can be processed in the existing pipeline.

In this case, rather than "automatic" segmentation of the network architecture, this transformation of the idea allows the network designer (i.e., the data scientist developing the algorithm) to specify the specific network components desired for each type of data. Each data type requires layers of the network architecture associated with that data type (ie convolutional layers for images, recursive layers/long-term short-term memory layers for speech, feed-forward layers for tabular data, etc.). These disparate layers, specific to the type of data in question, are designated to run on the "data server" side (almost like independent networks per se). The last layer of each "independent network" (by data type) sends the activations "through the split" to the "server side". The algorithm server side has one consistent "network" that properly handles incoming activations (from the data server side). In some ways this approach is similar to how an "ensemble of networks" (on the data server side) is aggregated into one final network on the algorithm server side (ultimately generating a final "answer" from the "ensemble" of networks).

Segmented learning is a collaborative deep learning technique, in which a deep learning network or neural network (NN) can be split into two parts, a client-side network A and a server-side network B, as discussed above. A NN includes weights, biases, and hyperparameters. In Figure 7, clients 702, 704, 706 where the data resides are only committing to the client-side portion of the network and server 710 is only committing to the server-side portion of the network. The client-side and server-side parts collectively form the overall network NN.

Training of the network is performed by a series of distributed training processes. Forward propagation and back propagation can happen as follows. Using the raw data, a client (assuming client 702 ) trains the client-side network 702A to a specific layer of the network, which may be referred to as a truncated layer or segmentation layer, and sends activations of the truncated layer to server 710 . do. Server 710 trains the remaining layers of the NN with activations received from client 702 . This completes a single forward propagation step. A similar process occurs in parallel for the second client 704 and its data sent to the client-side network 704A and server 710 and the activations generated. A further similar process occurs in parallel with the third client 706 and its data being sent to its client-side network 706A and server 710 and the activations generated.

Next, server 710 performs backpropagation up to the truncation layer and sends gradients of activations to individual clients 702, 704, 706. Using the gradients, each individual client 702, 704, 706 performs backpropagation on the rest of the network 702A, 704A, 706A. This completes a single pass of backpropagation between clients 702, 704, 706 and server 710.

This process of forward propagation and back propagation continues until the network is trained with all available clients 702, 704, 706 and convergence is reached. It is assumed that the architectural configurations in split learning are performed by a trusted party with direct access to the main server 710 . This authorized party selects the ML model (application-based) and network segmentation (finding the cutting layer) at the start of the run.

As mentioned above, the concept introduced in this disclosure relates to clients 702, 704, 706 each providing different types of data but also having a common association with the different types of data. Thus, the selection of the machine learning model may be based on the types of data processed on the client side, and the process of finding the truncation layer may vary depending on the types of data or the heterogeneity of different types of data. For example, for data types that are widely disparate across clients 702, 704, 706, the truncation layer can be chosen to have more or fewer layers in client-side networks 702A, 704A, 706A. there is. In another aspect, the number of layers before the cutting layer or segmentation layer may vary depending on the clients. Client 702 can process images and requires 8 layers before the cut layer, while client 704 can process text and only needs 4 layers before the cut layer. In this regard, as long as the vectors, activations or activation layer of the truncation layer are consistent across different clients 702, 704, 706 with different types of data, client-side networks 702A, 704A, 706A ) need not be the same number of layers.

Synchronization of the running process with multiple clients 702, 704, 706 may be performed in a centralized mode or peer-to-peer mode. In centralized mode, clients 702, 704, and 706, before starting training with server 710, have a trusted third-party server 710 that maintains an updated client-side model uploaded by the last trained client. ) to update the client-side models 702A, 704A, 706A. On the other hand, in peer-to-peer mode, clients 702, 704, and 706 update the client-side model by downloading directly from the last trained client. As mentioned above, previously trained models may have data type similarities with the current client that needs to update that model. For example, similarity may be based on data that is images, text data, audio data, video data, temporal data, and the like. Thus, it can intelligently select a previously trained client model to use for downloading from peers. Processing by server 710 may also in some cases be split between some processing on the server side and other processing on the federated server on the client side.

As introduced above, Client 1 702, Client 2 704 and Client 3 706 may have different data types. Server 710 creates two parts of the network and sends one part (702A, 704A, 706A) to all clients (702, 704, 706). The system repeats certain steps until a correctness condition or other condition is met, for example all clients send data to their part of the network and send output to server 710. Server 710 calculates a loss value for each client and an average loss across all clients. Server 710 may update its model using the weighted average of the gradients it computes during backpropagation and sends the gradients back to all clients 702, 704, 706. Clients 702, 704, 706 receive the gradients from server 710 and each client 702, 704, 706 performs backpropagation in the client-side network 702A, 704A, 706A and returns to each client-side network ( 702A, 704A, 706A) compute individual gradients. Individual gradients from client-side networks 702A, 704A, 706A may be sent back to server 710, which inverts the averaging of client-side updates and sends global results back to all clients 702, 704, 706. there is.

Server 710 functionality may also be divided into several servers, each performing different operations (such as updating a model by one server and averaging local client updates by another server, each located in different regions). Note that you can In the case of FIG. 7 , clients 702 , 704 , and 706 all process disparate types of data that may or may not normally be processed to develop an AI model.

For example purposes, the A-fib model above may be used to illustrate the process. Client 1 (702) can have electrocardiogram data, client 2 (704) can have X-ray data, and client 3 (706) can have genetic data. For example, client 1 702 may be a hospital, client 2 704 may be a medical diagnostics imaging company and client 3 706 may be a bank or financial institution in the manner depicted in FIG. 6 . One of the clients may have time-based data, such as incremental information about a patient regarding weekly hospital visits for checkups.

The approach shown in FIG. 7 allows the system to send new user instructions that allow the user to fetch different data types before a segmentation or truncation layer or with "correct" processing as shown in the blind decorrelation block 708. Illustrates how this can be implemented. Each part of the model can be independent and operate independently. In one aspect, the processing performed by blind correlation block 708 will result in an activation layer or activations being sent to server 710 . This approach is similar to the approach described above with the addition of differences in data types between clients 702, 704 and 706.

Server 710 will combine these activation layers in one of a number of ways. Server 710 may average them (also described above), but may also concatenate them into one long activation layer. In another aspect, server 710 may apply any mathematical function to achieve a desired combination of activation layers. Server 710 may then further process the combined activation layers using any suitable network architecture. In one aspect, the client-side server may receive the gradients, average the gradients to create a global model of the various clients 702, 704, 706 and send the global model to server 710 for subsequent or further processing. there is.

The idea illustrated in Figures 6 and 7 represents the extension and application of the segmentation federated learning tool set and provides a platform of off-the-shelf tools for bringing disparate data types together into a superset AI model. The processing may all be conducted privately and the offer may be placed on the market as described in the incorporated patent applications referenced above.

Not only can the system combine different data types, the system can also combine different AI/ML techniques. For example, client 1 702 can be a CNN (convolutional neural network), client 2 704 can be an ML routine (i.e. XGBoost), and client 3 706 can also apply other techniques. . Different AI/ML technologies are different in this regard, but if the result data of the truncation layer is consistent and properly structured, forward and back propagation can occur and models can be trained.

To help those skilled in the art understand how the MMAI approach works, the following are examples of actual commands by data type coming from three data providers 702, 704, 706. This code uses Python numbering conventions, so it starts with builder0 (tabular data from data provider 1 (702)). In this example, builder1 is for CT scan or image data. Commands are similar for X-ray, MRI and/or other imaging. Builder2 (from data provider 704) is text data. Note the "lstm" command, short for "long/short memory". The "Server" builder commands define a network that aggregates the other three at the "top" on the opposite side of the split.

builder0 = tb.NetworkBuilder()

builder0.add_dense_layer(100, 120)

builder0.add_relu()

builder0.add_dense_layer(120, 160)

builder0.add_relu()

builder0.add_dropout(0.25)

builder0.add_dense_layer(160, 200)

builder0.add_relu()

builder0.add_split()

builder1 = tb.NetworkBuilder()

builder1.add_conv2d_layer(1, 32, 3, 1)

builder1.add_batchnorm2d(32)

builder1.add_relu()

builder1.add_max_pool2d_layer(2, 2)

builder1.add_conv2d_layer(32, 64, 3, 1)

builder1.add_batchnorm2d(64)

builder1.add_relu()

builder1.add_max_pool2d_layer(2, 2)

builder1.add_flatten_layer()

builder1.add_split()

builder2 = tb.NetworkBuilder()

builder2.add_lstm_layer(39, 100, batch_first=True)

builder2.add_dense_layer(100, 39)

builder2.add_split()

server_builder = tb.NetworkBuilder()

server_builder.add_dense_layer(60000, 8000),

server_builder.add_relu()

server_builder.add_dense_layer(8000, 1000),

server_builder.add_relu()

server_builder.add_dense_layer(1000, 128),

server_builder.add_relu()

server_builder.add_dense_layer(128, 1)

8 illustrates an example method 800 for presenting MMAI concepts from the perspective of clients. The method includes receiving a first set of data from a first data source, a first data set (802) having a first data type, training a first client-side network on the first data set and generating first activations. ( 804 ) receiving a second data set from a second data source, a second data set ( 806 ) having a second data type and training a second client-side network on the second data set and Generating 808 activations.

The method comprises sending first activations and second activations to a server-side network, wherein the server-side network is trained (810) based on the first activations and second activations to generate gradients (810). )-, and receiving the gradient in the first client-side network and the second client-side network 812 . The first data type and the second data type may be different data types such as one being image based and the other being textual or time based as in speech.

9 illustrates an exemplary method 900 from the perspective of a server 710 and one or more clients 702 , 704 , 706 . The method comprises dividing a neural network into a first client-side network, a second client-side network and a server-side network 902, forwarding the first client-side network to a first client, wherein the first client-side network comprises a first configured to process first data from a client, wherein the first data is of a first type and the first client-side network comprises at least one first client-side layer (904); and a second client-side network. sending to a second client, wherein the second client-side network is configured to process second data from the second client, the second data has a second type, and the second client-side network includes at least one It may include a second client-side layer, where the first type and the second type have a common association (906).

The method includes training a first client-side network on first data from a first client and generating first activations (908), sending (910) first activations from the first client-side network to a server-side network. ), train the second client-side network on second data from the second client, generate second activations (912), send second activations from the second client-side network to the server-side network (914 ), training (916) at least one server-side layer of the server-side network based on the first activations and the second activations to generate gradients and from the server-side network the first client-side network and the second client-side layer. A step 918 of transmitting the gradients to the side network may be further included.

A common association between disparate types of data may include at least one of a device, person, consumer, patient, business, concept, medical condition, group of people, process, product, and/or service. Any concept, device or person can be a common association or subject for the various disparate types of data provided from different clients and processed by different independent client-side networks down to the truncation or segmentation layer. The server-side network may include a global machine learning model. A neural network can include weights, biases and hyperparameters. Hyperparameters are generally related to parameters whose values are used to control the learning process, such as topological parameters or the size of a neural network. For example, running rate, mini-batch size, number of client-side layers, or any parameter related to process control that may affect or relate to different data types may represent a hyperparameter.

Each of the at least one first client-side layer and the at least one second client-side layer may include the same number of layers or different numbers of layers. Because they operate independently, client-side networks can have different numbers of layers as long as they process the data to generate appropriately formatted activations or vectors to pass to the server-side network for further training. A truncation layer may exist between the server-side network and the first client-side network and the second client-side network.

10 illustrates an example method 1000 from the perspective of a server 710 . The method comprises dividing a neural network into a first client-side network, a second client-side network and a server-side network 1002, forwarding the first client-side network to a first client, wherein the first client-side network comprises a first configured to process first data from a client, wherein the first data is of a first type, wherein the first client-side network comprises at least one first client-side layer (1004); sending to a second client, wherein the second client-side network is configured to process second data from the second client, the second data has a second type, and wherein the second client-side network has at least one may include a second client-side layer of , where the first type and the second type have a common association (1006).

The method comprises receiving ( 1008 ) first activations from training of the first client-side network on first data from a first client, at the server-side network, second data from a second client, at the server-side network. Receiving (1010) second activations from training of a second client-side network for , training at least one server-side layer of the server-side network based on the first activations and the second activations to generate gradients. It may further include sending 1012 the gradients from the server-side network to the first client-side network and the second client-side network 1014 .

In each case, part of the process of server 710 in terms of training may be performed by server 710 and other parts, such as averaging of values for various clients, may be performed at the client site, at a separate location or at multiple clients. Note that this may be performed by different servers (not shown) that may span .

This approach makes it harder for the system to take a trained model as a result, when the system splits the neural network on blind correlations 708, breaks it and applies a training inference attack in a new way to the federated split learning toolset. enables the use of The system can split the neural network in half (or into two parts), and in the manner described above, all that is exchanged in the neural network parts 702A, 704A, 706A is a string or array of numbers, also described as activation layer numbers. . Since these are just numbers or arrays of letters, what happens in the first neural network portion 702A may be different than what happens in the second neural network portion 704A. For example, the first neural network portion 702A may be 2 layers deep and the second neural network portion 704A may be 90 layers deep. As long as each output is interpreted as a suitably configured sequence of numbers for transmission to the top portion of the neural network 710, forward propagation and back propagation can work and training can be achieved. This understanding paves the way for the new concept disclosed herein that different types of data processed across different parts 702A, 704A, 706A of a neural network can be properly received and processed to train models open it If the system can generate a different bottom half 702A, 704A, 706A for each of the different clients, then the clients 702, 704, 706 will generate the same type of data (e.g. between text and images). Neural network parts 702A, 704A, 706A, properly formatted, can process the disparate data and generate structured output that can be sent to server 710.

In one example, client 1 702 may provide a person's electrocardiogram, client 2 704 may provide a chest x-ray of the heart, and client 3 706 may provide the four most interesting samples of the patient's blood. Genetic profiles of proteins can be provided. If neural network portions 702A, 704A, 706A can process different individual data types up to the correct vector structure for output, and can provide disparate types of data to server 710, then server 710 is different and It can be configured with an appropriate neural network that combines all the information to train a model that will be used to make a diagnosis that can take advantage of disparate types of data.

In one aspect, while the neural network portions 702A, 704A, and 706A each process different types of data, there is some correlation factor associated with the data. In the example above, all of the data may generally relate to the same person, but some data relate to the electrocardiogram and other data relate to the genetic profile, but all to the same person. Thus, one aspect of the present disclosure is that data have a common association. In another aspect, the data may not relate to the same person, but common associations may relate to age, gender, race, project, concept, weather, stock market, or other factor. For example, all data may relate to women between the ages of 30 and 35 years. Common associations thus have some flexibility in how they are applied.

In another example, the data can be images from a camera of a jet engine stream, another stream of data can be sensor data, another data can be flight characteristics from an airplane, and a common association can be airplanes. In another aspect, a common association is a consumer's purchasing habits for one type of data, web surfing patterns for another type of data, emails sent by the user for another type of data, and emails for another type of data from Siri. Audio or other voice processing tools, and another type of data may be other types of data, such as the user's current location or a physical store the consumer frequents. The server's output may be an advertisement to present to the user based on an analysis of disparate types of input. Thus, a common association can be associated with any concept in which disparate types of data can be associated with a concept.

11 illustrates an exemplary computer device that may be used in connection with any of the systems disclosed herein. In this example, FIG. 11 illustrates a computing system 1100 that includes components that are in electrical communication with each other using a bus-like connection 1105 . System 1100 includes processing unit (CPU or processor) 1110 and various system components including system memory 1115 , such as read only memory (ROM) 1120 and random access memory (RAM) 1125 . ) to the processor 1110. System 1100 may include a high-speed memory cache that is directly coupled to, adjacent to, or incorporated as part of processor 1110 . System 1100 may copy data from memory 1115 and/or storage device 1130 to cache 1112 for fast access by processor 1110 . In this way, the cache may provide performance improvements that avoid processor 1110 delays while waiting for data. These and other modules may control or be configured to control the processor 1110 to perform various actions. Other system memory 1115 may also be available. Memory 1115 may include a number of different types of memory with different performance characteristics. The processor 1110 may be any general-purpose processor and hardware or software service or module, and software instructions are configured to control the processor 1110 and a special purpose processor to be integrated into the actual processor design. Processor 1110 may be a completely self-contained computing system that includes multiple cores or processors, a bus, memory controller, cache, and the like. Multi-core processors can be symmetric or asymmetric.

To enable user interaction with device 1100, input device 1145 may use any number of input mechanisms, such as a microphone for voice, a touch sensitive screen for gesture or graphic input, a keyboard, mouse, motion input, and voice. can indicate Output device 1135 may also be one or more of a number of output mechanisms known to those skilled in the art. In some examples, multimodal systems may allow a user to provide multiple types of input to communicate with device 1100 . Communications interface 1140 may generally control and manage user input and system output. Since there are no restrictions on operating in a particular hardware arrangement, the basic features herein can be easily replaced with improved hardware or firmware arrangements as they are developed.

Storage device 1130 is non-volatile memory and includes magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs) 1125, read only memory (ROM) ) 1120, and hybrids thereof, may be a hard disk or other types of computer readable media capable of storing data accessible by a computer.

The storage device 1130 may include services or modules 1132 , 1134 , and 1136 for controlling the processor 1110 . Other hardware or software modules are contemplated. Storage device 1130 can be coupled to system connection 1105 . In one aspect, a hardware module that performs a particular function comprises a software component stored on a computer readable medium in association with necessary hardware components such as processor 1110, connection 1105, output device 1135, etc. to perform the function. can include

In some cases, such a computing device or apparatus may include a processor, microprocessor, microcomputer, or other component of the device configured to perform the steps of the methods disclosed above. In some examples, such a computing device or apparatus may include one or more antennas for transmitting and receiving RF signals. In some examples, such a computing device or apparatus may include an antenna and modem for transmitting, receiving, modulating, and demodulating RF signals as previously described.

Components of a computing device may be implemented as circuitry. For example, components may include one or more programmable electronic circuits (eg, a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and/or other suitable electronic circuitry). may include and/or may include electronic circuitry or other electronic hardware, and/or may be implemented using same, and/or may include computer software, firmware, or any combination thereof to perform the various operations described herein. It may be implemented by including and/or using. A computing device may further include a display (as an example of, or in addition to, an output device), a network interface configured to communicate and/or receive data, any combination thereof, and/or other component(s). The network interface may be configured to communicate and/or receive Internet Protocol (IP) based data or other types of data.

The methods discussed above are illustrated as logical flow diagrams, the operations of which represent sequences of operations that may be implemented in hardware, computer instructions, or a combination of both. In the context of computer instructions, operations refer to computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not to be construed as limiting and any number of the described operations may be combined in any order and/or in parallel to implement a process.

Additionally, the methods disclosed herein may be performed under the control of one or more computer systems composed of executable instructions and codes (e.g., executable instructions, one or more computer programs) collectively executed by hardware on one or more processors. Or one or more applications), or a combination thereof is possible. As noted above, the code may be stored on a computer readable or machine readable storage medium, for example in the form of a computer program comprising a plurality of instructions executable by one or more processors. Computer readable or machine readable storage media may be non-transitory.

The term “computer readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, contain, or carry instruction(s) and/or data. Computer readable media may include non-transitory media on which data may be stored and which do not include carrier waves and/or transitory electronic signals that propagate wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory or memory devices. Computer readable media may store code and/or machine executable instructions, which may include procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes or instructions, data structures, or program statements. combinations can be represented. Code segments can be coupled to other code segments or hardware circuits by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be conveyed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, and the like.

In some embodiments, computer readable storage devices, media, and memories may include cables or radio signals including bit streams and the like. However, where referred to, non-transitory computer readable storage media expressly excludes such media as energy, carrier signals, electromagnetic waves, and signals themselves.

Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that embodiments may be practiced without these specific details. For clarity of explanation, the subject technology in some instances may be presented as comprising devices, device components, method steps or routines implemented in software, or individual functional blocks comprising a combination of hardware and software. Additional components may be used other than those shown in the drawings or described herein. For example, circuits, systems, networks, processes, and other components may be shown in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Individual embodiments may be described above as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flowcharts may describe operations as a sequential process, many operations may be performed in parallel or concurrently. Also, the order of operations may be rearranged. The process ends when the actions are completed, but there may be additional steps not included in the figure. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the return of the function to the calling function or to the main function.

Processes and methods according to the examples described above may be implemented using computer executable instructions stored on or otherwise available from computer readable media. Such instructions may include, for example, instructions and data that cause or configure a general purpose computer, special purpose computer, or processing device to perform a particular function or group of functions. Some of the computer resources used may be accessible via a network. Computer-executable instructions may be intermediate form instructions, such as, for example, binary, assembly language, firmware, or source code. Examples of computer readable media that can be used to store information generated during the method according to the described example, information used and/or instructions are magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networks storage devices, and the like.

Devices implementing processes and methods according to the present disclosure may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may use a variety of form factors. When implemented in software, firmware, middleware or microcode, program code or code segments (eg, computer program products) to perform necessary tasks may be stored on computer-readable or machine-readable media. The processor(s) may perform the necessary tasks. Common examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, PDAs, rackmount devices, standalone devices, and the like. The functions described herein may also be implemented with peripherals or add-on cards. These functions may also be implemented on a circuit board between different chips or different processes running in a single device as a further example.

Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

In the foregoing description, aspects of the application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the application is not limited thereto. Thus, although exemplary embodiments of the present application have been described in detail herein, the inventive concept may be embodied and used in many other ways, and the appended claims cover such variations except where limited by the prior art. It should be understood that it should be interpreted as The various features and aspects of the applications described above may be used individually or jointly. Further, the embodiments may be utilized in environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. Methods are described in a specific order for purposes of explanation. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.

") 및 이상("

") 기호로 대체될 수 있다는 것을 이해해야 한다.Those skilled in the art will understand that the less-than ("<") and greater-than (">") symbols or terms used herein do not depart from the scope of this description, and the following ("

") and over("

") symbol.

Where a component is described as being “configured” to perform a particular action, such configuration refers to electronic circuitry programmable to perform the action (e.g., by designing electronic circuitry or other hardware to perform the action). processor or other suitable electronic circuitry), or through any combination thereof.

The phrase "coupled" means any component that is physically connected, directly or indirectly, to another component and/or communicates directly or indirectly with another component (eg, via a wired or wireless connection and/or other appropriate communication interface). Refers to any component (connected to another component).

Claim language or other language reciting “at least one” and/or “one or more” of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language referring to “at least one of A and B” or “at least one of A or B” means A, B or A and B. In another example, claim language referencing "at least one of A, B, and C" or "at least one of A, B, or C" is A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language of "at least one" of a set and/or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language citing "at least one of A and B" or "at least one of A or B" could mean A, B or A and B, and items not listed in the set of A and B may additionally include.

Although various examples and other information have been used to describe aspects within the scope of the appended claims, the claims may be based on the specific features or arrangements of such examples as those skilled in the art may use such examples to derive a wide range of implementations. should not be implied. It is also to be understood that while some subject matter may be described in language specific to examples of structural features and/or method steps, subject matter defined in the appended claims is not necessarily limited to such described features or acts. For example, these functions may be differently distributed or performed in other components than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

Claim language that refers to "at least one" of a set indicates that one member of the set or several members of the set satisfy the claim. For example, claim language referring to “at least one of A and B” means A, B or A and B.

Claims (20)

in the method,
Dividing a neural network into a first client-side network, a second client-side network, and a server-side network;
forwarding the first client-side network to a first client, wherein the first client-side network is configured to process first data from the first client, the first data having a first type and the first client-side network includes at least one first client-side layer;
forwarding the second client-side network to a second client, wherein the second client-side network is configured to process second data from the second client, the second data having a second type and a second client-side network includes at least one second client-side layer, and the first type and the second type have a common association;
training the first client-side network on first data from the first client and generating first activations;
sending the first activations from the first client-side network to the server-side network;
training the second client-side network on second data from the second client and generating second activations;
sending the second activations from the second client-side network to the server-side network;
training at least one server-side layer of the server-side network based on the first activations and the second activations to generate gradients; and
sending the gradients from the server-side network to the first client-side network and the second client-side network. The method of claim 1 , wherein the common association includes at least one of a device, person, consumer, patient, business, concept, medical condition, group of people, process, product and/or service. The method of claim 1 , wherein the server-side network comprises a global machine learning model. 2. The method of claim 1, wherein the neural network includes weights, biases and hyperparameters. The method of claim 1 , wherein the at least one first client-side layer and the at least one second client-side layer include the same number of layers or different numbers of layers. 2. The method of claim 1, wherein a cut layer exists between the server-side network and the first client-side network and the second client-side network. The method of claim 1 , wherein the first type comprises text data and the second type comprises image data. 8. The method of claim 7, wherein the first client-side network and the second client-side network are independent and operate independently. The method of claim 1 , wherein the first type comprises tabular data and the second type comprises image data. in the system,
processor; and
A computer-readable storage device storing instructions that, when executed by the processor, cause the processor to:
dividing the neural network into a first client-side network, a second client-side network and a server-side network;
forwarding the first client-side network to a first client, wherein the first client-side network is configured to process first data from the first client, the first data having a first type; the first client-side network includes at least one first client-side layer;
forwarding the second client-side network to a second client, wherein the second client-side network is configured to process second data from the second client, the second data having a second type; the second client-side network includes at least one second client-side layer, and the first type and the second type have a common association;
receiving, at the server-side network, first activations from training of the first client-side network on first data from the first client;
receiving, at the server-side network, second activations from training of the second client-side network on second data from the second client;
training at least one server-side layer of the server-side network based on the first activations and the second activations to generate gradients; and
and transmitting the gradients from the server-side network to the first client-side network and the second client-side network. 11. The system of claim 10, wherein the common association includes at least one of a device, person, consumer, patient, business, concept, medical condition, group of people, process, product and/or service. 11. The system of claim 10, wherein the server-side network includes a global machine learning model. 11. The system of claim 10, wherein the neural network includes weights, biases and hyperparameters. 11. The system of claim 10, wherein the at least one first client-side layer and the at least one second client-side layer include the same number of layers or different numbers of layers. 11. The system of claim 10, wherein a truncation layer exists between the server-side network and the first client-side network and the second client-side network. 11. The system of claim 10, wherein the first type and the second type are different types of data. 11. The system of claim 10, wherein the first type includes tabular data or time series data, and the second type includes image data. in the method,
Dividing the neural network into a first client-side network, a second client-side network and a server-side network;
forwarding the first client-side network to a first client, wherein the first client-side network is configured to process first data from the first client, the first data having a first type and the first client-side network includes at least one first client-side layer;
forwarding the second client-side network to a second client, wherein the second client-side network is configured to process second data from the second client, the second data having a second type and the second client-side network includes at least one second client-side layer, and the first type and the second type have a common association;
receiving, at the server-side network, first activations from training of the first client-side network on first data from the first client;
receiving, at the server-side network, second activations from training of the second client-side network on second data from the second client;
training at least one server-side layer of the server-side network based on the first activations and the second activations to generate gradients; and
sending the gradients from the server-side network to the first client-side network and the second client-side network. 19. The method of claim 18, wherein the first type and the second type are different types of data. 19. The method of claim 18, wherein the first type includes tabular data or time series data, and the second type includes image data.

Images