WO2019132686A1

WO2019132686A1 - Method for generating mathematical models of a patient using artificial intelligence technologies

Info

Publication number: WO2019132686A1
Application number: PCT/RU2017/000820
Authority: WO
Inventors: Иван Сергеевич ДРОКИН; Олег Леонидович БУХВАЛОВ; Сергей Юрьевич СОРОКИН
Original assignee: Общество С Ограниченной Ответственностью "Интеллоджик"
Priority date: 2017-12-29
Filing date: 2017-12-29
Publication date: 2019-07-04
Also published as: RU2017137801A; RU2720363C2; RU2017137801A3

Abstract

The present technical solution relates to the field of artificial intelligence, and more particularly to methods for generating mathematical models of patients. The present method for generating mathematical models of a patient using artificial intelligence technologies includes the following steps: obtaining, on a server, a training dataset containing electronic medical records of patients grouped by patient; pre-processing, on the server, the data contained in the medical records of patients; converting, on the server, the processed data into a sequence of medical facts with respect to each patient, using medical ontologies; automatically tagging, on the server, the resulting sequence of medical facts with respect to each patient, using diagnoses or other facts of interest extracted from the patient's medical record; training, on the server, initial representations independently for each modality; training, on the server, combined representations; training, on the server, final models and aggregation parameters; obtaining, on the server, the medical record of a patient not included in the training dataset; pre-processing, on the server, the data contained in the patient medical record obtained; converting, on the server, the pre-processed data into a sequence of medical facts, using medical ontologies; generating, on the server, a vector representation of the patient.

Description

The method of forming a mathematical model of the patient using artificial intelligence technology

Technical field

This technical solution relates to the field of artificial intelligence, and in particular to methods of forming mathematical models of patients.

The level of technology

Currently, one of the most popular areas in the technology of artificial intelligence is the prediction of the development of diseases, diagnosis, the formation of recommendations, treatment strategies, etc. for a particular patient.

Usually, for each particular disease, neural network models are built that can diagnose one of the types of diseases. Data is submitted to such neural network models and a set of diagnoses is made for each patient. This approach is not very convenient and accurate, because does not allow to predict the development of diseases, treatment strategies.

To solve these problems, a vector representation of patients can be used, which makes it possible to build a mathematical model of the patient on the basis of anamnesis, medical history and treatment (methods and course of treatment, prescribed medications, etc.).

Essence

The technical problem solved in the framework of this technical solution is the creation of an effective mathematical model of the patient based on the records of his electronic medical history.

The method of forming a patient's mathematical models using artificial intelligence technologies includes the following steps:

receive on the server a training sample containing an electronic medical history of patients grouped by patient;

make on the server the preliminary processing of the data contained in the obtained patient histories;

convert the processed data on the server into a sequence of medical facts for each patient using medical ontologies;

produce on the server an automatic marking of the obtained sequence of medical facts for each patient, using diagnoses or other facts of interest extracted from the patient’s medical history;

primary representations are trained on the server independently for each of the modalities;

carry out joint representation training on the server;

produce on the server the training of final models and aggregation parameters; receive on the server the patient's history, not included in the training set;

make on the server the preliminary processing of the data obtained patient's history;

convert pre-processed data on the server into a sequence of medical facts using medical ontologies;

form the patient's vector representation on the server;

After the patient's vector representation has been formed, it can be used by various technical and software tools to analyze and predict the development of diseases, make diagnoses, simulate processes and trends in the patient's body, identify the effect of medications and prescribed treatment, determine the patient's mortality after surgery or treatment prescriptions.

Detailed description

The following are terms and definitions used in this technical solution.

Vector representation is a common name for various approaches to modeling a language and learning representations in natural language processing, aimed at matching words (and possibly phrases) from a certain dictionary of vectors from R ⁿ for n, significantly fewer words in the dictionary. The theoretical basis for vector representations is distributive semantics. There are several methods for constructing such a comparison, for example, using neural networks, methods of decreasing dimensionality applied to word co-occurrence matrices, and explicit representations studying explicit word representations.

The patient's vector representation is a mathematical model of the patient, based on the patient's physiological parameters, history, disease history and treatment (methods and course of treatment, prescribed drugs, etc.), etc., allowing to predict the development of diseases, diagnose, formulate recommendations treatment strategies, etc. for a particular patient.

Histology is a branch of biology that studies the structure, vital activity and development of tissues of living organisms.

Human histology is a branch of medicine that studies the structure of human tissues.

Distributive semantics is a field of linguistics, which is engaged in calculating the degree of semantic proximity between linguistic units based on their distribution (distribution) in large arrays of linguistic data (text corpuses). Distributive semantics is based on a distributive hypothesis: linguistic units occurring in similar contexts have similar meanings. Metadata - information about other information, or data related to additional information about the content or object. Metadata discloses information about the signs and properties that characterize any entity that allows you to automatically search for and manage them in large information flows.

Data modality is the belonging of data to a certain data source, determining the structure of the mentioned data, its format, and also allowing to correlate the said structure with one or another system of organs and / or nosologies and / or procedures. The source of data can be both the means of obtaining data about the patient and the patient himself.

Ontology is a comprehensive and detailed formalization of a certain area of knowledge using a conceptual scheme. Typically, such a scheme consists of a hierarchical data structure containing all the relevant classes of objects, their relationships and rules (theorems, constraints) adopted in this area.

Regularization (in statistics, machine learning, inverse problem theory) is a method of adding some additional information to a condition in order to solve an incorrectly posed problem or prevent retraining. This information often takes the form of a penalty for the complexity of a model, for example, it may be smoothness limitations of the resulting function or restrictions on the norm of a vector space.

Stemming (Engl stemming) - the process of finding the basis of a word for a given source word. The basis of the word does not necessarily coincide with the morphological root of the word.

Fact (medical fact) - the data describing the patient, including the methods of his treatment and the connection of the mentioned data with other medical facts.

Electronic patient case history (electronic medical record, patient electronic passport; Engl, electronic medical record - EMR, electronic health record) - a database containing information about the patient: physiological parameters of the patient, history, medical history and their treatment (methods and course of treatment prescribed drugs, etc.). Including an electronic case history of patients contains records of patients, including at least the following data: the date of addition of the record, codes of diagnoses, symptoms, procedures and drugs, textual description of the natural history of the disease, biomedical images associated with the history of the disease, research results and patient analyzes.

Medical personnel or another user of the medical information system uploads to the server through its interface data about the patient, which contain the patient's medical history, information about the patient's physiological parameters, other information.

In yet another implementation variant, the data on the server can be automatically downloaded without human intervention, for example, when collecting and examining tests, performing treatment procedures, etc.

s When a patient visits a doctor, undergoes examination, tests or other medical procedures, the medical information system generates (fills in and stores) the data for each such procedure. The data may include records of patient examinations, codes of diagnoses, symptoms, procedures and drugs prescribed and / or taken by the patient, a description of the history of the disease in a natural language, biomedical images, analysis results, studies, observations / measurements of physiological parameters, ECG, EEG, MRI , Ultrasound, biopsy, cytology, x-ray, mammography, but not limited to the specified data.

The specified data can be presented in a text, tabular format, in the form of time series, images, video, genomic data, signals, but not limited to. Data can also be presented in a structured and unstructured form. Additionally, the data between the above data can be used as data.

Analyzes may include, but are not limited to, blood tests, cerebrospinal fluid, urine, feces, genetic tests, etc. As part of the technical solution does not impose restrictions on the types of analyzes.

Consider the example illustrated in Figure 1:

One patient 105 comes for a primary examination to a specialist physician. The doctor performs the necessary medical actions, after which he forms a description of the symptoms of the patient, gives an appointment for testing. Next, the doctor enters this information into the computer through the interface of the medical information system 110, after which this data is stored in an electronic medical history. Another 106 patient comes to a second appointment with a general practitioner. The therapist makes a prescription of drugs to the patient, entering this data into an electronic medical record. Thus, for each patient, the necessary set of records in electronic case histories is formed, which can later be used by other doctors or decision support systems.

To build decision-making systems in the field of medicine, it is necessary to collect a certain amount of data that allows the system to learn to recognize diagnoses, groups of diagnoses or facts from medical data obtained. When the required amount of data is collected in the medical information system, it can be used as a training set. Most of the existing decision support systems use machine learning in its various manifestations.

One of the important components in the medical decision support system is a vector representation of the patient (a mathematical model of the patient), which allows to predict the development of diseases, diagnose, formulate recommendations, treatment strategies, etc. for a particular patient.

In one of the options for the implementation of the functionality that allows you to create a vector representation of the patient is located on a separate server. In one of the options for the implementation of the functionality that allows you to create a vector representation of the patient, is located on the same server, where the medical information system is located.

In one of the implementation options, the functional allowing to form a vector representation of the patient is a cloud service (eng, cloud service) using cloud computing (eng. Cloud computing) on a distributed server system.

At the first stage, a training sample 210 is formed to form a vector representation of the patient; FIG. 2 (English training dataset), which will later be used for learning by machine learning algorithms, including deep learning algorithms.

In one of the implementation options, the training sample is formed by the user of the medical information system, by selecting patient records.

In one of the implementation options for the selection of records can be carried out according to specified criteria. At the same time as the above criteria can be at least:

• on / off rules in the training set:

about patients with a history of a given set of anamnesis (for example, only patients with an oncological history);

o patients who meet the specified gender or age parameters (for example, only men aged 30 to 45 years);

about patients associated with patients already included in the training set, while the relationship is determined by at least the similarity of anamnesis, treatment methods, etc.

• rules for the inclusion of previously created training samples.

As part of this technical solution, the training sample contains an electronic medical history of patients grouped by patient. The patient's electronic medical record used to form a training set contains patient records including at least the following data: the date of adding the record, codes of diagnoses, symptoms, procedures and drugs, a textual description of the natural language history associated with the history of the disease biomedical images, research results and patient analyzes.

As an illustrative example, we present the following fragment from the patient’s medical history:

The presentation formats used may vary and change depending on the technology used. The described formats are not the only possible and are described for a better understanding of the principles laid down in this technical solution.

An electronic patient history can be presented in the format openEHR, HL7, etc. The choice of format and standard does not affect the essence of the technical solution.

In one embodiment, the record of the medical history is a set of fields containing at least parameters describing:

• patient's condition;

• methods of patient's radiation (methods, methods of their use, characteristics);

• means used in the patient's radiation (drugs, dosages, etc.);

• test results, etc;

and metadata linking the described parameters with parameters from other records

If the records in the training sample are not grouped by patient, after receiving the data, they are grouped using known algorithms or functions (for example, any sorting and sampling known from the prior art, including when sampling data from databases, using the 'GROUP BY ', ORDER BY "in SQL queries).

The data of the medical history can be presented in text, tabular format, in the form of time series, images, video, genomic data, signals, but not limited to. Data can also be presented in a structured and unstructured form.

The date of adding a record can only store the date, date and time, time stamp, while the records mentioned can contain the specified temporary objects, both in absolute form and relative (relative to temporary objects from other records).

The codes of diagnoses, symptoms, procedures, drugs may be presented in the ICD format (for example, ICD-10), SNOMED-CT, CCS (Clinical Classifications Software) or so on. The choice of format does not affect the essence of this technical solution.

The results of the analyzes can be presented in tabular form.

Textual description of the medical history can be presented in a structured and unstructured form (description in natural language). Biomedical images can be presented as images (jpg, png, tiff and other graphic formats), video (avi, mpg, mov, mkv and other video formats), 3D photos, 3D videos, 3D models (obj, max, dwg etc.). The results of ECG, EEG, MRI, ultrasound, biopsy, cytology, x-rays, mammography, etc. can be presented in the form of biomedical images.

RNA sequencing data can be presented in TDF (tiled data format) format, non-indexed formats such as GFF, BED and WIG, indexed formats such as BAM and Goby, as well as in bigWig and bigBed formats.

The formats described above reflect what minimum software is designed to work with the above-mentioned data (creation, modification, etc.).

After the training sample has been formed and obtained on the server, the server preprocesses the data 220 contained in the patient’s medical history selected from the training sample.

Preliminary data processing domain-specific and depends on the type of data and data source.

For each data type and / or data source, specialized handlers are defined. In the case when no data handler is provided for the data type and / or source, or it is not necessary, then an empty handler is used or a handler is skipped for this data type.

In one embodiment, the type of data is determined based on the metadata specified for at least one type of data field in an electronic patient record.

For example, in dicom, metadata indicates the modality of the data explicitly and the modality of the data is interpreted according to the internal definition of the dicom standard.

In one embodiment, the data type can be determined using signatures. At the same time on the server or external source there is a database of signatures with the help of which the data type in the record is determined.

For example, the presence of a byte sequence “GIF89a” at the beginning of the data (field or file) indicates that this is a bitmap in GIF format, and the presence of 'BM' bytes means that it is a bitmap in BMP format.

In one embodiment, the type of data may be determined based on the information contained in the record using predefined rules.

For example, the type of image data (Bitmap, Icon), multimedia data (video, sound) stored in the resources of the executable files (PE file, .exe) is determined based on the analysis of the structure of the resource section of the mentioned executable file. In one implementation variant, data of one type can be converted into data of another type (video — into a set of images and vice versa, a 3d object — into an image of the projections of the said object and vice versa, etc.).

For example, for CT images, a handler can be set, which transforms them into a series of raster images with possible normalization, if the parameters of the device on which the image was taken are known.

In another example, a text handler can be specified for the text, which produces standard text transformations for NLP (mainly lower case conversion, number replacement, removal of stop words and prepositions, stamming).

In yet another example, a natural language text can be assigned a processor that generates a sequence of medical facts from the text using its mapping (English mapping) on the terms of medical ontology and / or a glossary of medical terms.

In one of the implementation options for the analysis of the text in natural language, algorithms known from the prior art can be applied at least lexical analysis and syntactic analysis on the basis of which lexemes are extracted from the text and combined into objects representing a sequence of medical facts.

In one of the implementation options, when displaying text, each medical fact is annotated (marked) with a date and / or time corresponding to the date and / or time of the current record from the medical history.

For example, if a processor handles a field containing text in natural language from a patient record that has a date of 01/20/2017, then all medical facts will be annotated (marked) with date of 01/20/2017.

In yet another variant of implementation, a text recognition model is used previously for the text analysis, which is previously trained (by one of the machine learning methods), and as a result of which a set of medical facts is formed. At the same time, the mentioned model can be retrained (using teaching methods with a teacher) if the generated medical facts do not satisfy the predetermined criteria (for example, when analyzing the results by a specialist).

In one of the options for implementing a handler for a natural language, the handler searches for each word (after preprocessing) from text in an ontology or dictionary. If a word is found in an ontology or dictionary, then the processor stores the corresponding ontology concept or a word from the dictionary, while words not found in the ontology or dictionary are discarded. In one of the implementation options, more complex rules (procedures) for displaying natural language text in a sequence of facts can be used.

For example, for some concepts, additional patterns (regular expressions) may be specified to extract related concepts and / or quantities.

In one of the variants of implementation, ontologies and / or dictionaries are located locally on the server.

In another implementation variant, the server can receive ontologies and dictionaries from external sources,

For example, through the Ontology Lookup Service, which provides a web service interface for requesting many ontologies from one place with a unified data output format.

In one of the implementation options, as a source of knowledge, instead of ontologies or dictionaries, any source of medical data can be used from which a forest of knowledge can be formed (many acyclic directed knowledge graphs). Such sources include, in particular, medical scheme-guidelines, etc.

In one more variant of realization medical profile articles and / or textbooks can act as a source of knowledge. In this case, previously found articles are processed by methods of text recognition known from the prior art (using the above-described lexical and syntactic analyzes, the use of trained text recognition models, etc.).

In another of the implementation options, open biomedical ontologies are used as a source of knowledge (OBO, eng. Open Biomedical Ontologies).

In one of the implementation options, the handler performs data normalization (for each data type, its own normalization rules can be used).

For example, the measurement values of certain blood parameters, designed in the form of tables, can be processed by the processor to normalize the mentioned values (feature scaling), and the parameters of such a transformation are calculated on the training set. In particular, sample mean a and variance s ² are calculated, while

x— a

x = - s

In another example, for an image where the value of each pixel corresponds to the value of the measured density of the medium for X-rays in Houndsfield, the handler can display the Houndsfield scale in the range [-1, 1] (normalize all integer values to range [0..255] for black and white images to real values in the range [-1.0..1.0]). In particular, the normalization can be described by the formula:

Where

x is the normalized value in the space of values {};

Htt ~ the minimum value in the space of values {};

Htah - the minimum value in the space of values {};

c 'is the normalized value in the space of values {X};

Utt - the minimum value in the space of values {K};

Utah - the minimum value in the space of values {Y};

y 'is the normalized value in the space of values {Y};

In another example, the data from the table containing measurements of certain blood parameters are subjected to preprocessing - data normalization (reduction of each of the parameters to zero mean unit variance, the parameters of such a transformation are calculated on the training set).

In another example, the data presented as an image in the RGB format obtained from a microscope undergoes post-processing - binarization from probability to class. If the probability of pathology in the opinion of the model is greater than a predetermined threshold, then the image is marked as containing pathology, otherwise it does not contain.

In one of the implementation options, the processor handles noise filtering or noise reduction of the analyzed data (the process of removing noise from the useful signal in order to increase its subjective quality or to reduce the level of errors in transmission channels and digital data storage systems). In particular, one of the methods of spatial noise reduction (adaptive filtering, median filtering, mathematical morphology, methods based on discrete wavelet transform, etc.) can be used in image processing; for video, one of the temporal, spatial or space-time methods noise reduction.

Consider the example illustrated in FIG. 3:

Suppose there is an educational sample 310 consisting of patient records. For each sample record, the server preprocesses the data selected from the record fields. For example, the server retrieves one record 301 from the patient’s medical history from the training sample 310, determines the composition of the fields and / or the types of data contained in the record. In this example, the 301 record contains a date, a description in natural language, CT scans, and a blood test. Next, the server for each record field processes the data using the corresponding handler from the handler pool 320 (handlers 3201..32N). So, for example, the date can be processed with an empty handler 3201, text in natural language - handler 3202, performing standard text processing for NLP, CT images - handler CT 3203, blood test - handler 3204, making data normalization. After processing, the patient's record 301 contains the processed data 301 *, where the '*' symbol next to the field means that it contains modified records (different from the original ones).

In one of the implementation options, the handler is formed using one of the scripting languages (scripting languages), and in the form of plug-ins and libraries (including those that are executable files, such as PE, for example, dll).

In one of the options for implementation on the server, a set of procedures for elementary actions on data types is built in. Combining these procedures in the right order for the user allows you to create handlers yourself. Creating handlers in this case occurs with the help of built-in support for scripting languages or through the platform interface that allows you to create such handlers.

After the server has performed the necessary preliminary data processing, the server converts 230 processed data into a sequence of medical facts for each patient using medical ontologies.

The entire case history is converted by the server into a sequence of medical facts about the patient. Facts may contain additional information, such as a biomedical image, an ECG, test results, etc.

As an illustrative example, consider two entries from an electronic medical record and their result of conversion into a sequence of medical terms.

Record before conversion:

After converting the medical history into a set of medical facts, the server automatically marks the received sequence of medical facts 240 for each patient, using diagnoses taken from the patient's medical history or other facts of interest. If the data is marked up, this step is skipped by the server.

In one of the implementation options, the facts of interest are set by the server user or collected from users of this technical solution (for example, a doctor).

For example, the lists of inclusive and exclusive criteria for clinical trials may be the facts of interest, i.e. lists of criteria that a person must meet in order to be included in the clinic (including lists), or vice versa to be excluded from the clinic or not admitted to the clinic (excluding lists).

In another example, an inclusive criterion could be liver cancer with a tumor no larger than 5 mm.

In another example, smoking may be an exclusionary criterion, the patient's age above 65 years.

In one of the implementation options, the facts of interest are extracted from external sources.

For example, facts of interest can be extracted from medical information systems.

The server then organizes and groups the facts by inspections by time. Such a grouping is necessary in order to consider a group of facts within a single inspection simultaneously.

In one embodiment, the implementation of the analyzes may relate to the reception at which they were assigned, or allocated to a separate entity (separate inspection). In one of the options for the implementation of CT, MRI, histology refer to a separate examination.

In one more variant of realization, at least all research methods containing raw data (not a doctor's report, but an immediate result in the form of an image, video, time stamps) are referred to individual examinations. If only a doctor's report or the fact of passing the study is available, such data are considered as part of the examination.

For such grouping by inspections, the server uses information about the time and / or date associated with each fact.

Next, the server forms pairs {set of facts, diagnosis} or {set of facts, fact of interest} based on grouping by inspections.

In one of the options for the implementation of the pair formed by a simple search.

Next, the server prepares a training set for each of the data modalities. As mentioned earlier, data histology, x-rays, CT, mammography, etc. can be used as a data modality.

For example, to form a training sample for the modality of CT, the server selects records containing CT:

Then, after the formation of training samples for each modality, the server provides training for the primary representations 250 independently for each of the modalities.

For each modality on the server, a model (group of models) is set, for training in forecasting diagnoses found in this training sample that are present in these modalities.

In one of the implementation options, machine learning algorithms, such as:

• linear regression;

• logistic regression;

• algorithm to the nearest neighbors;

• random forest; • gradient boosting on trees;

• Bayes classifiers;

• deep neural networks (fully connected, convolutional, recurrent, their combinations).

In one more variant of realization, the mentioned model satisfies the requirement - compliance of the modality with which this model will work.

For example, convolutional networks are used for images.

In one of the implementation options for each modality, several models are defined.

In one of the variants of implementation, modalities are grouped into clusters (for example, all histology, X-ray, mammography, etc.) having a common architecture of models (one parametric family) and studying together, while each model from the cluster has different sets of weights.

In one of the implementation options for each modality, a set of parametric model families is formed. The parametric family means that there is a general view of models with a certain set of parameters, the definition of which uniquely defines the model.

For example, if a neural network is used as a model, one of the examples of the parametric family is a multilayer perceptron, and the parameters will be the number of layers and the number of neurons in the layers. Another example is any neural network with a fixed architecture that generates a parametric family (for example, a family designed for image segmentation, for classifying images, etc.).

The following parametric families can be used within this technical solution:

• convolutional neural networks for working with images, video, signals;

• recurrent neural networks for working with sequences of facts in the patient’s medical history and for building predictive models for processing unstructured textual information;

• Baeys approach and tabular data decision trees.

Let us examine in more detail on the example of working with images.

The following basic tasks are encountered in working with images: image classification (assign one or several classes or labels to each image), image segmentation (put one or several labels for each image pixel), localization of objects of interest (construct a framing rectangle within which object of interest (for each object in the image). For each of these tasks, an architecture is set that solves this problem. Modalities differ mainly in the size of the input image and in the number of target labels / objects of interest. The basis of each such model is taken by Dense-net, whose architecture concept is represented in FIG. five. The idea of this architecture is to use additional paths for the movement of information within the model, which makes it possible to effectively train even very large models with a large number of convolutional layers. With a given modality, the size of the input image and the number of classes, such a model forms a parametric family, and the weights of the neural network are just the parameters of the family. They are defined in the learning process, during which the models are presented with images and target labels, and the neural network changes its weights so that its response coincides with what is contained in the markup of the training set (the so-called target values or target response).

Further, for each modality, the server searches for family parameters that give the optimal result for this training sample.

In one of the implementation options for finding family parameters that give an optimal result, at least the following is used:

• Monte Carlo method;

• Bayesian optimization.

In one of the implementation options for the evaluation of models, it uses cross-validation (sliding control or cross-check), on the basis of which the best-performing model for this training set for this modality is selected. The sliding control procedure is as follows. A sample of X ^{L is} divided in two different ways into two non-intersecting subsamples:

x ^L = x ™ ux,

Where

- training subsample of length t,

- control subsample of length k = L— m,

n = l..N is the partition number.

For each partition, an algorithm is built.

O _n = DOS)

and calculates the value of the quality functional

Qn = (At _n , x)

The arithmetic average of Q _n values over all partitions is called the sliding control estimate:

cv (ji, x ^L = 1SZ. ₁ Q (um.xb ·

It is on the basis of the evaluation of the sliding control that the best model is chosen.

In one of the implementation options, the models are added as data is collected and new patient data sources are added. For example, a new type of research (eg, ECG) has become available in the medical information system. Next, a training sample is formed for the ECG modality, then a neural network model (Fig. 4) is trained on this training sample, from which the representation of this modality is obtained.

After the models are trained, the server forms the primary vector representations for each modality. To do this, the input of each model trained for this modality, the server provides previously prepared patient data, determines the output values of the model and the weights of the last hidden layer of this model for each record. The weights of the last hidden layer will then be used by the server as primary vector representations, which represent the mapping of modality to a vector of fixed size determined by the model. As a result, the server generates a new data set, which is a transformation of the original.

Each modality has its own vector dimension, for example, if the modality is m ^ and the dimension of the common space is n, then a mapping will be constructed:

Rmu - + R ⁿ

For example, let

x E R ^mu , y € R ⁿ ,

then the mapping:

y = ax + b,

Where

And E /? ^(p ' ^t e d ^p

Another example

y = f (Ax + b),

Where

/ - non-linear function (for example, ReLU, sigmoid, etc.).

That is, A is a matrix of size (n, that, and b is a vector of size n

In one of the implementation options, the vector representation of the text is built in the space in which the primary vector representations of all other modalities are displayed, i.e. the primary vector representation of the textual modality is mapped into the space of general representations by the identity mapping.

In the case of using a non-neural network model, two scenarios are possible:

• the output of the model as such is taken as the representation of the data; • The neural network model is a classifier that displays input data into a certain set of facts for which there are already vector representations. Since it is guaranteed that any model will generate a probability vector of the presence of a fact, then the vector representation will be a probability-weighted sum of vector-representations of the facts.

For example, a model is built.

K ^c , 0 · .c - * g,

Where

X - a set of signs

Y = {y i = 1, ..., n) is the set of target facts

x - model parameters.

Without loss of generality, we can reformulate the problem as follows:

P = (*, £): - I am ⁷¹ .

wherein

With such restrictions on the model _f, it can be interpreted as the probability of the fact that the patient has the fact yi (or other options, depending on the problem being solved, for example, the occurrence of a fact with a horizon a year and so on).

Having the training sample {*, ·, tj \, j = 1, ..., N, it is possible to determine the parameters of the model x. Denote the model parameters found in the learning process as £. Also, since each of y _t is a medical fact, it corresponds to a vector representation of 1 ^.

Then, for the new case we get the following: we construct the vector of input features x \ corresponding to this case and we obtain the corresponding vector of probabilities ρ = / (x, f); the vector representation of this modality in this case will be constructed as follows:

V ^ P V-

The server forms the primary vector representations for each of the modalities, resulting in a set of vector representations for medical facts and terms (diagnoses, symptoms, procedures and medications) and models for mapping the primary vector representations to the joint representation space.

In one of the implementation options, the server additionally pre-educates vector representations of medical terms (concepts), for example:

• if there is an additional source of data in the form of a large body of medical literature; • if there is an alternative training building that was assembled independently of the current one.

The server performs pre-training of medical terms (concepts) using distributive semantics and vector representation of words.

Each word is assigned its own context vector. A set of vectors forms a verbal vector space.

In one of the variants of implementation, pre-training of medical terms (concepts) is carried out using the Word2vec software for the analysis of the semantics of natural languages using ontology for regularization.

Regularization in statistics, machine learning, inverse problem theory is a method of adding some additional information to a condition in order to solve an incorrectly posed problem or prevent retraining. This information often has the appearance of a fine for the complexity of the model.

For example, these may be:

• restrictions on the smoothness of the resulting function;

• restrictions on the norm of the vector space;

• regularization on scales and on neuron activations;

• well-known regularization methods.

In this technical solution are used known from the prior art basic and generally accepted for machine and in-depth training methods of regularization. Suppose that E is the error function minimized in the learning process, W is the model weight, and A is the activation of all neurons of the hidden layers (if it is a neural network). Then, one of the most widely used regularization techniques called L _I

regularization can be described as follows: Instead of minimizing E, the problem of minimizing is solved

regularization of weights),

E + aL _x (A min (L _t regularization of activations),

Е + aL ₂ (A) - »min ( ₂ regularization of activations), where

norm.

There are also various options for the above cases. These regularizing terms (terms) impose additional (non-rigid) constraints (that is, not specified as an explicit system of equations and / or inequalities that generate the set W with R ⁿ allowable weights of the model) constraints on the possible weights of the model, thus avoiding overtraining. Also, besides L _X / L ₂ regularization can be used: • early stop:

The essence of this method is to select a small test sample from the training sample, which is not clearly involved in the learning process, but is used to measure the model error in the learning process. As soon as the error on this test sample begins to grow, learning stops.

• Synthetic increase of the training sample (data augmentation engl): the essence of this approach is that with each sample of the training sample a transformation is applied with some probability that does not change the desired response or allows applying a similar transformation to obtain a new expected response that will be correct. For example, by classifying an x-ray image of the chest for the presence or absence of signs of pneumonia, a mirror image about the vertical axis can be applied to the input image, since obviously this will not change the target label.

• restrictions on the model parameters can be explicitly imposed through the restriction on the value of the norms of the model weight vector: _X () <g or L ₂ (W) <g.

• other regularization methods widely used in machine and deep learning can be applied.

We illustratively describe the work of the word2vec tool: as an input, word2vec takes a large text corpus and matches a tin vector to the tin logo, giving the coordinates of the output words. First, he creates a dictionary, “learning” on the input text data, and then calculates a vector representation of the words. A vector representation is based on contextual intimacy: words that appear in the text next to identical words (and, therefore, have a similar meaning) will have close coordinates of word vectors in a vector representation. The resulting word vectors can be used for natural language processing and machine learning.

There are two basic learning algorithms in word2vec: CBOW (Continuous Bag of Words) and Skip-gram. CBOW (“continuous bag with words”) is a model architecture that predicts the current word based on its surrounding context. The architecture of the Skip-gram type works differently: it uses the current word to predict the words surrounding it. The word order of the context does not affect the result in any of these algorithms.

The resulting coordinate representations of word vectors allow us to calculate the "semantic distance" between words. Based on the context of these words, word2vec technology makes its predictions.

In one of the implementation options when using ontology as a regularization (restrictions on the structure of space), attention is used.

In one of the options for implementation when using ontology for regularization (restrictions on the structure of space) is used multilevel (hierarchical) error function, based on relations between terms from ontology. The ontology is used in the particular case in the form of a knowledge graph, which defines a hierarchy of terms and their categories. This allows you to pre-order the space of the vector representation, since it is obvious that proximity in the knowledge graph should mean proximity in the vector space between terms or categories. Using this, you can impose a fine on learning vector presentations. In the future, the penalty is minimized along with the main error function. Denote by c the vector of the current term. For q, we denote the binary measure of proximity between two terms from _x and ₂ on the ontology. If q (c ₁ , c ₂ ) = 0, then the terms can be considered close, if q (c _lt c ₂ ) = 1, then far. Then the error function on the ontology can be specified:

Now, in the process of learning vector representations, OD (c) can be used by analogy with the L _X / L ₂ regularization.

The use of regularization with the help of ontology allows to improve the quality of the model, without resorting to the expansion of the training sample. Due to a reasonable restriction on the space of representations imposed by regularization, the quality of the model is improved, which makes it possible in particular to avoid retraining and to make the algorithm more stable with respect to outliers and errors in the training set. Standard classical regularization methods also impose restrictions on the representation space, but they only narrow the options, in contrast to regularization on ontology, which imposes restrictions on the representation space based on external information about the domain domain.

In one embodiment, the ontology used for regularization is a parameter external to the system, can be set in advance and depend on the corresponding system of disease codes (for example, ldcd / S, ICD-10, etc.).

For each neural network model obtained in this step, the primary vector representation is extracted as the output of the hidden layer. This model allows you to display the input data of a given modality in the primary vector representation. This requires simple manipulations with a trained model, essentially angry at removing the output layer from the model.

After training and receiving the primary representations, the server performs the learning of the joint representations 260 (illustrated in Fig. 6) (English coordinated multimodal machine learning, more details “Multimodal Machine Learning: A Survey and Taxonomy”, Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency ).

In order to use non-textual data, for example, medical images, in the learning process of joint representations, it is necessary to train a model of the mapping from the space of this modality to the common vector space. For this, the primary vectorization of the modality and the learning function of the representation from the primary vector are used. representations in general, while the model described above can be used as a learning function (a model trained to map from a given modality space to a common vector space).

For example, if a model deals exclusively with image classification, then several hidden fully connected layers may follow the last convolutional layer. In this case, it is the output of the last hidden convolutional layer that is taken, and not the fully connected layer.

When a non-textual modality occurs in the history of a disease, for example, a CT study, its primary vector representation is taken, processed using a multilayer perceptron, and the output of this perceptron is considered to be the vector representation of this image and its cosine distance is considered on it.

Next, skip-gram is used, but when it comes to non-textual modalities (for example, medical images), as their vector representations, the output of the function for this modality is used, with the body of medical fact sequences extracted from the skip-gram input from medical records or medical texts.

Then, after training the joint representations, the server will train the final models and aggregation parameters 270.

Aggregation is a receipt from a set of vectors, where each vector represents a medical fact from the history of a selected patient a single vector.

Each fact is assigned a weight obtained in the learning process. A set of weights is formed, which are used in the process of obtaining a prediction / diagnosis for a specific patient - aggregation parameters. Then determine the weighted sum - multiplying each vector in the patient's history with the corresponding weight and the resulting vectors are added. Usually, the aggregation of vector representations uses the direct sum of vectors with _hell =

. where with _{hell is the} aggregated representation of the patient, - vector representations of the facts in the history of the patient. However, such an aggregation option may not always be optimal, since each of the facts may have a different weight in terms of decision-making for each of the nosology or for this patient. Therefore, it is proposed to use as an aggregation the following approach with _hell = Xf _{= 1} aiC _f , where a, is a scalar, S? = I ai = 1. Each of them can be either explicit model parameters and determined in the learning process, or be a function of the form a _{ = f (i, с _х , с ₂ , ..., с _к , г), where y is the parameters of this function, which are determined in the learning process with all the other weights of the model.

A graph of calculations is constructed, where weights are present as parameters. Further, the graph parameters are optimized for the current dataset using the gradient descent method. The resulting set of weights is a learner, that is, it is modified along with the rest of the model weights in the learning process. Weights define a specific function from a parametric family, which, from several input vectors, forms one output. All of the above can be summarized as follows: Classifier training for a group of diagnoses is based on graphs, the training sample is generated from available EHRs automatically based on the NLP technician (extracting the facts + arranging them in a time order, then "diagnosis"). The choice of the classifier is determined by the ability to work with vector non-categorical signs and in this method it is multi-layer fully connected neural networks with residues.

Figure 7 shows two pipelines for the ECG and biomedical images, for example, CT of the chest. The data first falls into the preprocessing block. The preprocessing block is domain-specific and transforms the input data into a form that the model can get. The transformed data is sent to the corresponding modality of the mentioned data model - for example, a CT scan of the thoracic section is sent to the neural network, which analyzes this study. The model can produce results in two forms (two outputs): this is the desired output of the model (probability of pathology, segmentation maps, etc.) and the vector representation of this particular example. The desired output of the model is sent to the post-process module that is associated with the model, and the output of this block is shown to the expert person, for example, or sent to the customer in the form of a report or in another convenient form.

The central diagram shows the vector space of medical concepts, which is based on skip-gram and ontology regularization, and each concept is mapped to a specific point in this space. For each model, the mapping into this space of medical concepts from vector representation, which is generated by the pipeline model, through the mapping function is also constructed.

Further, vectors for a particular patient are taken from this common space and sent to the final model, where they are aggregated into a single patient model, where it is later diagnosed and / or treatment is recommended, etc.

After all the necessary actions have been completed, the administrator, doctor or other user adds (sends) patient records to the server, which must be analyzed.

Displaying the results can be in the form of recommendations, highlighting regions of interest on medical images, in the form of reports.

FIG. 8 shows an example of a general-purpose computer system on which a given technical solution can be performed and which includes a multi-purpose computing device in the form of a computer 20 or a server including a processor 21, a system memory 22 and a system bus 23 that interconnects various system components, including system memory with a processor 21.

The system bus 23 may be any of various types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. Systemic the memory includes a read-only memory (ROM) 24 and a random access memory (RAM) 25. The ROM 24 stores the basic input / output system 26 (BIOS), consisting of the main routines that help to exchange information between the elements inside the computer 20, for example, moment of launch.

Computer 20 may also include a hard disk drive 27 for reading from and writing to a hard disk, a magnetic disk drive 28 for reading from or writing to a removable disk 29, and a storage device 30 for an optical disk to read from or writing to a removable optical disk. a disk 31 such as a compact disk, a digital video disk and other optical means. The hard disk drive 27, the magnetic disk drive 28 and the optical disk drive 30 are connected to the system bus 23 via the hard disk drive interface 32, the magnetic disk drive interface 33 and the optical drive interface 34. Drives and their respective computer-readable media provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for the computer 20.

Although the typical configuration described here uses a hard disk, removable magnetic disk 29 and removable optical disk 31, the technician will take into account that other types of computer-readable media that can store data that is accessible by computer can also be used in a typical operating environment. such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memory (RAM), read-only memory (ROM), etc.

Various program modules, including the operating system 35, can be stored on a hard disk, magnetic disk 29, optical disk 31, ROM 24 or RAM 25. Computer 20 includes a file system 36 associated with the operating system 35 or included in it or more software application 37, other software modules 38, and software data 39. A user can enter commands and information into computer 20 using input devices such as keyboard 40 and pointing device 42. Other input devices (not shown) may include Be a microphone, joystick, gamepad, satellite dish, scanner, or any other.

These and other input devices are often connected to the processor 21 via the serial port interface 46, which is connected to the system bus, but can be connected via other interfaces, such as the parallel port, the game port, or the universal serial bus (USB). A monitor 47 or other type of visual display device is also connected to system bus 23 via an interface, for example, video adapter 48. In addition to monitor 47, personal computers typically include other peripheral output devices (not shown), such as speakers and printers.

Computer 20 may operate in a networked environment through logical connections to one or more remote computers 49. Remote computer (or computers) 49 may be another computer, server, router, network PC, peer device or another node of a single network, and usually includes most or all of the elements described above with respect to computer 20, although only the device is shown storage information 50. Logical connections include a local area network (LAN) 51 and a global computer network (GCS) 52. Such network environments are common in institutions, corporate computer networks, the Internet.

The computer 20 used in the LAN network environment is connected to the local network 51 via a network interface or adapter 53. The computer 20 used in the network environment of the GCS typically uses a modem 54 or other means to establish communication with the global computer network 52, such as the Internet.

The modem 54, which may be internal or external, is connected to the system bus 23 via the serial port interface 46. In a networked environment, program modules, or parts of them, described with reference to computer 20, can be stored on a remote storage device. It is necessary to take into account that the network connections shown are typical, and other means can be used to establish the communication link between the computers and the link.

In conclusion, it should be noted that the information given in the description are examples that do not limit the scope of this technical solution defined by the formula. The person skilled in the art will recognize that there may be other embodiments of this technical solution consistent with the nature and scope of this technical solution.

Literature:

1. htps: //hackernoon.com/atention-mechanism-in-neural-networK-30aaf5e39512

2. https: //medium.eom/@Svnced/a-brief-overview-of-attention-mechanism-

13c578ba9129

3. “Medical Concept Representation Learning for Failure Prediction”, Edward Choi, Andy Schuetz, Walter F. Stewart, Jimeng Sun, 02/11/2016.

4. “Graph-based atention model for healthcare representation learning”, Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F. Stewart, Jimeng Sun, 2017.

Claims

FORMULA

• receive on the server a training set containing electronic

case history of patients grouped by patient;

• make on the server the preliminary processing of the data contained obtained histories of diseases of patients;

• convert the processed data on the server into a sequence of medical facts for each patient using medical ontologies;

• produce on the server an automatic markup obtained

sequences of medical facts for each patient, using diagnoses taken from the patient’s medical history or other

facts of interest;

• produce primary representations on the server independently for each of the modalities;

• implement joint representations on the server;

• make on the server training of final models and parameters

aggregation;

• receive on the server the patient’s non-medical history

training sample;

• make on the server the preliminary processing of the patient's medical history data;

• convert pre-processed data on the server into a sequence of medical facts using medical ontologies;

• form the patient's vector representation on the server;