KR20240014941A - AI analysis system based on continuous but incomplete data - Google Patents

Abstract

The present invention relates to a system including artificial intelligence to compensate for incomplete data due to missing values when multi-modal data is acquired in time series.
The artificial intelligence analysis system based on incomplete continuous data according to the present invention is effective in analyzing data even if there are missing data in a multi-modal data environment.

Description

AI analysis system based on continuous but incomplete data}

The present invention relates to an artificial intelligence analysis system based on incomplete continuous data.

Recently, technologies utilizing artificial intelligence (AI) have been actively developed, and securing data is an essential prerequisite for utilizing artificial intelligence.

In areas that require various types of data for analysis, for example, areas related to mental health, it is possible to diagnose specific symptoms by observing interview questions, behavior, and conversations, and combining video data that can confirm input information or behavior. You can.

As a prior art, technologies are being disclosed to replace data that is acquired sequentially in such a time sequence when some of it is missing. In this regard, Republic of Korea Patent Publication No. 10-2020-0030303 is disclosed.

However, this prior art uses data in a comprehensive situation where one or more modal data at a specific point in time is missing in a continuous multi-modal data environment collected at intervals, or some attributes are missing or noise is added within the modal data. There was a limitation in analyzing.

Republic of Korea Patent Publication No. 10-2020-0030303

The purpose of the present invention is to provide an artificial intelligence analysis system for compensating for incomplete data in conventional continuous data collected at time intervals and analyzing it based on this.

As a means of solving the above problem, a data collection unit that acquires continuous multi-modal data acquired according to orders at time intervals, and the characteristics of individual data included in the multi-modal data acquired according to each order An input data generation unit that extracts and converts individual vectors and connects individual vectors to generate an input vector containing multi-modal vectors; a missingness compensation artificial intelligence unit that compensates for missing values if there are missing values in the input vector and creates a supplemented input vector and an analysis artificial intelligence unit that receives and analyzes the supplemented input vector and outputs the analyzed result vector. The learning of the missingness compensation artificial intelligence unit is performed by substituting random missing values into multi-modal vectors according to multiple orders without missingness. An artificial intelligence analysis system based on incomplete continuous data can be provided that undergoes unsupervised learning based on the generated learning data and modifies weights by applying an error backpropagation algorithm.

Meanwhile, the missingness compensation artificial intelligence unit includes an input layer, a hidden layer, and an output layer. The input layer and the hidden layer are fully connected, the hidden layer and the output layer are fully connected, and the nodes of the hidden layer and the output layer are activated. The activation function may be a non-linear function.

Meanwhile, learning data for learning the missingness compensating artificial intelligence unit can be generated by selecting an arbitrary number of values in a multi-modal vector according to a plurality of orders without missingness and replacing them with 0.

At this time, the error function of missingness compensation artificial intelligence is , n is the number of training data, m is the dimension of the input vector, is the jth attribute of the ith input vector, may be the jth attribute value of the output vector when connecting the ith learning data.

Meanwhile, the error function of missingness compensation artificial intelligence is , n is the number of training data, m is the dimension of the input vector, is the jth attribute of the ith input vector, may be the jth attribute value of the output vector when connecting the ith learning data.

Meanwhile, the complementary artificial intelligence unit may include a recurrent connection to reflect the characteristics of the time series in the hidden layer.

Meanwhile, in the complementary artificial intelligence unit, any one node of the hidden layer can be recurrent to each of the other remaining nodes.

Additionally, when any node in the hidden layer recurrently recurs to each of the other remaining nodes, each node may have a unique weight.

Meanwhile, multi-modal data may include at least two of video, voice, text, sensor values, gamegater, and video specific values.

Meanwhile, multi-modal data may be medical data obtained during diagnosis and treatment of a disease.

The artificial intelligence analysis system based on incomplete continuous data according to the present invention is effective in analyzing data even if there are missing data in a multi-modal data environment.

1 is a block diagram of an artificial intelligence analysis system based on incomplete continuous data according to an embodiment of the present invention.
Figure 2 is a block diagram illustrating the concept of data processing in an artificial intelligence analysis system based on incomplete continuous data according to an embodiment of the present invention.
Figure 3 is a conceptual diagram showing the function of the input data generator.
Figure 4 is a conceptual diagram illustrating the concept of deriving an input vector from multi-modal data in an input data generator.
Figure 5 is a conceptual diagram showing the function of the missingness compensation artificial intelligence unit.
Figure 6 is a conceptual diagram illustrating the concept of an input vector for input to a missingness correction artificial intelligence unit.
Figure 7 is a conceptual diagram of a data set showing examples of multi-modal data, input vectors, and complemented input vectors.
Figure 8 is a diagram showing the neural network of the missingness compensation artificial intelligence unit.
Figure 9 is a diagram showing a neural network of a missingness compensation artificial intelligence unit in another embodiment of the present invention.

Hereinafter, an artificial intelligence analysis system based on incomplete continuous data according to an embodiment of the present invention will be described in detail with reference to the attached drawings. And in the description of the following embodiments, the names of each component may be referred to by different names in the art. However, if there is functional similarity and identity between them, they can be viewed as equivalent configurations even if modified embodiments are adopted. Additionally, symbols added to each component are described for convenience of explanation. However, the content shown in the drawings in which these symbols are written does not limit each component to the scope within the drawings. Likewise, even if an embodiment in which the configuration in the drawing is partially modified is adopted, if there is functional similarity and identity, it can be viewed as an equivalent configuration. Additionally, if it is recognized as a component that should naturally be included in light of the general level of technicians in the relevant technical field, the description thereof will be omitted.

1 is a block diagram of an artificial intelligence analysis system based on incomplete continuous data according to an embodiment of the present invention.

Referring to FIG. 1, the artificial intelligence system 1 based on incomplete continuous data according to an embodiment of the present invention is configured to be able to supplement and analyze missing values when there are missing values in data acquired in various modes.

In this disclosure, multi-modal data refers to data acquired in various ways. As an example, it can be data that serves as various indicators for determining a patient's symptoms. In this case, multimodal data can be image data, voice data, text data, sensor data, game data, video data, etc. At this time, multimodal data may include information about the acquired time.

Image data may include information about an image acquired from a patient and information about the time at which the image was acquired. For example, the image data may be an X-RAY image or CT image of the human body.

Voice data may include information such as the patient's vocalization, conversation content, or tone of voice.

Text data may include information entered by medical staff or guardians under the jurisdiction of the patient. As an example, it may include identification information such as the patient's age and name, and may also include observation result data for questionnaires already used in clinical practice. In this case, for example, the degree to which the patient's attention is lacking can be observed and input as a number.

Sensor data may include information obtained from a patient using various types of sensors. Sensor data may be, for example, the patient's temperature, weight, blood pressure, electrocardiogram, etc.

Game data may include content for exposing patients to specific situations, observing reactions, or requesting specific input. It may also include data obtained from the user when exposed to specific content while playing the game.

Video data is a sequence of images, and may be an observation video image of a patient, as an example. For example, in mental health, a patient's observation video may be a video of a patient's behavior to diagnose diseases such as ADHD or autism or to determine the severity. In this case, the video data may include information about the patient's behavior, expression, and posture. Additionally, video data may include images obtained from the human body, such as ultrasound diagnostic video images and angiography.

Ultimately, multimodal data can be comprehensive information that includes various clues to diagnose the symptoms of a specific subject.

However, although the above-mentioned example was described as an example for making a diagnosis related to mental health in the medical field, the embodiment according to the present invention is not limited to this, and multi-modal data may be various types of data for specific analysis. .

Referring again to FIG. 1, the artificial intelligence analysis system based on incomplete continuous data according to the present invention may be configured to include a data collection unit 10 and a server 20.

The data collection unit 10 is configured to receive various information. The data collection unit 10 may be configured to acquire image and video data from cameras, scanners, etc., and the microphone may be configured to acquire video data. Additionally, the data collection unit 10 can obtain information about content by extracting data from the game program. In other words, the data collection unit 10 refers to a wide variety of means for acquiring at least one piece of data over time from various devices such as cameras, microphones, and other sensors.

The server 20 may include a database unit 100, an input data generation unit 200, a missingness compensation artificial intelligence unit 300, and an analysis artificial intelligence unit 400. The server 20 may include a processor, memory, communication module, etc. The server 20 may receive data by communicating with the data collection unit 10 described above. However, since the configuration of the server 20 may include widely known configurations, further detailed description will be omitted.

The database unit 100 is configured to store multi-modal data transmitted from the data collection unit 10. It is saved as multi-modal data and can be converted into a database by synchronizing the time of each data. The data received from the database unit 100 is original data and does not often contain complete data over time. Inaccuracies or missing information may occur due to malfunction of some sensors, missing sensing, noise, etc. The original data may be incomplete, including these missing values.

The input data generator 200 divides orders according to a predetermined time interval, that is, original data into predetermined time intervals, and extracts multi-modal data from each order. Then, features for each individual data included in the multi-modal data at each order are extracted and vectorized individually.

Individual vectorization can be performed, for example, by preprocessing voice data, performing scale conversion, extracting it in the form of a heat map, and then vectorizing it. (Please check if the contents are correct. It is good.)

The input data generator 200 connects each individual vector according to time order in a predetermined order to generate a multi-modal vector. The function of this input data generation unit 200 can be understood as a pre-processing process before inputting data into the missingness compensation artificial intelligence unit 300 and the analysis artificial intelligence unit 400. Meanwhile, the input data generator 200 generates a multi-modal vector by replacing the vector value with 0 when there is a missing value in the original data.

The missingness compensation artificial intelligence unit 300 is configured to remove and compensate for the influence of missing values in the multi-modal vector.

The missingness compensation artificial intelligence unit 300 compensates for missing values by combining multi-modal vectors for a predetermined time, for example, multi-modal vectors for t-2, t-1, and t into one if it is of order 3. At this time, the size of simultaneously input data in the missingness compensation artificial intelligence unit 300, for example, how many orders of multi-modal vectors are selected, can be determined by windowing. In other words, a moving window, like a moving window, first inputs multi-modal data at t-3, t-2, and t-1, and then at t-2, t-1, and Multi-modal data at t can be entered. That is, the missingness correction artificial intelligence unit 300 can receive data for a continuous certain time interval when data is stored by order over time.

When multi-modal vectors are sequentially selected and input according to a moving window, the missingness compensation artificial intelligence unit 300 compensates for missing values to generate a supplemented output vector, and the supplemented output vector is sent to the analysis artificial intelligence unit 400. becomes an input for

The missingness compensation artificial intelligence unit 300 may be in a pre-trained state. The missingness compensation artificial intelligence unit 300 can be learned through unsupervised learning. At this time, the learning data may include multi-modal vectors for multiple orders over time. At this time, the multi-modal vector for learning is based on perfect data with no missing values, and missing values are randomly generated for random numbers and positions within the multi-modal vector. Ultimately, the input data includes randomly generated vectors with a value of 0.

The input data generated through the above process is input to the missingness correction artificial intelligence unit 300 and a complemented input vector is obtained. At this time, the weight is modified by applying the error backpropagation algorithm. The error function used at this time is MSE (Mean Square Error), and is specifically as follows.

Additionally, the error function may be cross entropy, specifically as follows.

At this time, n is the number of input data, m is the dimension of the input vector is the jth attribute of the ith input data, refers to the jth attribute value output by the missingness correction artificial intelligence unit when connecting the ith input data.

The analysis artificial intelligence unit 400 may receive a supplemented output vector from the missingness correction artificial intelligence unit 300 and output an analysis result vector.

At this time, the analysis artificial intelligence unit 400 can derive meaningful results that can be judged based on the output vector supplemented by supervised learning. For example, when learned to produce results related to mental health, when a supplemented output vector is input, the analysis result vector can be output to classify symptoms such as depression and autism. Since the analysis artificial intelligence unit 400 may use various known types of neural networks, detailed descriptions thereof will be omitted.

Figure 2 is a block diagram illustrating the concept of data processing in an artificial intelligence analysis system 1 based on incomplete continuous data according to an embodiment of the present invention.

Referring to FIG. 2, the multi-modal original data obtained from the data collection unit is collected by converting individual data for each order into feature vectors through the input data generation unit 200, and generating an input vector (X) including the multi-modal vector. creates . When generating a multi-modal vector (

Afterwards, the missingness compensation artificial intelligence unit 300 receives the input vector (X) and compensates for the missing values in the multi-modal vector that were replaced with the value of ‘0’ to generate a complemented input vector (X’).

The analysis artificial intelligence unit 400 receives the supplemented input vector (X'), analyzes it, and outputs the result vector (Y) to complete the analysis.

Figure 3 is a conceptual diagram showing the function of the input data generator.

Referring to FIG. 3, the original data may include multi-modal data for each order (t-2, t-1, t) over time.

At this time, the input data generator 200 performs individual feature vectorization for each order, for example, image features, audio features, text data, sensor data, game data, game data, and video features at time t-2, By collecting these, an input vector at time t-2 is generated. Also, an input vector (t-1) is generated at another order, that is, at time t-1, and an input vector (t) is also generated at time t.

However, although the generation of the input vector by the input data generator 200 has been described in terms of three orders, the same process may be performed at multiple points in time.

Figure 4 is a conceptual diagram illustrating the concept of deriving an input vector from multi-modal data in an input data generator.

Referring to FIG. 4, as explained with reference to FIG. 3, the input data generator 200 uses the original data to generate an input vector including a multi-modal vector. However, if there is a missing value, the corresponding value is generated. The input vector is created by replacing it with '0'. As an example, if data and sensor data for image features are missing at time t, the value for the corresponding feature vector is replaced with 0.

Figure 5 is a conceptual diagram showing the function of the missingness compensation artificial intelligence unit, and Figure 6 is a conceptual diagram showing the concept of an input vector for input to the missingness compensation artificial intelligence unit.

Referring to FIG. 5, the missingness compensation artificial intelligence unit 300 receives an input vector including a multi-modal vector for each order (t-4, t-3, t-2, t-1, t). You can. At this time, the missingness correction artificial intelligence unit 300 may receive input vectors for a predetermined number of adjacent orders, for example, three orders, through a moving window. As an example, referring to FIG. 6, three input vectors selected by window 2 in FIG. 5 (input vector at time t-3, input vector at time t-2, input vector at time t-1) They can be continuously connected and input into the missingness correction artificial intelligence unit 300.

That is, in window 1, the converted input vectors are simultaneously received at time points t-4, t-3, and t-2, and missing values are compensated. After missing values are corrected, the window moves to the next order. That is, in the next order, window 2 simultaneously receives the converted input vectors at time points t-3, t-2, and t-1, and compensates for any missing values. The missingness compensation artificial intelligence unit 300 compensates for missing values through the above-described process for numerous viewpoints acquired over time and generates a supplemented input vector.

Figure 7 is a conceptual diagram of a data set showing examples of multi-modal data, input vectors, and complemented input vectors.

Referring to FIG. 7, the data for the image feature (t), which was replaced with 0 in the input vector (X) by the missingness compensation artificial intelligence unit, and the missing values of the sensor data (t) are supplemented in the input vector (X') This has been done. Ultimately, the missing data is complemented by the missingness correction artificial intelligence unit using time series relationships, and a complete multi-modal vector can be created. The supplemented input vector including the supplemented multi-modal vector can be input to the analysis artificial intelligence unit, and the analysis artificial intelligence unit can output a result vector based on this.

As an example according to the present disclosure, assuming that a patient undergoes a diagnostic test every week, the time interval may be one week. At this time, CT scan, weight, and As an example, weight and X-ray imaging may be performed in the secondary examination, excluding CT scanning. In this case, the missing CT scan data can be supplemented with the CT scan data estimated by the missing value supplement artificial intelligence unit.

Additionally, the present disclosure does not necessarily require that the acquired multimodal data be acquired at the same time interval. In other words, even if the time interval between the first, second, and third orders in which multimodal data is acquired is not the same, missing values can be compensated for by the missingness compensation artificial intelligence unit.

As an example, according to the present disclosure, if there are missing values in data obtained over three rounds targeting ADHD patients, these can be supplemented. As an example, the time interval may be different for each order, such as the interval between the first order and the second order is 10 days, and the interval between the second order and the third order is 14 days. As a specific example, the patient plays Diagnostic Game 1, Diagnostic Game 2, and Diagnostic Game 3 in Wave 1, Diagnostic Game 2 and Diagnostic Game 3 in Wave 2, and Diagnostic Game 1 and Diagnostic Game 3 in Wave 3. can do. In this case, the missing game data may be data for the second round of diagnostic game 1 and data for the third round of diagnostic game 2. In this case, with the present disclosure, it is possible to supplement the data by estimating the missing data for the second round of diagnostic game 1 and the data for the third round of diagnostic game 2, and finally derive the results by analyzing the patient's condition. .

Hereinafter, the structure of the above-described missingness correction artificial intelligence unit will be described with reference to FIGS. 8 and 9.

Figure 8 is a diagram showing the neural network of the missingness compensation artificial intelligence unit.

Referring to FIG. 8, the missingness correction artificial intelligence unit 300 may include an input layer, a hidden layer, and an output layer. The input layer and hidden layer are fully connected, and the hidden layer and output layer can also be fully connected. The activation function of the nodes of the hidden layer and output layer is a non-linear function.

Meanwhile, in the missingness compensation artificial intelligence unit 300, the dimensions of the input vector (X) and the complemented input vector (X) are the same. That is, the number of nodes in the input layer and the number of nodes in the output layer are the same.

At this time, the complemented input vector (X') output by the missingness correction artificial intelligence unit 300 has the missing value replaced (imputation). For example, if the j-th element X(j) of the input vector (X) is a missing value, the input vector ( has

Figure 9 is a diagram showing a neural network of a missingness compensation artificial intelligence unit in another embodiment of the present invention.

Referring to FIG. 9, in another embodiment of the present invention, the missingness compensation artificial intelligence unit 300 may have a recurrent connection added similar to a Recurrent Neural Network (RNN).

The recurrent connection compensates for missingness. The artificial intelligence unit 300 is provided to reflect the characteristics of the time series in the hidden layer.

At this time, a hidden layer st (state) and a hidden layer st-1 are provided, and the hidden layer st-1 is configured to have the same number of nodes as the hidden layer st.

The connection (W) from hidden layer st to hidden layer st-1 has only one node connected (W1, W2, Wn), and the weight of each of these connections is 1. In other words, the values of the current hidden layer node are connected to each hidden layer in the previous state and the values are transmitted. Afterwards, each node of the hidden layer st-1 is connected to all nodes of the hidden layer st. That is, the connection (U) from hidden layer st-1 to hidden layer st is fully connected. At this time, each node in the hidden layer st-1 and the connection to each node in the hidden layer st have a unique weight.

As described above, the artificial intelligence analysis system based on incomplete continuous data according to the present invention has the effect of improving analysis accuracy by analyzing data even if there are missing data in a multi-modal data environment.

20: Server
100: database part
200: input data generation unit
300: Missing Compensation Artificial Intelligence Department
400: Analysis Artificial Intelligence Department

Claims (10)

a data collection unit that acquires continuous multi-modal data acquired according to orders at time intervals;
an input data generator that extracts features of individual data included in multi-modal data obtained according to each order, converts them into individual vectors, and connects the individual vectors to generate an input vector including the multi-modal vector;
When there is a missing value in the input vector, a missingness compensation artificial intelligence unit that supplements the missing value to generate a supplemented input vector; and
It includes an analysis artificial intelligence unit that receives and analyzes the supplemented input vector and outputs the analyzed result vector,
The learning of the missingness compensation artificial intelligence unit is,
Unsupervised learning is performed based on learning data generated by substituting random missing values into multi-modal vectors of multiple orders without missingness,
An artificial intelligence analysis system based on incomplete continuous data that modifies weights by applying an error backpropagation algorithm. According to claim 1,
The missingness compensation artificial intelligence department,
It includes an input layer, a hidden layer, and an output layer,
The input layer and the hidden layer are fully connected,
The hidden layer and the output layer are fully connected,
An artificial intelligence analysis system based on incomplete continuous data where the activation function of the nodes of the hidden layer and the output layer is a non-linear function. According to clause 2,
The learning data for learning the missing compensation artificial intelligence unit is,
An artificial intelligence analysis system based on incomplete continuous data that is created by selecting a random number of values in the multi-modal vector according to the plurality of orders without missingness and replacing them with 0. According to clause 3,
The error function of the missingness compensation artificial intelligence is,
and
where n is the number of learning data, m is the dimension of the input vector,
remind is the jth attribute of the ith input vector,
remind is an artificial intelligence analysis system based on incomplete continuous data, which is the j th attribute value of the output vector when the i th learning data is connected. According to clause 3,
The error function of the missingness compensation artificial intelligence is,
and
where n is the number of learning data, m is the dimension of the input vector,
remind is the jth attribute of the ith input vector,
remind is an artificial intelligence analysis system based on incomplete continuous data, which is the j th attribute value of the output vector when the i th learning data is connected. According to clause 3,
The supplementary artificial intelligence department,
An artificial intelligence analysis system based on incomplete continuous data that includes recurrent connections to reflect the characteristics of the time series in the hidden layer. According to clause 6,
The supplementary artificial intelligence department,
An artificial intelligence analysis system based on incomplete continuous data in which any one node of the hidden layer recurrently recurs to each of the other remaining nodes. According to clause 7,
An artificial intelligence analysis system based on incomplete continuous data, each of which has a unique weight when any one node of the hidden layer is recurrent to each of the other remaining nodes. According to claim 1,
The multi-modal data is an artificial intelligence analysis system based on incomplete continuous data including at least two of video, voice, text, sensor values, gamegater, and video specific values. According to claim 1,
The multi-modal data is,
An artificial intelligence analysis system based on incomplete continuous data, which is medical data obtained during the diagnosis and treatment of disease.

Images