TW202329153A - Data fusion system and method thereof - Google Patents

Abstract

The data fusion system is used to form a composite feature matrix. The composite feature matrix includes data in different formats. The data fusion system includes a data receiving device for collecting a plurality of data having different data formats. A feature extraction device, based on the data formats, uses the corresponding neural network architecture to perform feature matrix extraction on the data to generate a plurality of feature matrices. A feature integration device receives the feature matrices to combine the feature matrices to form the composite feature matrix.

Description

Data fusion system and its operation method

The present invention relates to a system for forming data and its operating method, in particular to a data fusion system and its operating method for forming a composite feature matrix including data in different formats.

As the population continues to grow and the population ages, more and more people need to seek medical help. In recent years, the application of big data analysis to medical assistance has been increasing day by day. Many doctors and engineers have cooperated to develop models that can help the efficiency and fault tolerance of the medical system, which can reduce the waste of medical system resources and strengthen precision medicine to a certain extent.

Traditionally, physiological data in different formats are trained with different algorithms to establish a corresponding judgment model. Models are built by training them separately with different algorithms. However, such an approach has limitations in the characteristics that can be learned from the data. For example, the judgment results generated by simply using the patient's image physiological data, such as X-rays, cannot be fully diagnosed by the doctor, and it is usually necessary to further combine the patient's physiological data Or the judgment results generated by the imaging physiological data, only by making a comprehensive judgment can an accurate diagnosis be made.

Therefore, there is room for improvement in the past model training method based on a single physiological data feature.

An implementation aspect of this case is to provide a data fusion system for forming a composite feature matrix including data in different formats, including: a data receiving element for collecting multiple data, wherein the data includes multiple data formats; A feature extraction component, which applies the corresponding neural network framework to extract feature matrices from the data according to the data format, and generates a plurality of feature matrices; and a feature integration component is used to receive the feature matrices to combine the The feature matrix becomes the composite feature matrix.

In some embodiments, the data include digital physiological data in text format, visual physiological data in optical format, and sound physiological data in acoustic format.

In some embodiments, the extraction component uses a feed-forward neural network architecture to extract a first feature matrix with a first dimension from the data physiological data in text format, and uses a convolutional neural network to extract a first feature matrix from the text format. A second feature matrix with a second dimension and a third feature matrix with a third dimension are respectively extracted from the image physiological data in the optical format and the sound physiological data in the acoustic format.

In some embodiments, the feature integration element directly connects the first feature matrix, the second feature matrix and the third feature matrix to form the composite feature matrix.

In some embodiments, the composite feature matrix has a fourth dimension, and the fourth dimension is equal to the sum of the first dimension, the second three-dimensional dimension and the third dimension.

In some embodiments, the feature integration element performs a parameter average calculation on the first feature matrix, the second feature matrix and the third feature matrix to form the composite feature matrix.

In some embodiments, before the feature integration element performs the parameter average calculation, it further includes performing a dimensionality reduction process of a matrix-vector product through a fully connected layer, so that the first feature matrix, the second feature matrix, and the second feature matrix The three feature matrices have the same dimensions.

In some embodiments, the feature integration element performs the parameter average calculation to form the composite feature matrix further includes: the composite feature matrix=a*(the first feature matrix)+b*(the second feature matrix 112 )+c*(the third characteristic matrix 113), wherein, a, b, c are respectively the corresponding weights of the first characteristic matrix, the second characteristic matrix and the third characteristic matrix, wherein a+b+c=1 .

In some embodiments, the data fusion system further includes a data cleaning component coupled to the data receiving component for sorting the data.

Another implementation aspect of this case is to provide a data fusion method for forming a composite feature matrix including data in different formats, including: collecting multiple data, wherein the data includes multiple data formats; according to the data formats Applying the corresponding neural network architecture to extract feature matrices from the data to generate a plurality of feature matrices; and directly combining these feature matrices or performing a parameter average calculation to form the composite feature matrix.

The present invention combines different data formats into a composite data providing model for learning. According to this, the model can carry out an integrated study of the data characteristics of different formats in this composite data. Therefore, it can combine the patient's data physiological data, image physiological data and sound physiological data to provide a comprehensive judgment, greatly improving the accuracy of judgment .

The above-mentioned description will be described in detail in the following embodiments, and a further explanation will be provided for the technical solution of the present invention.

The following will clearly illustrate the spirit of this case with drawings and detailed descriptions. Anyone with common knowledge in the technical field can change and modify the technology taught in this case after understanding the embodiment of this case. It does not depart from the spirit of this case. Spirit and scope.

The terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting of the present case. Singular forms such as "a", "the", "the", "this" and "the", as used herein, also include plural forms.

As used herein, "coupling" or "connection" can mean that two or more elements or devices are in direct physical contact with each other, or that they are in indirect physical contact with each other, or that two or more elements or devices are in physical contact with each other. Interaction or action.

"Includes", "including", "has", "containing" and so on used in this article are all open terms, meaning including but not limited to.

As used herein, "and/or" refers to any or all combinations of the above things.

Regarding the terms (terms) used in this article, unless otherwise specified, generally have the ordinary meaning of each term used in this field, in the content of this case and in the special content. Certain terms used to describe the subject matter are discussed below or elsewhere in this specification to provide those skilled in the art with additional guidance in describing the subject matter.

In order to solve the traditional problem of only using a single physiological data feature for learning, it can only provide one-way judgment results in the end, and cannot conduct a comprehensive evaluation to achieve the purpose of accurate diagnosis. Therefore, by integrating different data formats, this case combines data physiological data in text format, image physiological data in optical format, and sound physiological data in acoustic format into a composite data providing model for learning. According to this, the model can carry out an integrated study of the data characteristics of different formats in this composite data. Therefore, it can combine the patient's data physiological data, image physiological data and sound physiological data to provide a comprehensive judgment, greatly improving the accuracy of judgment .

FIG. 1 is a schematic diagram of a data fusion system integrating feature matrices of different data formats according to an embodiment of the present case. The data fusion system 100 can form a composite feature matrix 110 including data in different formats. The data fusion system 100 includes a data receiving component 101 , a data cleaning component 102 , a feature extraction component 104 and a feature integration component 108 . In one embodiment, the data cleaning component 102, the feature extraction component 104, and the feature integration component 108 can be integrated into a single component or separate components, which can be composed of a central processing unit, a microprocessor, a microcontroller, a digital signal processor, a special Application of integrated circuits to achieve.

The data receiving component 101 is used to collect data. In one embodiment, the data collected in this case includes different data formats, such as data physiological data in text format, image physiological data in optical format, and sound physiological data in acoustic format. In one embodiment, the data receiving component 101 is used to collect at least two of data physiological data in a text format, image physiological data in an optical format, and sound physiological data in an acoustic format.

The data sorting component 102 is coupled to the data receiving component 101 and is used for sorting the data collected by the data receiving component 101 to improve the quality of the data and form a data format suitable for computer processing. In one embodiment, the data sorting component 102 sorts out missing data, repeated data, inconsistency, and digital misplanted data. In one embodiment, the data organizing component 102 performs corresponding sorting according to different formats of data. In one implementation, for the physiological data in text format, the data organizing component 102 performs data normalization and classifies the data according to data attributes, and provides the subsequent feature extraction component 104 for feature matrix extraction. For the image physiological data in optical format, the data processing unit 102 can perform image conversion, strengthen the image and expand the contrast of gray scale or color in the image, remove the noise caused by poor transmission or interference or poor imaging or quantization in the image, and Features such as points, lines, edges, corners, and regions in the image are extracted, and provided to the subsequent feature extraction component 104 for feature matrix extraction. For the sound physiological data in the acoustic format, use MFCC (Mel-Frequency Cepstral Coefficients, Mel-Frequency Cepstral Coefficients) to organize to form the corresponding Mel-Frequency Cepstral Coefficient features, which are provided to the subsequent feature extraction component 104 for feature matrix fetch.

The feature extraction component 104 is coupled to the data organizing component 102 and performs feature matrix extraction on the data processed by the data organizing component 102 . In one embodiment, the feature extraction component 104 extracts corresponding feature matrices according to data in different formats. In one implementation, the feature extraction component 104 uses a feedforward neural network (Feedforward Neural Networks, FNN) to extract the feature matrix 105 from the physiological data in text format processed by the data organizing component 102 . In one embodiment, the extracted feature matrix 105 has 128 dimensions. In one implementation, the feature extraction component 104 uses a convolutional neural network (Convolutional Neural Networks, CNN) to extract the feature matrix 106 from the image physiological data in optical format processed by the data organizing component 102 . In one embodiment, the extracted feature matrix 106 has 256 dimensions. In one implementation, the feature extraction unit 104 uses a convolutional neural network (Convolutional Neural Networks, CNN) to extract the feature matrix 107 from the acoustic physiological data processed by the data organizing unit 102 in an acoustic format. In one embodiment, the extracted feature matrix 107 has 256 dimensions. It should be noted that the dimensions of the above-mentioned feature matrix are only one embodiment, and in other embodiments, feature matrices of different dimensions can also be generated according to requirements. In addition, the neural network architecture used in this case is not limited to the above-mentioned, other forms of neural network architecture, for example, artificial neural network (Artificial Neural Networks, ANN) architecture, recursive (circular) neural network ( Recurrent Neural Networks (RNN) architecture can also be used in the present invention to extract the feature matrix.

The feature integration component 108 is coupled to the feature extraction component 104 for receiving the feature matrix 105 , the feature matrix 106 and the feature matrix 107 generated by the feature extraction component 104 to combine into a composite feature matrix 110 to provide a model for learning. In one embodiment, the feature integration component 108 concatenates the feature matrix 105 , the feature matrix 106 and the feature matrix 107 to form a composite feature matrix 110 . Among them, the dimensions of feature matrix 105, feature matrix 106, and feature matrix 107 are directly connected, as shown in FIG. 2, which is a schematic diagram of a composite feature matrix 110 according to an embodiment of this case. In one embodiment, feature matrix 105 has 128 dimensions. The feature matrix 106 has 256 dimensions. The feature matrix 107 has 256 dimensions. Therefore, the formed composite feature matrix 110 will have a size of 128+256+256=640 dimensions.

In another embodiment, the feature integration component 108 performs weight average calculation on the feature matrix 105 , the feature matrix 106 and the feature matrix 107 to form a composite feature matrix 110 . That is to say, in this case, different weights can be assigned to the feature matrix according to the importance of the features considered. For example, when distinguishing a tumor, the image physiological data in optical format, such as the X-ray features of the patient, can be used in tumor discrimination. The importance of the data is generally higher than that of text-based physiological data, such as the patient's blood fat or blood pressure data characteristics, and acoustic physiological data, such as the patient's heart murmur characteristics. Therefore, according to the importance of the reference features, the feature matrix 106 extracted from the optical format image physiological data can be assigned a higher weight when performing tumor discrimination training. Accordingly, in order to arrange different weights for the feature matrix according to the importance of the feature to form a composite feature matrix 110 . Therefore, the feature matrix 105, the feature matrix 106, and the feature matrix 107 are first formed into a same dimension, as shown in FIG. 3, which is a schematic diagram of a composite feature matrix formed according to different weights according to an embodiment of the present case. In an embodiment, since the feature matrix 105 has 128 dimensions, the feature matrix 106 and the feature matrix 107 have 256 dimensions. Therefore, the feature matrix 106 and the feature matrix 107 are first reduced to 128 dimensions to be connected with the feature matrix 105 . In one embodiment, the feature matrix 106 and the feature matrix 107 are input into a fully connected layer (Fully connected layer, FC layer) 111 for dimensionality reduction, wherein the fully connected layer 111 performs a matrix-vector product, and the features are respectively transformed through a transformation matrix The matrix 106 and feature matrix 107 are dimensionally reduced from 256 dimensions to 128 dimensions, forming feature matrix 112 and feature matrix 113 to be connected with feature matrix 105 . In one embodiment, when the feature matrix 105, the feature matrix 112, and the feature matrix 113 are all 128 dimensions, the corresponding weights of the feature matrix 105, the feature matrix 112, and the feature matrix 113 can be arranged according to the importance of the reference feature, and the feature integration element 108 performs parameter average (weight average) calculation to form a composite feature matrix 110 . Wherein the composite feature matrix 110=a*(feature matrix 105)+b*(feature matrix 112)+c*(feature matrix 113). Wherein, a, b, and c are the corresponding weights of the feature matrix 105, the feature matrix 112, and the feature matrix 113, respectively, and a+b+c=1. Accordingly, the formed composite feature matrix 110 has 128 dimensions.

As shown in FIG. 4, it is a flow chart of forming a compound feature matrix including data in different formats according to an embodiment of the present case. Please also refer to Figures 1 to 4. First, in step 401, data collection is performed. In one embodiment, the data receiving component 101 is used to collect data, wherein the data collected in this case includes different data formats, such as data physiological data in text format, image physiological data in optical format, and sound physiological data in acoustic format.

In step 402, organize the collected data. In one embodiment, the data sorting component 102 is used for sorting the data collected by the data receiving component 101 to improve the quality of the data and to form a data format suitable for computer processing.

In step 403, feature matrix extraction is performed on the sorted data. In one embodiment, a feature extraction component 104 extracts corresponding feature matrices according to data in different formats. In one implementation, the feature extraction component 104 uses a feedforward neural network (Feedforward Neural Networks, FNN) to extract the feature matrix 105 from the processed physiological data in text format. A feature matrix 106 and a feature matrix 107 are respectively extracted from the image physiological data in the optical format and the physiological sound data in the acoustic format by using a convolutional neural network (CNN).

In step 404 , combine the feature matrix 105 , the feature matrix 106 and the feature matrix 107 into a composite feature matrix 110 . In one embodiment, a feature integration element 108 directly connects the feature matrix 105 with the first dimension, the feature matrix 106 with the second dimension, and the feature matrix 107 with the third dimension to form a feature matrix with the fourth dimension. The composite feature matrix 110, wherein the size of the fourth dimension is equal to the sum of the first dimension, the second dimension and the third dimension.

In another embodiment, a feature integration component 108 calculates the weight average of the feature matrix 105 , the feature matrix 106 , and the feature matrix 107 according to the importance of features to form a composite feature matrix 110 . In one embodiment, the feature matrix 106, the feature matrix 107 and the feature matrix 105 have the same dimension through a fully connected layer (Fully connected layer, FC layer) 111 matrix-vector product dimension reduction processing, and then according to the importance of features The characteristic matrix 105 , the characteristic matrix 112 and the characteristic matrix 113 are arranged with corresponding weights, and the characteristic integration component 108 performs parameter average (weight average) calculation to form a composite characteristic matrix 110 . Wherein, the composite feature matrix 110=a*(feature matrix 105)+b*(feature matrix 112)+c*(feature matrix 113). Wherein, a, b, and c are weights corresponding to feature matrix 105, feature matrix 112, and feature matrix 113, respectively.

To sum up, in this case, by integrating different data formats, data physiological data in text format, image physiological data in optical format, and sound physiological data in acoustic format are combined into a composite data providing model for learning. According to this, the model can carry out an integrated study of the data characteristics of different formats in this composite data. Therefore, it can combine the patient's data physiological data, image physiological data and sound physiological data to provide a comprehensive judgment, greatly improving the accuracy of judgment .

Although this case is disclosed as above with the embodiment, it is not used to limit this case. Anyone who is familiar with this technology can make various changes and modifications without departing from the spirit and scope of this case. Therefore, the scope of protection of this case should be regarded as attached The scope of the patent application shall prevail.

100:Data Fusion System 101: Data receiving component 102:Data cleaning component 104: Feature extraction component 105:Eigen matrix 106:Eigen matrix 107:Eigen matrix 108: Feature integration components 110:Composite feature matrix 111: Fully connected layer 112:Eigen matrix 113: Feature matrix 401-404: Steps

The drawings here are incorporated into the specification and constitute a part of the specification. These drawings show embodiments consistent with the present invention, and are used together with the description to illustrate the technical solutions of the embodiments of the present invention. FIG. 1 is a schematic diagram of a data fusion system according to an embodiment of the present case. FIG. 2 is a schematic diagram of forming a composite feature matrix according to an embodiment of the present case. FIG. 3 is a schematic diagram of forming a composite feature matrix according to another embodiment of the present application. FIG. 4 is a flowchart of a data fusion method according to an embodiment of the present case.

100:Data Fusion System

101: Data receiving component

102:Data cleaning component

104: Feature extraction component

105:Eigen matrix

106:Eigen matrix

107:Eigen matrix

108: Feature integration components

110:Composite feature matrix

Claims (10)

A data fusion system for forming a composite feature matrix including data in different formats, including: A data receiving component is used to collect multiple data, wherein the data includes multiple data formats; A feature extraction component, which applies a corresponding neural network framework to extract feature matrices from the data according to the data format to generate a plurality of feature matrices; and A feature integration element is used for receiving the feature matrices to combine the feature matrices to form the composite feature matrix. The data fusion system as described in Claim 1, wherein the data include data physiological data in text format, image physiological data in optical format, and sound physiological data in acoustic format. The data fusion system as described in claim 2, wherein the extraction component uses a feed-forward neural network architecture to extract a first feature matrix with a first dimension from the data physiological data in text format, and uses a The convolutional neural network extracts a second feature matrix with a second dimension and a third feature matrix with a third dimension from the image physiological data in the optical format and the sound physiological data in the acoustic format respectively. The data fusion system according to claim 3, wherein the feature integration component directly connects the first feature matrix, the second feature matrix and the third feature matrix to form the composite feature matrix. The data fusion system according to claim 4, wherein the composite feature matrix has a fourth dimension, and the fourth dimension is equal to the sum of the first dimension, the second three-dimensional dimension and the third dimension. The data fusion system according to claim 3, wherein the feature integration component performs a parameter average calculation on the first feature matrix, the second feature matrix, and the third feature matrix to form the composite feature matrix. The data fusion system as described in claim 6, wherein before the feature integration element performs the average calculation of the parameter, it further includes performing a dimensionality reduction process of a matrix-vector product through a fully connected layer, so that the first feature matrix, the second feature matrix The second feature matrix and the third feature matrix have the same dimension. The data fusion system as described in claim item 7, wherein the feature integration element performs the parameter average calculation to form the composite feature matrix further includes: the composite feature matrix=a*(the first feature matrix)+b*( The second characteristic matrix 112)+c*(the third characteristic matrix 113), wherein, a, b, and c are respectively the corresponding weights of the first characteristic matrix, the second characteristic matrix and the third characteristic matrix, wherein a +b+c=1. The data fusion system as described in Claim 1 further includes a data cleaning component coupled to the data receiving component for sorting the data. A data fusion method for forming a composite feature matrix including data in different formats, including: collecting plural data, which data includes plural data formats; Applying the corresponding neural network architecture to extract feature matrices from the data according to the data format to generate a plurality of feature matrices; and The composite feature matrix is formed by directly concatenating these feature matrices or performing a parameter average calculation.

Images