WO2021159751A1

WO2021159751A1 - Semantic and image recognition-based electrocardiography information extraction method and apparatus, computer device, and storage medium

Info

Publication number: WO2021159751A1
Application number: PCT/CN2020/125059
Authority: WO
Inventors: 宋青原; 王健宗; 吴天博; 程宁
Original assignee: 平安科技（深圳）有限公司
Priority date: 2020-09-22
Filing date: 2020-10-30
Publication date: 2021-08-19
Also published as: CN111931717B; CN111931717A

Abstract

A semantic and image recognition-based electrocardiography information extraction method and apparatus, a computer device, and a storage medium, which relate to artificial intelligence technology, and may be applied to smart medical treatment scenarios. The method comprises: performing semantic recognition on current text information to be recognized, and obtaining a semantic vector corresponding to the current text information to be recognized; receiving an uploaded electrocardiogram image, calling an attention mechanism-based Res2Net classification network, classifying the electrocardiogram image according to the attention mechanism-based Res2Net classification network, and obtaining a corresponding output vector; calling a pre-trained Light GBM model, inputting the semantic vector and the output vector into the Light GBM model for classification, and obtaining a corresponding classification result. The method also relates to medical technology and blockchain technology, in that text information uploaded by a user end and image information corresponding to an electrocardiogram image are combined, and classification is then performed by means of a Light GBM algorithm, thereby improving the accuracy and credibility of a classification result.

Description

Method, device, computer equipment and storage medium for extracting ECG information based on semantic and image recognition

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on September 22, 2020, the application number is 202011001748.5, and the application title is "Electrocardiographic Information Extraction Method and Apparatus Based on Semantic and Image Recognition". The entire content of the application is approved The reference is incorporated in this application.

Technical field

This application relates to the technical field of artificial intelligence intelligent decision-making, and in particular to a method, device, computer equipment and storage medium for extracting ECG information based on semantic and image recognition.

Background technique

At present, the ECG intelligent diagnosis system has been widely used in daily life. For example, some smart wearable devices (such as Apple's Apple Watch) can collect the user's ECG information, and can also generate the collected ECG information and upload it to The server performs subsequent image recognition to generate report information. The inventor realizes that the above method has the following defects:

1) Only the electrocardiogram is uploaded, and the text description is lacking;

2) ECG information is obtained based on smart portable measurement, and the accuracy and reliability are far inferior to medical electrocardiographs;

3) The accuracy of the image recognition model that recognizes the electrocardiogram is low.

Summary of the invention

The embodiments of this application provide a method, device, computer equipment and storage medium for extracting ECG information based on semantic and image recognition, aiming to solve the problem that the ECG information in the prior art is obtained based on smart portable measurement, with accuracy and reliability The problem of low accuracy and low accuracy of the image recognition model that recognizes the electrocardiogram.

In the first aspect, an embodiment of the present application provides a method for extracting ECG information based on semantic and image recognition, which includes:

Receive text description information uploaded by the client;

Determine whether the text description information includes preset keywords;

If the text description information includes the keyword, acquiring the text description information as the current text information to be recognized;

If the keyword is not included in the text description information, call the pre-stored guide question set and send it to the user terminal;

Receiving the reply text information correspondingly sent by the user terminal according to the guide question set as the current text information to be recognized;

Performing semantic recognition on the currently to-be-recognized text information to obtain a semantic vector corresponding to the currently-to-be-recognized text information;

Receiving the uploaded electrocardiogram image, invoking the pre-trained Res2Net classification network based on the attention mechanism, and classifying the electrocardiogram image according to the attention mechanism-based Res2Net classification network to obtain the corresponding output vector; and

The pre-trained Light GBM model is called, the semantic vector and the output vector are input to the Light GBM model for classification, and the corresponding classification result is obtained.

In the second aspect, an embodiment of the present application provides an electrocardiographic information extraction device based on semantic and image recognition, which includes:

The text description information receiving unit is used to receive the text description information uploaded by the client;

The keyword judgment unit is used to judge whether the text description information includes preset keywords;

The first text information obtaining unit is configured to obtain the text description information as the current text information to be recognized if the keyword is included in the text description information;

A guiding question set sending unit, configured to call a pre-stored guiding question set and send it to the user terminal if the keyword is not included in the text description information;

The second text information acquiring unit is configured to receive the reply text information correspondingly sent by the user terminal according to the guide question set as the current text information to be recognized;

A semantic vector acquiring unit, configured to perform semantic recognition on the currently to-be-recognized text information to obtain a semantic vector corresponding to the currently-to-be-recognized text information;

The image classification unit is configured to receive uploaded electrocardiogram images, call a pre-trained Res2Net classification network based on the attention mechanism, and classify the electrocardiogram images according to the Res2Net classification network based on the attention mechanism to obtain a corresponding output vector; as well as

The classification result obtaining unit is configured to call a pre-trained Light GBM model, input the semantic vector and the output vector to the Light GBM model for classification, and obtain a corresponding classification result.

In a third aspect, an embodiment of the present application further provides a computer device, which includes a memory, a processor, and a computer program that is stored on the memory and can run on the processor, and the processor executes the following steps:

Receive text description information uploaded by the client;

Determine whether the text description information includes preset keywords;

In a fourth aspect, the embodiments of the present application also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to perform the following operations:

Receive text description information uploaded by the client;

Determine whether the text description information includes preset keywords;

The embodiments of the present application provide a method, device, computer equipment and storage medium for extracting ECG information based on semantic and image recognition, including semantic recognition of the current text information to be recognized, so as to obtain information corresponding to the current text information to be recognized Receiving the uploaded ECG image, calling the pre-trained Res2Net classification network based on the attention mechanism, classifying the ECG image according to the Res2Net classification network based on the attention mechanism, and obtaining the corresponding output vector; and calling The pre-trained Light GBM model inputs the semantic vector and the output vector to the Light GBM model for classification, and obtains a corresponding classification result. This method combines the text information uploaded by the user terminal and the image information corresponding to the ECG image, and then performs classification through the Light GBM algorithm, which improves the accuracy and credibility of the classification results.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following will briefly introduce the drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present application. Ordinary technicians can obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic diagram of an application scenario of an ECG information extraction method based on semantic and image recognition provided by an embodiment of the application;

2 is a schematic flowchart of a method for extracting ECG information based on semantic and image recognition provided by an embodiment of the application;

FIG. 3 is a schematic diagram of a sub-process of a method for extracting ECG information based on semantics and image recognition provided by an embodiment of the application;

4 is a schematic diagram of another sub-process of the method for extracting ECG information based on semantics and image recognition provided by an embodiment of the application;

FIG. 5 is a schematic block diagram of an electrocardiographic information extraction device based on semantic and image recognition provided by an embodiment of the application;

FIG. 6 is a schematic block diagram of subunits of an electrocardiographic information extraction device based on semantic and image recognition provided by an embodiment of the application; FIG.

FIG. 7 is a schematic block diagram of another subunit of the device for extracting ECG information based on semantics and image recognition according to an embodiment of the application;

FIG. 8 is a schematic block diagram of a computer device provided by an embodiment of the application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

It should be understood that when used in this specification and appended claims, the terms "including" and "including" indicate the existence of the described features, wholes, steps, operations, elements and/or components, but do not exclude one or The existence or addition of multiple other features, wholes, steps, operations, elements, components, and/or collections thereof.

It should also be understood that the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in the specification of this application and the appended claims, unless the context clearly indicates other circumstances, the singular forms "a", "an" and "the" are intended to include plural forms.

It should be further understood that the term "and/or" used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations .

Please refer to Figures 1 and 2. Figure 1 is a schematic diagram of an application scenario of an ECG information extraction method based on semantic and image recognition provided by an embodiment of this application; Figure 2 is an ECG based on semantic and image recognition provided by an embodiment of this application The schematic flow chart of the information extraction method. The ECG information extraction method based on semantic and image recognition is applied to a server, and the method is executed by application software installed in the server.

As shown in Figure 2, the method includes steps S110 to S180.

S110: Receive the text description information uploaded by the user terminal.

In this embodiment, in order to obtain the output text information based on the electrocardiogram image and the description text more completely, the user terminal may be prompted to upload the text description information at this time. The user edits a self-description on the user terminal according to his own situation and uploads it to the server.

S120: Determine whether the text description information includes preset keywords.

In this embodiment, since the text description information uploaded by the user through the user terminal may be some descriptions that are not closely related to the user’s own health status, at this time, in order to obtain the text information more accurately, the server can first detect and determine the description. Whether the text description information includes preset keywords (such as chest tightness, shortness of breath, fast heartbeat, etc.).

S130: If the text description information includes the keyword, obtain the text description information as the current text information to be recognized.

In this embodiment, when it is determined that the text description information includes the keyword, it means that the text description information includes valid information, and the text description information is directly acquired as the current text information to be recognized.

S140: If the keyword is not included in the text description information, call a pre-stored guide question set and send it to the user terminal.

In this embodiment, when it is judged that the text description information does not include the keyword, it means that the text description information does not include valid information. At this time, the server needs to call the guided question set and send the guided question set to the client To guide users to supplement information. For example, the guidance question set includes multiple guidance questions such as "whether chest tightness and shortness of breath?" and "whether the heartbeat is fast?" By setting up this guide question set, users can efficiently guide users to reply to valid information, and the efficiency of subsequent keyword extraction and semantic vector extraction can be improved.

S150: Receive the reply text information correspondingly sent by the user terminal according to the guide question set as the current text information to be recognized.

In this embodiment, after multiple rounds of dialogue between the user and the server based on the guide question set, the text answers of the users are integrated, and the reply text information can be obtained as the current text information to be recognized. Through this guided response approach, the currently obtained response text information includes more effective information.

S160. Perform semantic recognition on the currently to-be-recognized text information to obtain a semantic vector corresponding to the currently-to-be-recognized text information.

In this embodiment, after obtaining the current text information to be recognized, in order to extract the key information therein, the word vectors corresponding to the keywords can be obtained to form a semantic vector.

In one embodiment, as shown in FIG. 3, step S160 includes:

S161. Invoke a pre-trained BERT model, extract keywords from the current text information to be recognized through the BERT model, and obtain a text keyword set corresponding to the current text information to be recognized; wherein, the BERT model represents Two-way encoder representation model of Transformers model;

S162: Perform one-hot encoding on each text keyword in the text keyword set to obtain a word vector corresponding to each text keyword;

S163: Calculate the semantic vector corresponding to the current text to be recognized according to the word vector corresponding to each text keyword and the weight value corresponding to each text keyword.

In this embodiment, when acquiring keywords in the text, keyword extraction is performed on the current text information to be recognized through the BERT model (that is, the two-way encoder representation model of the Transformers model), and then each keyword can be extracted The correction corresponds to the medical terminology (for example, the fast correction is replaced by fast heartbeat). Finally, the corresponding keywords are converted into word vectors and then the semantic vectors corresponding to the current text to be recognized are calculated.

Among them, the BERT model uses Transformer Encoder (ie the encoder in the Transformer structure) as the feature extractor, which is composed of Nx exactly the same layers, and each layer has 2 sub-layers (ie sub-layers), which are: Multi- Head Self-Attention mechanism (ie multi-head self-attention mechanism), Position-Wise fully connected forward neural network. For each sub-layer, two operations are added: Residual Connection and Normalization.

Moreover, the input of the BERT model is a linear sequence, which supports single sentence text and sentence pair text. The beginning of the sentence is represented by the symbol [CLS], and the end of the sentence is represented by the symbol [SEP]. If it is a sentence pair, the symbol [SEP] is added between the sentences.

The pre-training of the BERT model adopts two strategies for model pre-training: MLM (MLM is the abbreviation of Masked LM, which stands for masked language model) and NSP (NSP is the abbreviation of Next Sentence Prediction, which stands for predicting the next sentence model).

After keyword extraction is performed on the current text information to be recognized through the BERT model, and the text keyword set corresponding to the current text information to be recognized is obtained, each text keyword in the text keyword set is one-hot encoded, Obtain the word vector corresponding to each text keyword. Since the weight value of each keyword is known in the corpus, at this time, according to the word vector corresponding to each text keyword and the weight value corresponding to each text keyword, the semantic vector corresponding to the current text to be recognized is calculated . The semantic vector extracted in this way can more accurately represent the current text information to be recognized.

S170. Receive the uploaded electrocardiogram image, call a pre-trained Res2Net classification network based on the attention mechanism, and classify the electrocardiogram image according to the Res2Net classification network based on the attention mechanism to obtain a corresponding output vector.

In this embodiment, since the corresponding semantic vector has been obtained after semantic recognition based on the current text information to be recognized, if the semantic vector is only used as the data of the classification model, the content of the vector representation may be less, which affects the final The classification result, at this time, can further prompt the user to upload the ECG image, and add some picture features combined with the semantic vector, so that the final vector representation content is rich, which is more conducive to obtaining accurate classification results.

In one embodiment, as shown in FIG. 4, step S170 includes:

S171. Obtain a pixel matrix corresponding to the electrocardiogram image;

S172: Use the pixel matrix as an input of the Res2Net network in the Res2Net classification network based on the attention mechanism to perform operations to obtain a morphological feature vector;

S173. Use the morphological feature vector as the attention mechanism structure in the attention mechanism-based Res2Net classification network to perform operations to obtain an output vector.

In this embodiment, when the ECG image is received, the pixel matrix corresponding to the ECG image is input to the deep learning Res2Net network to learn the morphological special diagnosis of the picture, and then input it into the attention structure, allowing the model to focus more on finding the input The useful information in the data is more relevant to the output, and then more attention resources are devoted to this area, thereby improving the quality of the output. Res2Net (which is an upgraded version of the ResNet network, that is, the residual network). Compared with ResNet, Res2Net not only improves the accuracy of recognition, but also optimizes the size and parameters of the model. This more lightweight model can improve Response speed and reduce the server's hardware requirements.

In an embodiment, step S172 includes:

The pixel matrix is input into the Res2Net network for sequential convolution, identity mapping, pooling, and full connection are performed in a multi-layer residual structure to obtain a morphological feature vector.

In this embodiment, in the Res2Net network, all the convolutional layers in the multi-layer convolutional layer in the convolutional neural network except the first convolutional layer are transformed into a residual block to realize the identity mapping. Thereby improving the accuracy of recognition of the entire Res2Net network.

In an embodiment, before step S170, the method further includes:

Send the prompt information for obtaining the ECG image to the client or the smart electrocardiograph;

Receive the electrocardiogram image sent by the user terminal or the smart electrocardiograph according to the prompt information.

In this embodiment, the server provides two interfaces for uploading ECG images from the client and the smart electrocardiograph. When acquiring the ECG images, the server can send to the client or the smart electrocardiograph for obtaining the ECG images. Prompt information. Through this notification method, after the extraction of the semantic vector, the process of obtaining the image classification result can be triggered more quickly.

S180. Invoke a pre-trained Light GBM model, input the semantic vector and the output vector into the Light GBM model for classification, and obtain a corresponding classification result.

In this embodiment, after integrating the semantic vector and the output vector corresponding to the ECG image, the server obtains a complete set of feature vectors, and performs learning and judgment based on the current Light GBM model to obtain the corresponding classification results. Among them, the Light GBM model is a learning algorithm based on decision trees, which has faster training speed, higher accuracy and big data processing capabilities.

In an embodiment, step S180 includes:

Combining the semantic vector and the output vector with independent features to obtain a graphic feature vector;

The graphic feature vector is classified through the histogram-based decision in the Light GBM model to obtain a corresponding classification result.

In this embodiment, the purpose of combining the semantic vector and the output vector with independent features is to reduce feature dimensions and improve calculation efficiency. Since the semantic vector and the output vector are mutually exclusive, the two features are bundled together so that no information will be lost. When the independent feature combination is completed and the graphic feature vector is obtained, the histogram-based decision is used for classification. Since the histogram only needs to calculate the information gain for the histogram statistics, it is compared with the pre-sorting algorithm, which traverses all values every time , The calculation amount of information gain is much smaller, and the memory space needs to be relatively small.

In an embodiment, after step S180, the method further includes:

Call a pre-stored text template, and fill the classification result into the text template to obtain the current text;

Sending the current text to the client;

Upload the current text to the blockchain network.

In this embodiment, when the semantic vector obtained by the text semantic recognition and the output vector corresponding to the ECG image are combined, the classification by the Light GBM model is more accurate, and the classification result determined by the semantic vector and the output vector can be determined (for example, there are heart disease). At this point, you can call the pre-stored text template in the server, fill the classification result into the text template and get the current text (in addition to the classification result, there are some professional name explanations, etc.), and then send the current text to the user End for users to view.

In addition, the server can be used as a blockchain node device to upload the current text to the blockchain network, making full use of the non-tamperable characteristics of the blockchain data, and realizing the solidification of data evidence.

Wherein, the corresponding summary information is obtained based on the current text. Specifically, the summary information is obtained by hashing the current text, for example, obtained by using the sha256 algorithm. Uploading summary information to the blockchain can ensure its security and fairness and transparency to users. The user equipment can download the summary information from the blockchain to verify whether the current text has been tampered with. The blockchain referred to in this example is a new application mode of computer technology such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain, essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information for verification. The validity of the information (anti-counterfeiting) and the generation of the next block. The blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

This method combines the text information uploaded by the user terminal and the image information corresponding to the ECG image, and then performs classification through the Light GBM algorithm, which improves the accuracy and credibility of the classification results.

The embodiment of the present application also provides an electrocardiographic information extraction device based on semantic and image recognition. The electrocardiographic information extraction device based on semantic and image recognition is used to perform any of the foregoing electrocardiographic information extraction methods based on semantic and image recognition. Examples. Specifically, please refer to FIG. 5, which is a schematic block diagram of an electrocardiographic information extraction device based on semantic and image recognition provided by an embodiment of the present application. The electrocardiographic information extraction device 100 based on semantic and image recognition can be configured in a server.

As shown in FIG. 5, the electrocardiographic information extraction device 100 based on semantic and image recognition includes: a text description information receiving unit 110, a keyword judgment unit 120, a first text information acquiring unit 130, a guide question set sending unit 140, and a second The text information acquisition unit 150, the semantic vector acquisition unit 160, the image classification unit 170, and the classification result acquisition unit 180.

The text description information receiving unit 110 is configured to receive the text description information uploaded by the client.

The keyword judgment unit 120 is configured to judge whether the text description information includes preset keywords.

The first text information obtaining unit 130 is configured to obtain the text description information as the current text information to be recognized if the keyword is included in the text description information.

The guiding question set sending unit 140 is configured to call a pre-stored guiding question set and send it to the user terminal if the keyword is not included in the text description information.

The second text information acquiring unit 150 is configured to receive the reply text information correspondingly sent by the user terminal according to the guide question set as the current text information to be recognized.

The semantic vector obtaining unit 160 is configured to perform semantic recognition on the current text information to be recognized to obtain a semantic vector corresponding to the current text information to be recognized.

In an embodiment, as shown in FIG. 6, the semantic vector obtaining unit 160 includes:

The keyword extraction unit 161 is configured to call a pre-trained BERT model, extract keywords from the current text information to be recognized through the BERT model, and obtain a text keyword set corresponding to the current text information to be recognized; wherein , The BERT model represents a bidirectional encoder representation model of the Transformers model;

The word vector obtaining unit 162 is configured to perform one-hot encoding on each text keyword in the text keyword set to obtain the word vector corresponding to each text keyword;

The semantic vector calculation unit 163 is configured to calculate the semantic vector corresponding to the current text to be recognized according to the word vector corresponding to each text keyword and the weight value corresponding to each text keyword.

The image classification unit 170 is configured to receive the uploaded electrocardiogram images, call the pre-trained Res2Net classification network based on the attention mechanism, and classify the electrocardiogram images according to the attention mechanism-based Res2Net classification network to obtain the corresponding output vector .

In one embodiment, as shown in FIG. 7, the image classification unit 170 includes:

The matrix obtaining unit 171 is configured to obtain a pixel matrix corresponding to the electrocardiogram image;

The morphological feature vector obtaining unit 172 is configured to use the pixel matrix as the input of the Res2Net network in the Res2Net classification network based on the attention mechanism to perform operations to obtain a morphological feature vector;

The output vector calculation unit 173 is configured to use the morphological feature vector as the attention mechanism structure in the attention mechanism-based Res2Net classification network to perform operations to obtain an output vector.

In an embodiment, the morphological feature vector obtaining unit 172 is further configured to:

In an embodiment, the device 100 for extracting ECG information based on semantic and image recognition further includes:

The reminder information sending unit is used to send the reminder information for obtaining the electrocardiogram image to the user terminal or the smart electrocardiograph;

The electrocardiogram image receiving unit is used to receive the electrocardiogram image sent by the user terminal or the smart electrocardiograph according to the prompt information.

In this embodiment, the server provides two interfaces for uploading ECG images on the client side and uploading ECG images from the smart electrocardiograph. The server can send ECG images to the client or the smart electrocardiograph when acquiring the ECG images. Prompt information. Through this notification method, after the extraction of the semantic vector, the process of obtaining the image classification result can be triggered more quickly.

The classification result obtaining unit 180 is configured to call a pre-trained Light GBM model, input the semantic vector and the output vector into the Light GBM model for classification, and obtain a corresponding classification result.

In an embodiment, the classification result obtaining unit 180 includes:

A graphic feature vector obtaining unit, configured to merge the semantic vector and the output vector with independent features to obtain a graphic feature vector;

The decision classification unit is configured to classify the graphic feature vector through a histogram-based decision in the Light GBM model to obtain a corresponding classification result.

The current text generation unit is configured to call a pre-stored text template, and fill the classification result into the text template to obtain the current text;

The current text sending unit is used to send the current text to the user terminal;

The on-chain unit is used to upload the current text to the blockchain network.

The device realizes the combination of the text information uploaded by the user terminal and the image information corresponding to the ECG image, and then the Light GBM algorithm is used for classification, which improves the accuracy and credibility of the classification results.

The above-mentioned apparatus for extracting ECG information based on semantics and image recognition can be implemented in the form of a computer program, and the computer program can be run on a computer device as shown in FIG. 8.

Please refer to FIG. 8, which is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 is a server, and the server may be an independent server or a server cluster composed of multiple servers.

Referring to FIG. 8, the computer device 500 includes a processor 502, a memory, and a network interface 505 connected through a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.

The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute an electrocardiographic information extraction method based on semantics and image recognition.

The processor 502 is used to provide calculation and control capabilities, and support the operation of the entire computer device 500.

The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the method for extracting ECG information based on semantics and image recognition. .

The network interface 505 is used for network communication, such as providing data information transmission. Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

Wherein, the processor 502 is configured to run a computer program 5032 stored in a memory to implement the method for extracting ECG information based on semantics and image recognition disclosed in the embodiments of the present application.

Those skilled in the art can understand that the embodiment of the computer device shown in FIG. 8 does not constitute a limitation on the specific configuration of the computer device. In other embodiments, the computer device may include more or less components than those shown in the figure. Or some parts are combined, or different parts are arranged. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structures and functions of the memory and the processor are consistent with the embodiment shown in FIG. 8 and will not be repeated here.

It should be understood that, in this embodiment of the application, the processor 502 may be a central processing unit (Central Processing Unit, CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSPs), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor.

In another embodiment of the present application, a computer-readable storage medium is provided. The computer-readable storage medium may be non-volatile or volatile. The computer-readable storage medium stores a computer program, where the computer program is executed by a processor to realize the method for extracting electrocardiographic information based on semantic and image recognition disclosed in the embodiments of the present application.

Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working process of the above-described equipment, device, and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here. A person of ordinary skill in the art may be aware that the units and algorithm steps of the examples described in the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both, in order to clearly illustrate the hardware and software Interchangeability, in the above description, the composition and steps of each example have been generally described in accordance with the function. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

In the several embodiments provided in this application, it should be understood that the disclosed equipment, device, and method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, or the units with the same function may be combined into one. Units, for example, multiple units or components can be combined or integrated into another system, or some features can be omitted or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may also be electrical, mechanical or other forms of connection.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments of the present application.

In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or software functional unit.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application is essentially or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium. It includes several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), magnetic disk or optical disk and other media that can store program codes.

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of various equivalents within the technical scope disclosed in this application. Modifications or replacements, these modifications or replacements shall be covered within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

An ECG information extraction method based on semantics and image recognition, including:

Receive text description information uploaded by the client;

Determine whether the text description information includes preset keywords;

If the text description information includes the keyword, acquiring the text description information as the current text information to be recognized;

If the keyword is not included in the text description information, call the pre-stored guide question set and send it to the user terminal;

Receiving the reply text information correspondingly sent by the user terminal according to the guide question set as the current text information to be recognized;

Performing semantic recognition on the currently to-be-recognized text information to obtain a semantic vector corresponding to the currently-to-be-recognized text information;

Receiving the uploaded electrocardiogram image, invoking the pre-trained Res2Net classification network based on the attention mechanism, and classifying the electrocardiogram image according to the attention mechanism-based Res2Net classification network to obtain the corresponding output vector; and

The pre-trained Light GBM model is called, the semantic vector and the output vector are input to the Light GBM model for classification, and the corresponding classification result is obtained.
The method for extracting ECG information based on semantic and image recognition according to claim 1, wherein said performing semantic recognition on said current text information to be recognized to obtain a semantic vector corresponding to said current text information to be recognized, include:

Invoke a pre-trained BERT model, extract keywords from the current text information to be recognized through the BERT model, and obtain a text keyword set corresponding to the current text information to be recognized; wherein, the BERT model represents the Transformers model The two-way encoder representation model;

Performing one-hot encoding on each text keyword in the text keyword set to obtain the word vector corresponding to each text keyword;

According to the word vector corresponding to each text keyword and the weight value corresponding to each text keyword, the semantic vector corresponding to the current text to be recognized is calculated.
The method for extracting ECG information based on semantic and image recognition according to claim 1, wherein said receiving the uploaded ECG image, calling a pre-trained Res2Net classification network based on the attention mechanism, and classifying the ECG image according to the The Res2Net classification network based on the attention mechanism performs classification, and before the corresponding output vector is obtained, it also includes:

Send the prompt information for obtaining the ECG image to the client or the smart electrocardiograph;

Receive the electrocardiogram image sent by the user terminal or the smart electrocardiograph according to the prompt information.
The method for extracting ECG information based on semantic and image recognition according to claim 1, wherein the pre-trained Res2Net classification network based on the attention mechanism is invoked, and the ECG image is based on the Res2Net based on the attention mechanism. The classification network performs classification and obtains the corresponding output vector, including:

Acquiring a pixel matrix corresponding to the electrocardiogram image;

Using the pixel matrix as an input of the Res2Net network in the Res2Net classification network based on the attention mechanism to perform operations to obtain a morphological feature vector;

The morphological feature vector is used as the attention mechanism structure in the Res2Net classification network based on the attention mechanism to perform operations to obtain an output vector.
The method for extracting ECG information based on semantics and image recognition according to claim 4, wherein the pixel matrix is used as the input of the Res2Net network in the Res2Net classification network based on the attention mechanism to perform operations to obtain morphological features Vectors, including:

The pixel matrix is input into the Res2Net network for sequential convolution, identity mapping, pooling, and full connection are performed in a multi-layer residual structure to obtain a morphological feature vector.
The method for extracting ECG information based on semantics and image recognition according to claim 1, wherein said inputting said semantic vector and said output vector to said Light GBM model for classification to obtain a corresponding classification result comprises :

Combining the semantic vector and the output vector with independent features to obtain a graphic feature vector;

The graphic feature vector is classified through the histogram-based decision in the Light GBM model to obtain a corresponding classification result.
The method for extracting ECG information based on semantics and image recognition according to claim 1, wherein said calling a pre-trained Light GBM model, and inputting said semantic vector and said output vector to said Light GBM model for classification , After obtaining the corresponding classification results, it also includes:

Call a pre-stored text template, and fill the classification result into the text template to obtain the current text;

Sending the current text to the client;

Upload the current text to the blockchain network.
The method for extracting ECG information based on semantics and image recognition according to claim 1, wherein the receiving text information corresponding to the response sent by the user terminal according to the guide question set is used as the current text information to be recognized, comprising:

Obtain multiple rounds of dialogue texts replies from the client and the server according to the guide question set, and extract the reply text information from the multiple rounds of dialogue text as the current text information to be recognized.
An ECG information extraction device based on semantic and image recognition, including:

The text description information receiving unit is used to receive the text description information uploaded by the client;

The keyword judgment unit is used to judge whether the text description information includes preset keywords;

The first text information obtaining unit is configured to obtain the text description information as the current text information to be recognized if the keyword is included in the text description information;

A guiding question set sending unit, configured to call a pre-stored guiding question set and send it to the user terminal if the keyword is not included in the text description information;

The second text information acquiring unit is configured to receive the reply text information correspondingly sent by the user terminal according to the guide question set as the current text information to be recognized;

A semantic vector obtaining unit, configured to perform semantic recognition on the current text information to be recognized to obtain a semantic vector corresponding to the current text information to be recognized;

The image classification unit is configured to receive uploaded electrocardiogram images, call a pre-trained Res2Net classification network based on the attention mechanism, and classify the electrocardiogram images according to the Res2Net classification network based on the attention mechanism to obtain a corresponding output vector; as well as

The classification result obtaining unit is configured to call a pre-trained Light GBM model, input the semantic vector and the output vector to the Light GBM model for classification, and obtain a corresponding classification result.
A computer device includes a memory, a processor, and a computer program that is stored on the memory and can run on the processor, and the processor implements the following steps when the processor executes the computer program:

Receive text description information uploaded by the client;

Determine whether the text description information includes preset keywords;

If the text description information includes the keyword, acquiring the text description information as the current text information to be recognized;

If the keyword is not included in the text description information, call the pre-stored guide question set and send it to the user terminal;

Receiving the reply text information correspondingly sent by the user terminal according to the guide question set as the current text information to be recognized;

Performing semantic recognition on the currently to-be-recognized text information to obtain a semantic vector corresponding to the currently-to-be-recognized text information;

Receiving the uploaded electrocardiogram image, invoking the pre-trained Res2Net classification network based on the attention mechanism, and classifying the electrocardiogram image according to the attention mechanism-based Res2Net classification network to obtain the corresponding output vector; and

The pre-trained Light GBM model is called, the semantic vector and the output vector are input to the Light GBM model for classification, and the corresponding classification result is obtained.
The computer device according to claim 9, wherein said performing semantic recognition on said current text information to be recognized to obtain a semantic vector corresponding to said current text information to be recognized comprises:

Invoke a pre-trained BERT model, extract keywords from the current text information to be recognized through the BERT model, and obtain a text keyword set corresponding to the current text information to be recognized; wherein, the BERT model represents the Transformers model The two-way encoder representation model;

Performing one-hot encoding on each text keyword in the text keyword set to obtain the word vector corresponding to each text keyword;

According to the word vector corresponding to each text keyword and the weight value corresponding to each text keyword, the semantic vector corresponding to the current text to be recognized is calculated.
The computer device according to claim 9, wherein said receiving the uploaded electrocardiogram image, calling a pre-trained Res2Net classification network based on the attention mechanism, and processing the electrocardiogram image according to the Res2Net classification network based on the attention mechanism Before classifying and obtaining the corresponding output vector, it also includes:

Send the prompt information for obtaining the ECG image to the client or the smart electrocardiograph;

Receive the electrocardiogram image sent by the user terminal or the smart electrocardiograph according to the prompt information.
The computer device according to claim 9, wherein the pre-trained Res2Net classification network based on the attention mechanism is invoked to classify the electrocardiogram image according to the Res2Net classification network based on the attention mechanism to obtain the corresponding output Vectors, including:

Acquiring a pixel matrix corresponding to the electrocardiogram image;

Using the pixel matrix as an input of the Res2Net network in the Res2Net classification network based on the attention mechanism to perform operations to obtain a morphological feature vector;

The morphological feature vector is used as the attention mechanism structure in the Res2Net classification network based on the attention mechanism to perform operations to obtain an output vector.
The computer device according to claim 12, wherein said using the pixel matrix as the input of the Res2Net network in the Res2Net classification network based on the attention mechanism to perform operations to obtain a morphological feature vector comprises:

The pixel matrix is input into the Res2Net network for sequential convolution, identity mapping, pooling, and full connection are performed in a multi-layer residual structure to obtain a morphological feature vector.
The computer device according to claim 9, wherein said inputting said semantic vector and said output vector to said Light GBM model for classification to obtain a corresponding classification result comprises:

Combining the semantic vector and the output vector with independent features to obtain a graphic feature vector;

The graphic feature vector is classified through the histogram-based decision in the Light GBM model to obtain a corresponding classification result.
The computer device according to claim 9, wherein the pre-trained Light GBM model is invoked, the semantic vector and the output vector are input to the Light GBM model for classification, and after the corresponding classification result is obtained, further include:

Call a pre-stored text template, and fill the classification result into the text template to obtain the current text;

Sending the current text to the client;

Upload the current text to the blockchain network.
9. The computer device according to claim 9, wherein the receiving text information corresponding to the response sent by the user terminal according to the guide question set is used as the current text information to be recognized, comprising:

Obtain multiple rounds of dialogue texts replies from the client and the server according to the guide question set, and extract the reply text information from the multiple rounds of dialogue text as the current text information to be recognized.
A computer-readable storage medium that stores a computer program that, when executed by a processor, causes the processor to perform the following operations:

Receive text description information uploaded by the client;

Determine whether the text description information includes preset keywords;

If the text description information includes the keyword, acquiring the text description information as the current text information to be recognized;

If the keyword is not included in the text description information, call the pre-stored guide question set and send it to the user terminal;

Receiving the reply text information correspondingly sent by the user terminal according to the guide question set as the current text information to be recognized;

Performing semantic recognition on the currently to-be-recognized text information to obtain a semantic vector corresponding to the currently-to-be-recognized text information;

Receiving the uploaded electrocardiogram image, invoking the pre-trained Res2Net classification network based on the attention mechanism, and classifying the electrocardiogram image according to the attention mechanism-based Res2Net classification network to obtain the corresponding output vector; and

The pre-trained Light GBM model is called, the semantic vector and the output vector are input to the Light GBM model for classification, and the corresponding classification result is obtained.
18. The computer-readable storage medium according to claim 17, wherein said performing semantic recognition on said currently to-be-recognized text information to obtain a semantic vector corresponding to said currently-to-be-recognized text information comprises:

Invoke a pre-trained BERT model, extract keywords from the current text information to be recognized through the BERT model, and obtain a text keyword set corresponding to the current text information to be recognized; wherein, the BERT model represents the Transformers model The two-way encoder representation model;

Performing one-hot encoding on each text keyword in the text keyword set to obtain the word vector corresponding to each text keyword;

According to the word vector corresponding to each text keyword and the weight value corresponding to each text keyword, the semantic vector corresponding to the current text to be recognized is calculated.
The computer-readable storage medium according to claim 17, wherein said receiving the uploaded ECG image, calling a pre-trained Res2Net classification network based on the attention mechanism, and classifying the ECG image according to the attention mechanism-based Res2Net Before the classification network performs classification and obtains the corresponding output vector, it also includes:

Send the prompt information for obtaining the ECG image to the client or the smart electrocardiograph;

Receive the electrocardiogram image sent by the user terminal or the smart electrocardiograph according to the prompt information.
The computer-readable storage medium according to claim 17, wherein the pre-trained Res2Net classification network based on the attention mechanism is invoked, and the ECG image is classified according to the attention mechanism-based Res2Net classification network to obtain The corresponding output vector includes:

Acquiring a pixel matrix corresponding to the electrocardiogram image;

Using the pixel matrix as an input of the Res2Net network in the Res2Net classification network based on the attention mechanism to perform operations to obtain a morphological feature vector;

The morphological feature vector is used as the attention mechanism structure in the Res2Net classification network based on the attention mechanism to perform operations to obtain an output vector.