WO2021057423A1 - Procédé de traitement d'image, appareil de traitement d'image, et support de stockage - Google Patents

Procédé de traitement d'image, appareil de traitement d'image, et support de stockage Download PDF

Info

Publication number
WO2021057423A1
WO2021057423A1 PCT/CN2020/113114 CN2020113114W WO2021057423A1 WO 2021057423 A1 WO2021057423 A1 WO 2021057423A1 CN 2020113114 W CN2020113114 W CN 2020113114W WO 2021057423 A1 WO2021057423 A1 WO 2021057423A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature
image
expert
fusion
unsupervised
Prior art date
Application number
PCT/CN2020/113114
Other languages
English (en)
Chinese (zh)
Inventor
姜立
周雨熙
梁思阳
吴梦
李玉德
李红燕
韩立通
Original Assignee
京东方科技集团股份有限公司
北京大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司, 北京大学 filed Critical 京东方科技集团股份有限公司
Publication of WO2021057423A1 publication Critical patent/WO2021057423A1/fr

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/318—Heart-related electrical modalities, e.g. electrocardiography [ECG]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7271—Specific aspects of physiological measurement analysis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B6/00—Apparatus for radiation diagnosis, e.g. combined with radiation therapy equipment
    • A61B6/52—Devices using data or image processing specially adapted for radiation diagnosis
    • A61B6/5211—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/5215—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data

Definitions

  • the embodiments of the present disclosure relate to an image processing method, an image processing device, and a storage medium.
  • Image classification refers to automatically classifying input images into a set of predefined categories according to certain classification rules. For example, according to the semantic information contained in the image, the input image can be classified into objects and scenes. For example, a preset target object contained in the input image can be recognized and classified according to the recognized object. For another example, images with similar content can also be classified into the same category according to semantic information in the input image.
  • At least one embodiment of the present disclosure provides an image processing device, including: a depth feature extractor configured to obtain the depth feature of an image to be recognized, the image to be recognized is a medical image; and an expert feature extractor, configured to obtain the Identify the expert features of the image; a fusion processor configured to fuse the depth feature and the expert feature to obtain the fusion feature of the image to be recognized; the classification processor is configured to pair according to the fusion feature of the image to be recognized The image to be recognized is classified.
  • the depth feature extractor is further configured to obtain the depth feature of the image to be recognized by using a deep neural network.
  • the expert feature extractor is further configured to extract the expert features of the image to be recognized based on empirical formulas, rules, and feature values obtained from medical image data.
  • the category of the expert feature includes at least one of statistics, morphology, time domain, and frequency domain.
  • the unsupervised feature extractor is further configured to use principal components before acquiring the unsupervised features of the image to be recognized based on the unsupervised feature extractor.
  • At least one of the analysis method, the random projection method and the sequence autoencoder is trained to obtain the unsupervised feature extractor.
  • the fusion processor is further configured to splice the depth feature, the expert feature, and the unsupervised feature to obtain the fusion feature.
  • the fusion processor is further configured to: perform a global pooling operation and an average pooling operation on the depth feature, the expert feature, and the unsupervised feature, respectively. Operation to obtain the global vector and mean vector of the depth feature, the global vector and mean vector of the expert feature, and the global vector and mean vector of the unsupervised feature; splicing the global vector and mean of the depth feature At least one of the vectors, at least one of the global vector and the mean vector of the expert feature, and at least one of the global vector and the mean vector of the unsupervised feature, to obtain the fusion feature.
  • the classification processor is further configured to determine whether the image to be identified contains atrial fibrillation features according to the fusion feature of the image to be identified.
  • At least one embodiment of the present disclosure provides an image processing method, including: obtaining a depth feature of an image to be recognized based on a depth feature extractor, where the image to be recognized is a medical image; and an expert who obtains the image to be recognized based on an expert feature extractor Feature; fusion of the depth feature and the expert feature to obtain the fusion feature of the image to be recognized; classify the image to be recognized according to the fusion feature of the image to be recognized.
  • the image processing method provided by at least one embodiment of the present disclosure further includes obtaining unsupervised features of the image to be recognized based on an unsupervised feature extractor; fusing the depth feature, the expert feature, and the unsupervised feature, To obtain the fusion feature of the image to be recognized.
  • fusing the depth feature, the expert feature, and the unsupervised feature to obtain the fusion feature of the image to be recognized includes: stitching the depth Feature, the expert feature, and the unsupervised feature to obtain the fusion feature.
  • fusing the depth feature, the expert feature, and the unsupervised feature to obtain the fusion feature of the image to be recognized includes: The depth feature, the expert feature and the unsupervised feature perform a global pooling operation and an average pooling operation to obtain the global vector and the mean vector of the depth feature, the global vector and the mean vector of the expert feature, and the total The global vector and the mean vector of the unsupervised feature; splicing at least one of the global vector and the mean vector of the depth feature, at least one of the global vector and the mean vector of the expert feature, and the global vector of the unsupervised feature And at least one of the mean vector to obtain the fusion feature.
  • At least one embodiment of the present disclosure further provides an image processing device, including: a processor; a memory; one or more computer program modules, the one or more computer program modules are stored in the memory and configured to be configured by Executed by the processor, the one or more computer program modules include instructions for executing the image processing method provided by any embodiment of the present disclosure.
  • FIG. 1A is a flowchart of an image processing method provided by at least one embodiment of the present disclosure
  • FIG. 1B shows an exemplary scene diagram of an image processing system according to an embodiment of the present disclosure
  • 3A is a flowchart of a fusion operation provided by at least one embodiment of the present disclosure
  • FIG. 3B is a schematic diagram of a fusion operation provided by at least one embodiment of the present disclosure.
  • FIG. 4 is a flowchart of another image processing method provided by at least one embodiment of the present disclosure.
  • FIG. 5 is a schematic diagram of another fusion operation provided by at least one embodiment of the present disclosure.
  • FIG. 7 is a schematic block diagram of another image processing apparatus provided by at least one embodiment of the present disclosure.
  • FIG. 8 is a schematic block diagram of another image processing apparatus provided by at least one embodiment of the present disclosure.
  • FIG. 9 is a schematic diagram of an electronic device provided by at least one embodiment of the present disclosure.
  • FIG. 10 is a schematic diagram of a storage medium provided by at least one embodiment of the present disclosure.
  • Electrocardiogram is widely used in the diagnosis of various heart diseases, although medical equipment such as current medical electrocardiographs and wearable automatic monitoring equipment all have some basic electrocardiogram automatic analysis functions (such as automatic measurement of waveform parameters, rhythm parameters, etc.) ), but for some types of arrhythmia such as atrial fibrillation, due to the high error rate of automatic analysis and diagnosis of medical equipment, the interpretation and diagnosis of some arrhythmia such as atrial fibrillation are still mainly completed by medical experts.
  • the existing atrial fibrillation recognition methods can include a method based on feature engineering and a method based on deep learning.
  • Traditional atrial fibrillation recognition methods basically use feature engineering-based methods.
  • feature engineering-based methods can be divided into methods based on atrial activity analysis, methods based on ventricular response analysis, and combined with atrial activity And the way the ventricles react.
  • the method based on atrial activity analysis mainly focuses on the disappearance of P waves in atrial fibrillation or the appearance of F waves in the TQ interval.
  • the method based on atrial activity analysis mainly focuses on changes in the shape of the ECG data caused by changes in atrial activity.
  • the method based on ventricular response analysis mainly focuses on the change of the time interval (RR interval length) between heart beats detected by QRS.
  • the RR interval is mainly determined based on the peak position of the R wave with the largest fluctuation in the ECG signal data.
  • the method based on ventricular response analysis can be much less interfered by noise than the method based on atrial activity analysis, and it is also more suitable for real-time atrium Diagnose the problem with tremor.
  • Methods that combine atrial activity and ventricular response can provide greater performance by combining periodic independent signals.
  • the methods of combining atrial activity and ventricular response include: RR interval Markov model combining P wave morphological similarity measurement and PR interval variability, and fuzzy logic classification combining RR interval irregularity, P wave absence, and F wave appearance method.
  • the quality of a deep neural network model largely depends on the quality of the training samples.
  • the training samples inevitably contain various noises, and these training samples containing noise will have an important impact on the final recognition results. Therefore, in the detection of atrial fibrillation, the deep neural network model has a higher error rate.
  • the noise segment in the ECG sample (for example, the noise segment can include noise and
  • the semantic ambiguity caused by the current ECG sample belongs to other arrhythmia types) is also the main reason for the low accuracy of the arrhythmia recognition model. Deep learning technology is used to automatically extract the ECG signal data from the noisy data segment. Learning features will cause the wrong features to be mapped to the data distribution of the current arrhythmia type, resulting in the deterioration of the quality of the deep neural network model.
  • the existing methods have insufficient consideration of the problems caused by the modeling perspective, domain knowledge and other challenges, resulting in deep learning methods that are difficult to be accepted by the actual domain due to the lack of domain knowledge.
  • the domain knowledge-based atrial fibrillation detection method has the limitation of low accuracy, which makes it difficult to truly apply the results of atrial fibrillation detection in research scenarios in actual scenarios.
  • the image processing method, image processing device, and storage medium provided by the above-mentioned embodiments of the present disclosure use the representation and extraction of expert features based on domain knowledge, and the representation and extraction of deep features based on deep neural networks, and adopt a unified framework for experts
  • Features and depth features are expressed and fused to achieve the purpose of improving the accuracy of automatic detection of atrial fibrillation, thereby providing high-precision auxiliary methods for real-time and dynamic atrial fibrillation recognition and diagnosis, helping doctors to diagnose and accurately discover the patient’s atrial fibrillation in time
  • To help patients understand the changes in their condition in time thereby improving the quality of medical care, reducing the incidence of life-threatening conditions such as sudden cardiac death, and ultimately reducing the health and economic burdens brought to the family and society.
  • Step S120 Obtain expert features of the image to be recognized based on the expert feature extractor.
  • Step S130 Fusion of the depth feature and the expert feature to obtain the fusion feature of the image to be recognized.
  • Step S140 Classify the image to be recognized according to the fusion feature of the image to be recognized.
  • the deep feature extractor may be implemented as a deep neural network (for example, including a fully connected layer), and this step S110 includes using the deep neural network to obtain the depth features of the image to be recognized.
  • the deep neural network may be a convolutional neural network, such as any one of the Inception series network (such as Googlenet, etc.), the VGG series network, the Resnet series network, etc., or at least a part of any one of the foregoing networks.
  • the Inception series network such as Googlenet, etc.
  • VGG series network such as Googlenet, etc.
  • Resnet series network etc.
  • the deep feature extractor may also include an activation function layer connected to a deep neural network (for example, an Identity layer). ).
  • an activation function layer connected to a deep neural network (for example, an Identity layer).
  • the depth features extracted by the deep neural network can be reduced to obtain the reduced depth features, which can be matched with the dimensions of expert features or unsupervised features, so as to solve the above problems and improve the integration of features.
  • Diversity in order to achieve the purpose of improving the accuracy of automatic detection of atrial fibrillation.
  • the parameters in the deep neural network can be obtained by training in the training stage S1 in FIG. 2.
  • the deep neural network can be connected to the classifier.
  • the classifier is a Softmax classifier or an SVM (Support Vector Machine) classifier, etc.
  • the Softmax classifier is taken as an example for description, which is not limited in the embodiments of the present disclosure.
  • the classifier can classify the input data of the input deep neural network according to the extracted features.
  • the classification result of the classifier is output through the output layer as the final output of the deep neural network model.
  • FIG. 2 is a schematic diagram of extracting a depth feature provided by at least one embodiment of the present disclosure.
  • the deep feature extraction algorithm includes two phases: training phase and extraction phase.
  • a deep neural network is first trained based on a specific task (for example, judging whether it is atrial fibrillation), and the architecture and weights of the deep neural network are saved.
  • the deep neural network can output deep features, and output the deep features to the Softmax layer, so as to be in the Softmax layer (That is, the layer where the Softmax classifier is located) outputs the classification result based on the depth features extracted by the deep neural network (for example, the predicted probability that the image to be recognized belongs to a preset category (for example, atrial fibrillation)), and determines the label corresponding to the prediction result , That is, whether it is atrial fibrillation.
  • the depth features extracted by the deep neural network for example, the predicted probability that the image to be recognized belongs to a preset category (for example, atrial fibrillation)
  • the training process of the deep neural network may also include an optimizer, and the optimization function in the optimizer may calculate the error value of the parameters of the deep neural network according to the system loss value calculated by the system loss function, And according to the error value, the parameters of the deep neural network to be trained are corrected, so that the deep neural network can output more accurate depth features.
  • the optimization function may use a stochastic gradient descent (SGD) algorithm, a batch gradient descent (BGD) algorithm, etc., to calculate the error value of the parameters of the deep neural network.
  • SGD stochastic gradient descent
  • BGD batch gradient descent
  • the Softmax layer is a regression function layer for outputting classification results (for example, determining whether it is atrial fibrillation).
  • the last Softmax layer connected to the deep neural network is replaced by the Identity layer (for example, the output layer, using the Identity activation function), and the network structure of the other fully connected layers in the deep neural network is maintained
  • the sum weight is unchanged, that is, the deep features are extracted by the trained deep neural network, which can obtain the high-precision deep features, and reduce the dimensionality in the Identity layer to output the depth that matches the expert features or unsupervised features feature.
  • the deep features can be output from the Identity layer, so that the representation and extraction of deep features can be realized.
  • the deep feature extraction may not rely on the special architecture of the deep neural network, that is, any network architecture that can realize feature extraction can be used and is not limited to one type of network.
  • the above-mentioned deep feature extraction method is not limited to the above-mentioned neural network, and can also be implemented by conventional methods in the art, such as HOG+SVM, which is not limited in the embodiments of the present disclosure.
  • step S120 includes: extracting expert features of the image to be recognized based on empirical formulas, rules, and feature values obtained from medical image data.
  • the medical image data may include electrocardiogram data.
  • the category of the expert feature may include at least one of statistics, morphology, time domain, and frequency domain, and of course may also include other categories, which are not limited in the embodiments of the present disclosure.
  • statistical expert features include, for example, Mean, Maximum, Minimum, Variance, Skewness, Kurtosis, Percentile, and Statistical values such as Threshold. For example, if the maximum difference in the ECG monitoring time series data is large, it indicates that the patient may have atrial fibrillation.
  • the definition of morphology is directly related to specific areas. For example, in the field of atrial fibrillation detection, when the P-band disappears in the timing data of the ECG monitoring, it indicates that the patient may have atrial fibrillation.
  • time-domain analysis pays attention to the rhythmic characteristics of time series data in the time dimension. For example, if the RR interval of the time series data of ECG monitoring is irregular, it indicates that the patient may have atrial fibrillation.
  • the step S130 may include: stitching depth features and expert features to obtain fusion features.
  • FIG. 3A is a flowchart of a fusion operation provided by at least one embodiment of the present disclosure. That is, FIG. 3A is a flowchart of at least one example of step S130 shown in FIG. 1B.
  • the fusion operation includes step S1311 to step S1312.
  • FIG. 3B is a schematic diagram of a fusion operation provided by at least one embodiment of the present disclosure.
  • the fusion operation provided by at least one embodiment of the present disclosure will be described in detail with reference to FIG. 3A and FIG. 3B.
  • Step S1311 Perform a global pooling operation and an average pooling operation on the depth feature and the expert feature respectively to obtain the global vector and the mean vector of the depth feature and the global vector and the mean vector of the expert feature respectively.
  • the depth of features and specialist features global pool operation (e.g., maximum pooling) and mean cell operations, respectively, to obtain the global vector v max and mean vectors v mean and the specialist feature depth features are global vector v max And the mean vector v mean , where v max ⁇ R d and v mean ⁇ R d .
  • the global vector v max is obtained by calculating the maximum value of each column vector in the vector matrix V, for example, and the mean vector v mean is obtained by calculating the average value of each column vector in the vector matrix V, for example, by column.
  • Step S1312 Splice at least one of the global vector and the mean vector of the depth feature and at least one of the global vector and the mean vector of the expert feature to obtain a fusion feature.
  • concatenate the global vector and the mean vector of all features from different sources for example, concatenate the global vector and the mean vector of the depth feature from the deep feature extractor 110 and the global vector of the expert feature from the expert feature extractor 120 v max and the mean vector v mean to obtain the final fusion feature, provide easy-to-use and accurate multi-source fusion features for the atrial fibrillation detection task, thereby providing an accurate auxiliary method for atrial fibrillation detection.
  • the splicing operation is to merge the global vector and the mean vector of features from different sources into one vector, for example, one one-dimensional vector.
  • the depth feature and the expert feature may have multiple channels.
  • the depth feature can be a tensor of size H1*W1*C1, where H1 can be the size of the depth feature in the first direction (for example, the length direction), and W1 can be the depth feature in the second direction (for example, the width direction)
  • H1 and W1 can be the size in units of the number of pixels
  • C1 can be the number of channels of the depth feature.
  • the expert feature can be a tensor of size H2*W2*C2, where H2 can be the size of the expert feature in the first direction (for example, the length direction), and W2 can be the expert feature in the second direction (for example, the width direction) Size, H2 and W2 can be the size in units of the number of pixels, and C2 can be the number of channels of expert features.
  • C1 and C2 are integers greater than 1.
  • the depth feature can have 100 channels.
  • the depth feature in each channel is a one-dimensional vector of 1 row*M column (M is an integer greater than 1), and the depth features of the 100 channels can be combined into one A vector matrix of 100 rows*M columns.
  • the expert features may also have 100 channels.
  • the expert features of the 100 channels may be combined into a vector matrix with 100 rows*M columns.
  • a fusion feature of 200 channels can be obtained by splicing depth features and expert features.
  • the fusion feature of 200 channels is a vector matrix of 200 rows*M columns.
  • the fusion feature with 200 channels integrates the information of depth feature and expert feature.
  • a fusion processor can be provided, and the fusion processor can obtain the fusion feature of the image to be recognized according to the fusion depth feature and the expert feature; for example, the central processing unit (CPU), image processor (GPU), A tensor processor (TPU), a field programmable logic gate array (FPGA), or other forms of processing units with data processing capabilities and/or instruction execution capabilities and corresponding computer instructions implement the fusion processor.
  • the processing unit may be a general-purpose processor or a special-purpose processor, and may be a processor based on the X86 or ARM architecture.
  • the atrial fibrillation recognition method based on the deep neural network has better adaptive ability to noise. Since the depth features are automatically extracted rather than manually extracted, it can overcome the difficulty of the method based on domain knowledge. It is difficult to accurately distinguish between atrial fibrillation and other types of arrhythmia in the complex situation where various arrhythmia are mixed.
  • the deep neural network-based atrial fibrillation recognition method requires large-scale well-annotated learning samples, which is very difficult. Therefore, in the embodiments of the present disclosure, expert features that are not strongly dependent on the amount of well-annotated samples will be used.
  • Atrial fibrillation recognition method based on domain knowledge or feature engineering
  • a more reliable prediction of atrial fibrillation even when the number of well-labeled learning samples is insufficient Accuracy is incorporated into the modeling process to integrate deep features and expert features, so that it can still provide a more reliable prediction of atrial fibrillation even when the number of well-labeled learning samples is insufficient Accuracy.
  • FIG. 4 is a flowchart of another image processing method provided by at least one embodiment of the present disclosure. As shown in FIG. 4, based on the example shown in FIG. 1, the image processing method further includes step S150.
  • the principal component analysis method, random projection method and sequence autoencoder are used at least one of the unsupervised learning methods trains an unsupervised feature extractor, learns automatic representation and extraction of unsupervised features, and extracts transformed low-dimensional data as unsupervised features.
  • unsupervised learning methods Unlike supervised learning methods that build models on labeled data, unsupervised learning methods only build models on unlabeled data. Although due to the lack of labeled data, unsupervised learning features are not as effective as supervised learning features, but in practical applications, it may be difficult to obtain labeled data, so automatic learning methods for unsupervised features play an important role in this scenario. The role of.
  • the unsupervised feature extractor can be obtained by using principal component analysis method, random projection method or sequential autoencoder training.
  • the principal component analysis Principal Component Analysis, PCA
  • PCA Principal Component Analysis
  • Sequence to Sequence Autoencoder is a variant of Autoencoder (AE), which replaces the fully connected layer in the encoder and decoder with a cyclic layer.
  • the sequence auto-encoder first transforms the time series data into the representation of the hidden layer, and then the decoder transforms the representation of the hidden layer into the time series data again, and tries to minimize the distance between the original sequence and the decoded sequence. Finally, the representation of the hidden layer is extracted as unsupervised features.
  • step S130 can be expressed as follows:
  • Step S130 Fusion of depth features, expert features, and unsupervised features to obtain fusion features of the image to be recognized.
  • feature vectors from different sources are fused, for example, depth features, expert features, and unsupervised features can be spliced to obtain fusion features.
  • Global vector and mean vector concatenate at least one of the global vector and the mean vector of the depth feature, at least one of the global vector and the mean vector of the expert feature, and at least one of the global vector and the mean vector of the unsupervised feature to obtain a fusion feature.
  • the specific fusion method can refer to the introduction of the fusion process shown in FIG. 3B, which will not be repeated here.
  • this step S130 may not be limited to the fusion of the above-mentioned features, and may also include the fusion of more other features, which is not limited in the embodiment of the present disclosure.
  • the fusion feature combines the depth feature, the expert feature, and the unsupervised feature to achieve data dimensionality reduction, which can overcome the serious noise pollution problem in the depth feature and the expert feature, thereby further improving the automatic detection of atrial fibrillation. Accuracy.
  • the graphics processing method provided by the embodiments of the present disclosure combines the advantages of multiple different technologies (deep neural network, domain-based knowledge, and unsupervised learning) at the same time. In the case of insufficient, for noisy ECG data, it can still provide more reliable atrial fibrillation prediction accuracy.
  • classifying the image to be recognized according to the fusion feature of the image to be recognized includes: judging whether the image to be recognized includes atrial fibrillation features according to the fusion feature of the image to be recognized.
  • the fusion feature acquired in the above-mentioned embodiment (for example, the fusion feature acquired in FIG. 3B or the fusion feature acquired in FIG. 5) is input into the atrial fibrillation detector to realize atrial fibrillation recognition and detection.
  • the atrial fibrillation detector may be a neural network classifier or an SVM classifier, etc., which is not limited in the embodiment of the present disclosure.
  • a classification processor can be provided, and the classification processor can classify the image to be recognized according to the fusion characteristics of the image to be recognized; for example, the central processing unit (CPU), image processor (GPU), or tensor processor can also be used to classify the image to be recognized.
  • CPU central processing unit
  • GPU image processor
  • tensor processor can also be used to classify the image to be recognized.
  • TPU Field Programmable Logic Gate Array
  • FPGA Field Programmable Logic Gate Array
  • the flow of the image processing method may include more or fewer operations, and these operations may be executed sequentially or in parallel.
  • the flow of the image processing method described above includes multiple operations appearing in a specific order, it should be clearly understood that the order of the multiple operations is not limited.
  • the image processing method described above may be executed once, or may be executed multiple times according to predetermined conditions.
  • the foregoing graphics processing method can be implemented by the image processing system shown in FIG. 1B.
  • the image processing system 10 may include a user terminal 11, a network 12, a server 13, and a database 14.
  • the user terminal 11 may be, for example, the computer 11-1 and the mobile phone 11-2 shown in FIG. 1B. It is understandable that the user terminal 11 may be any other type of electronic device capable of performing data processing, which may include, but is not limited to, a desktop computer, a notebook computer, a tablet computer, a smart phone, a smart home device, a wearable device, and in-vehicle electronics. Equipment, monitoring equipment, etc. The user terminal may also be any equipment provided with electronic equipment, such as vehicles, robots, and so on.
  • the user terminal provided according to the embodiment of the present disclosure may be used to receive the image to be recognized, and use the method provided by the embodiment of the present disclosure to realize image recognition and classification.
  • the user terminal 11 may collect an image to be recognized through an image acquisition device (not shown in the figure, such as a camera, a video camera, etc.) provided on the user terminal 11.
  • the user terminal 11 may also receive the image to be recognized from an independently set image acquisition device.
  • the user terminal 11 may also receive the image to be recognized from the server 13 via the network.
  • the image to be recognized can be a single image or a frame in the video.
  • the image to be recognized is a medical image
  • the user terminal may also receive the image to be recognized with the medical acquisition device.
  • the processing unit of the user terminal 11 may be used to execute the image processing method provided in the embodiments of the present disclosure.
  • the user terminal 11 may use a built-in application program of the user terminal 11 to execute the image processing method.
  • the user terminal 11 may execute the image processing method provided by at least one embodiment of the present disclosure by calling an application program stored externally of the user terminal 11.
  • the server 13 may be a single server or a server group, and each server in the group is connected through a wired or wireless network.
  • a server group can be centralized, such as a data center, or distributed.
  • the server 13 may be local or remote.
  • the database 15 may be a stand-alone device. In other embodiments, the database 15 may also be integrated in at least one of the user terminal 11 and the server 14. For example, the database 15 may be set on the user terminal 11 or on the server 14. For another example, the database 15 may also be distributed, a part of which is set on the user terminal 11, and the other part is set on the server 14.
  • FIG. 6 is a schematic block diagram of an image processing apparatus provided by at least one embodiment of the present disclosure.
  • the image processing device 100 includes a depth feature extractor 110, an expert feature extractor 120, a fusion processor 130 and a classification processor 140.
  • these feature extractors and processors can be implemented by hardware (for example, circuit) modules or software modules, etc. The following embodiments are the same as this, and will not be repeated.
  • a central processing unit CPU
  • an image processor GPU
  • TPU tensor processor
  • FPGA field programmable logic gate array
  • Processing units and corresponding computer instructions implement these processors or extractors.
  • the depth feature extractor 110 is configured to obtain the depth feature of the image to be recognized.
  • the image to be recognized is a medical image.
  • the depth feature extractor 110 can implement step S110, and its specific implementation method can refer to the related description of step S110, which will not be repeated here.
  • the classification processor 140 is configured to classify the image to be recognized according to the fusion feature of the image to be recognized.
  • the classification processor 140 can implement step S140, and the specific implementation method can refer to the related description of step S140, which will not be repeated here.
  • the expert feature extractor 120 is further configured to extract the expert features of the image to be recognized based on empirical formulas, rules, and feature values obtained from medical image data.
  • the category of expert features includes at least one of statistics, morphology, time domain, and frequency domain.
  • the classification processor 140 is further configured to determine whether the image to be identified contains atrial fibrillation features according to the fusion features of the image to be identified.
  • FIG. 7 is a schematic block diagram of another image processing apparatus provided by at least one embodiment of the present disclosure.
  • the image processing apparatus 100 further includes an unsupervised feature extractor 150.
  • the unsupervised feature extractor 150 is further configured to use principal component analysis and random projection before acquiring the unsupervised features of the image to be recognized based on the unsupervised feature extractor.
  • At least one of the method and sequence autoencoder is trained to obtain an unsupervised feature extractor.
  • the fusion processor 130 is also configured to fuse depth features, expert features, and unsupervised features to obtain fusion features of the image to be recognized.
  • the fusion processor 130 is further configured to splice depth features, expert features, and unsupervised features to obtain fusion features.
  • the fusion processor 130 is further configured to perform global pooling operations and average pooling operations on depth features, expert features, and unsupervised features, respectively, to obtain depths respectively.
  • circuits or units may be included, and the connection relationship between the respective circuits or units is not limited, and may be determined according to actual requirements.
  • the specific structure of each circuit is not limited, and may be composed of analog devices according to the circuit principle, or may be composed of digital chips, or be composed in other suitable manners.
  • FIG. 8 is a schematic block diagram of another image processing apparatus provided by at least one embodiment of the present disclosure.
  • the image processing apparatus 200 includes a processor 210, a memory 220, and one or more computer program modules 221.
  • the processor 210 may be a central processing unit (CPU), an image processing unit (GPU), or another form of processing unit with data processing capabilities and/or instruction execution capabilities, and may be a general-purpose processor or a special-purpose processor, and Other components in the image processing apparatus 200 can be controlled to perform desired functions.
  • CPU central processing unit
  • GPU image processing unit
  • Other components in the image processing apparatus 200 can be controlled to perform desired functions.
  • the memory 220 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory.
  • the volatile memory may include random access memory (RAM) and/or cache memory (cache), for example.
  • the non-volatile memory may include read-only memory (ROM), hard disk, flash memory, etc., for example.
  • One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 210 may run the program instructions to implement the functions (implemented by the processor 210) and/or other desired functions in the embodiments of the present disclosure, For example, image processing methods.
  • Various application programs and various data such as depth features, expert features, and various data used and/or generated by the application programs, can also be stored in the computer-readable storage medium.
  • the embodiment of the present disclosure does not provide all the components of the image processing apparatus 200.
  • those skilled in the art can provide and set other unshown component units according to specific needs, and the embodiments of the present disclosure do not limit this.
  • FIG. 9 is a schematic diagram of an electronic device provided by at least one embodiment of the present disclosure.
  • the electronic device 300 includes a central processing unit (CPU) 301, which can be loaded to a random access memory according to a program stored in a read-only memory (ROM) 302 or from a storage device 308 (RAM)
  • the program in 303 executes various appropriate actions and processing.
  • the RAM 303 various programs and data required for the operation of the computer system are also stored.
  • the CPU 301, the ROM 302, and the RAM 303 are connected by this through the bus 304.
  • An input/output (I/O) interface 305 is also connected to the bus 304.
  • the following components are connected to the I/O interface 305: an input device 306 including a keyboard, a mouse, etc.; an output device 307 such as a liquid crystal display (LCD) and a speaker; a storage device 308 including a hard disk; and a storage device 308, such as a LAN card, A communication device 309 of a network interface card such as a modem.
  • the communication device 309 performs communication processing via a network such as the Internet.
  • the driver 310 is also connected to the I/O interface 305 as needed.
  • a removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that the computer program read from it can be installed into the storage device 309 as needed.
  • the electronic device 300 may further include an image acquisition device (not shown in the figure), a peripheral interface (not shown in the figure), and the like.
  • the image acquisition device may include an imaging sensor and a lens
  • the image sensor may be of a CMOS type or a CCD type
  • the lens may include one or more lenses (convex lens or concave lens, etc.).
  • the peripheral interface can be various types of interfaces, such as a USB interface, a lightning interface, and the like.
  • the communication device 309 can communicate with a network and other devices through wireless communication, such as the Internet, an intranet, and/or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and/or a metropolitan area network (MAN).
  • Wireless communication can use any of a variety of communication standards, protocols and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA) , Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g. based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and/or IEEE 802.11n standards), voice transmission based on Internet protocol (VoIP), Wi-MAX, protocols used for e-mail, instant messaging and/or short message service (SMS), or any other suitable communication protocol.
  • GSM Global System for Mobile Communications
  • EDGE Enhanced Data GSM Environment
  • W-CDMA Wideband Code Division Multiple Access
  • CDMA Code Division Multiple Access
  • TDMA Time Division Multiple Access
  • Wi-Fi e.g. based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and/or IEEE 802.11n standards
  • VoIP Internet protocol
  • Wi-MAX
  • the electronic device can be any device such as a mobile phone, a tablet computer, a notebook computer, an e-book, a game console, a television, a digital photo frame, a navigator, etc., or can be any combination of electronic devices and hardware. This is not limited.
  • the electronic device may be a medical electronic device.
  • the image acquisition device may be used to acquire an image to be recognized, for example, a medical image.
  • the medical images mentioned here can be, for example, medical images collected by CT, MRI, ultrasound, X-ray, radionuclide imaging (such as SPECT, PET), etc., or can be displays such as electrocardiogram, electroencephalogram, optical photography, etc. Images of human body physiological information.
  • the medical electronic equipment may be any medical imaging equipment such as CT, MRI, ultrasound, X-ray equipment.
  • the image acquisition device may be implemented as the imaging unit of the above-mentioned medical imaging device, and the image processing device 100/200 may be implemented by the internal processing unit (for example, a processor) of the medical imaging device.
  • FIG. 10 is a schematic diagram of a storage medium provided by at least one embodiment of the present disclosure.
  • the storage medium 400 does not store computer-readable instructions 401.
  • the image processing method provided in any embodiment of the present disclosure can be executed.
  • the storage medium may be any combination of one or more computer-readable storage media.
  • one computer-readable storage medium contains computer-readable program code for extracting depth features in the image to be recognized
  • another computer-readable storage medium contains computer-readable program codes that fuse the depth feature and expert feature of the image to be recognized to obtain the fusion feature.
  • the computer can execute the program code stored in the computer storage medium, and execute, for example, the image processing method provided in any embodiment of the present disclosure.
  • the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), Portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the foregoing storage media may also be other suitable storage media.
  • RAM random access memory
  • ROM read-only memory
  • EPROM erasable programmable read-only memory
  • CD-ROM Portable compact disk read-only memory
  • flash memory or any combination of the foregoing storage media may also be other suitable storage media.

Abstract

La présente invention concerne un procédé de traitement d'image, un appareil de traitement d'image et un support de stockage. L'appareil de traitement d'image (100) comprend : un extracteur de caractéristique de profondeur (110) configuré pour acquérir une caractéristique de profondeur d'une image à identifier, l'image à identifier étant une image médicale; un extracteur de caractéristique d'expert (120) configuré pour acquérir une caractéristique d'expert de l'image à identifier; un processeur de fusion (130) configuré pour fusionner la caractéristique de profondeur avec la caractéristique d'expert pour acquérir une caractéristique fusionnée de l'image à identifier; et un processeur de classification (140) configuré pour classer, en fonction de la caractéristique fusionnée de l'image à identifier, l'image à identifier.
PCT/CN2020/113114 2019-09-29 2020-09-03 Procédé de traitement d'image, appareil de traitement d'image, et support de stockage WO2021057423A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910935208.5A CN110664395B (zh) 2019-09-29 2019-09-29 图像处理方法、图像处理装置以及存储介质
CN201910935208.5 2019-09-29

Publications (1)

Publication Number Publication Date
WO2021057423A1 true WO2021057423A1 (fr) 2021-04-01

Family

ID=69080116

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/113114 WO2021057423A1 (fr) 2019-09-29 2020-09-03 Procédé de traitement d'image, appareil de traitement d'image, et support de stockage

Country Status (2)

Country Link
CN (1) CN110664395B (fr)
WO (1) WO2021057423A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117745723A (zh) * 2024-02-20 2024-03-22 常熟理工学院 芯片引线键合质量检测方法、系统及存储介质

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110664395B (zh) * 2019-09-29 2022-04-12 京东方科技集团股份有限公司 图像处理方法、图像处理装置以及存储介质
CN113384277B (zh) * 2020-02-26 2022-09-20 京东方科技集团股份有限公司 心电数据的分类方法及分类系统
CN111436926B (zh) * 2020-04-03 2021-04-20 山东省人工智能研究院 基于统计特征和卷积循环神经网络的房颤信号检测方法
CN112971800B (zh) * 2021-01-27 2023-01-03 东南大学 基于深度学习的心电早搏信号处理方法、模块和装置
CN113317797B (zh) * 2021-04-05 2022-11-08 宁波工程学院 一种结合医疗领域知识的可解释心律失常分类方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017165801A1 (fr) * 2016-03-24 2017-09-28 The Regents Of The University Of California Classification de cancer sur la base d'apprentissage profond au moyen d'une infrastructure de classification hiérarchique
CN109063712A (zh) * 2018-06-22 2018-12-21 哈尔滨工业大学 一种基于超声图像的多模型肝脏弥漫性疾病智能诊断方法及系统
CN109171707A (zh) * 2018-10-24 2019-01-11 杭州电子科技大学 一种智能心电图分类方法
CN109598709A (zh) * 2018-11-29 2019-04-09 东北大学 基于融合深度特征的乳腺辅助诊断系统及方法
CN110226921A (zh) * 2019-06-27 2019-09-13 广州视源电子科技股份有限公司 心电信号检测分类方法、装置、电子设备和存储介质
CN110664395A (zh) * 2019-09-29 2020-01-10 京东方科技集团股份有限公司 图像处理方法、图像处理装置以及存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8977350B2 (en) * 2011-03-17 2015-03-10 Medtronic, Inc. Methods for ectopy rejection for atrial fibrillation detection based on ventricular cycle lengths
CN107811626A (zh) * 2017-09-10 2018-03-20 天津大学 一种基于一维卷积神经网络和s变换的心律失常分类方法
CN108256452A (zh) * 2018-01-06 2018-07-06 天津大学 一种基于特征融合的ecg信号分类的方法
CN108714026B (zh) * 2018-03-27 2021-09-03 杭州电子科技大学 基于深度卷积神经网络和在线决策融合的细粒度心电信号分类方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017165801A1 (fr) * 2016-03-24 2017-09-28 The Regents Of The University Of California Classification de cancer sur la base d'apprentissage profond au moyen d'une infrastructure de classification hiérarchique
CN109063712A (zh) * 2018-06-22 2018-12-21 哈尔滨工业大学 一种基于超声图像的多模型肝脏弥漫性疾病智能诊断方法及系统
CN109171707A (zh) * 2018-10-24 2019-01-11 杭州电子科技大学 一种智能心电图分类方法
CN109598709A (zh) * 2018-11-29 2019-04-09 东北大学 基于融合深度特征的乳腺辅助诊断系统及方法
CN110226921A (zh) * 2019-06-27 2019-09-13 广州视源电子科技股份有限公司 心电信号检测分类方法、装置、电子设备和存储介质
CN110664395A (zh) * 2019-09-29 2020-01-10 京东方科技集团股份有限公司 图像处理方法、图像处理装置以及存储介质

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
HONG SHENDA; WU MENG; ZHOU YUXI; WANG QINGYUN; SHANG JUNYUAN; LI HONGYAN; XIE JUNQING: "ENCASE: An ENsemble ClASsifiEr for ECG classification using expert features and deep neural networks", 2017 COMPUTING IN CARDIOLOGY (CINC), CCAL, 24 September 2017 (2017-09-24), pages 1 - 4, XP033343673, DOI: 10.22489/CinC.2017.178-245 *
SHENDA HONG; YUXI ZHOU; MENG WU; JUNYUAN SHANG; QINGYUN WANG; HONGYAN LI; JUNQING XIE: "Combining deep neural networks and engineered features for cardiac arrhythmia detection from ECG recordings", PHYSIOLOGICAL MEASUREMENT., INSTITUTE OF PHYSICS PUBLISHING, BRISTOL, GB, vol. 40, no. 5, 4 June 2019 (2019-06-04), GB, pages 054009, XP020336415, ISSN: 0967-3334, DOI: 10.1088/1361-6579/ab15a2 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117745723A (zh) * 2024-02-20 2024-03-22 常熟理工学院 芯片引线键合质量检测方法、系统及存储介质

Also Published As

Publication number Publication date
CN110664395A (zh) 2020-01-10
CN110664395B (zh) 2022-04-12

Similar Documents

Publication Publication Date Title
WO2021057423A1 (fr) Procédé de traitement d'image, appareil de traitement d'image, et support de stockage
CN109409416B (zh) 特征向量降维方法和医学图像识别方法、装置及存储介质
Wang et al. Arrhythmia classification algorithm based on multi-head self-attention mechanism
Kwaśniewska et al. Deep features class activation map for thermal face detection and tracking
Chen et al. Smart heart monitoring: Early prediction of heart problems through predictive analysis of ECG signals
CN111772619B (zh) 一种基于深度学习的心搏识别方法、终端设备及存储介质
Xiang et al. ECG-based heartbeat classification using two-level convolutional neural network and RR interval difference
KR20210005206A (ko) 이미지 처리 방법, 전자 기기 및 저장 매체
Sepahvand et al. A novel multi-lead ECG personal recognition based on signals functional and structural dependencies using time-frequency representation and evolutionary morphological CNN
CN111657925A (zh) 基于机器学习的心电信号分类方法、系统、终端以及存储介质
Yang et al. Proegan-ms: A progressive growing generative adversarial networks for electrocardiogram generation
CN115470828A (zh) 基于卷积与自注意力机制的多导联心电图分类识别方法
Ullah et al. An end-to-end cardiac arrhythmia recognition method with an effective densenet model on imbalanced datasets using ecg signal
Dadiz et al. Detecting depression in videos using uniformed local binary pattern on facial features
CN115530788A (zh) 基于自注意力机制的心律失常分类方法
CN115281688A (zh) 一种基于多模态深度学习的心脏肥大多标签检测系统
US20230100365A1 (en) Automatic classification of healthy and disease conditions from images or digital standard 12-lead ECGs
Kwaśniewska et al. Real-time facial features detection from low resolution thermal images with deep classification models
CN111407261A (zh) 生物信号的周期信息的测量方法及装置、电子设备
CN115470832A (zh) 一种基于区块链的心电信号数据处理方法
Yang et al. Heart rate estimation from facial videos based on convolutional neural network
CN110786847B (zh) 心电信号的建库方法和分析方法
Lee et al. Comparative neural network based on template cluster for automated abnormal beat detection in electrocardiogram signals
US20230153997A1 (en) Method and system for data extraction from echocardiographic strain graphs
Yang et al. Automatic recognition of coronary artery disease and congestive heart failure using a multi-granularity cascaded hybrid network

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20868847

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20868847

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 24.05.22)

122 Ep: pct application non-entry in european phase

Ref document number: 20868847

Country of ref document: EP

Kind code of ref document: A1