WO2020259666A1 - 图像分类方法、装置、设备、存储介质和医疗电子设备 - Google Patents

图像分类方法、装置、设备、存储介质和医疗电子设备 Download PDF

Info

Publication number
WO2020259666A1
WO2020259666A1 PCT/CN2020/098407 CN2020098407W WO2020259666A1 WO 2020259666 A1 WO2020259666 A1 WO 2020259666A1 CN 2020098407 W CN2020098407 W CN 2020098407W WO 2020259666 A1 WO2020259666 A1 WO 2020259666A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
target
feature
image feature
target image
Prior art date
Application number
PCT/CN2020/098407
Other languages
English (en)
French (fr)
Inventor
揭泽群
赵波
冯佳时
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Priority to EP20831068.0A priority Critical patent/EP3992851A4/en
Priority to KR1020217028280A priority patent/KR102605969B1/ko
Priority to JP2021548679A priority patent/JP7297081B2/ja
Publication of WO2020259666A1 publication Critical patent/WO2020259666A1/zh
Priority to US17/459,920 priority patent/US11900647B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G06T7/0014—Biomedical image inspection using an image reference approach
    • G06T7/0016—Biomedical image inspection using an image reference approach involving temporal comparison
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30036—Dental; Teeth
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30041—Eye; Retina; Ophthalmic
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30061—Lung
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30068—Mammography; Breast
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images

Definitions

  • This application relates to the field of image processing, in particular to an image classification method, device, equipment, storage medium, medical electronic equipment, and an image processing method.
  • Image classification refers to automatically classifying input images into a set of predefined categories according to certain classification rules. For example, according to the semantic information contained in the image, the input image can be classified into objects and scenes. For example, it is possible to identify preset target objects contained in the input image and classify them according to the identified objects. For another example, images with similar content can also be classified into the same category according to semantic information in the input image.
  • the purpose of this application is to provide an image classification method, device, equipment, storage medium, medical electronic equipment, and image processing method.
  • an image classification method executed by an electronic device, the method comprising: receiving a target image and a reference image about the target image, wherein the target image is a medical image; using the same Method to determine the first image feature of the target image and the second image feature of the reference image; fuse the first image feature and the second image feature to determine the image feature to be classified; and use the image feature to be classified.
  • the step of determining the probability that the target image belongs to a preset category includes: using the features of the image to be classified to obtain a vector of multiple dimensions, and the elements in the vector respectively indicate that the target image and the reference image belong to the preset category.
  • the confidence score of the category determines the probability that the target image belongs to the preset category according to the confidence score that the target image belongs to the preset category.
  • an image classification device including: a receiving unit configured to receive a target image and a reference image related to the target image, wherein the target image is a medical image; and an image feature determination unit , Configured to use the same method to determine the first image feature of the target image and the second image feature of the reference image; a fusion unit configured to fuse the first image feature and the second image feature to determine the to-be-classified Image features; and a classification result generating unit configured to use the features of the image to be classified to determine the probability that the target image belongs to a preset category, wherein the classification result generating unit is configured to use the features of the image to be classified to obtain multiple dimensions
  • the elements in the vector respectively represent the confidence scores that the target image and the reference image belong to a preset category. According to the confidence scores that the target image belongs to the preset category, it is determined that the target image belongs to the preset category Probability.
  • an image processing method executed by an electronic device comprising: receiving a target image; using a first neural network to determine a first target image feature of the target image; Second, the neural network determines the second target image feature of the target image; determines the first image processing result and the second image processing result of the target image according to the first target image feature and the second target image feature; fusion The first image processing result and the second image processing result determine the image processing result of the target image, wherein the first neural network and the second neural network are different networks trained by the same training method
  • the first neural network is trained using a first training set
  • the second neural network is trained using a second training set
  • the positive training images included in the first training set and the second training set are The negative sample ratio is different.
  • a medical electronic device including: an image acquisition unit configured to acquire a target image and a reference image related to the target image, wherein the target image is a medical image; image feature determination Unit configured to determine the first image feature of the target image and the second image feature of the reference image in the same manner; the fusion unit configured to fuse the first image feature and the second image feature to determine the Classification image features; and a classification result generation unit configured to determine the probability that the target image belongs to a preset category using the characteristics of the image to be classified, wherein the classification result generation unit is configured to use the characteristics of the image to be classified to obtain a plurality of A vector of dimensions, the elements in the vector respectively represent the confidence scores that the target image and the reference image belong to a preset category, and according to the confidence scores that the target image belongs to the preset category, it is determined that the target image belongs to the preset category The probability.
  • an image classification device including a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the The processor executes the image classification method as described above.
  • a computer-readable storage medium having instructions stored thereon, and when the instructions are executed by a processor, the processor executes the image classification method described above.
  • Fig. 1 shows an exemplary scene diagram of the image processing system according to the present application
  • Fig. 2 shows a schematic flowchart of an image classification method according to an embodiment of the present application
  • Figure 3A shows a schematic process of the image classification method according to the present application
  • Figure 3B shows the cc-bit image and mlo-bit image of the left breast of the human body, and the cc-bit image and mlo-bit image of the right breast;
  • Fig. 4 shows a schematic flowchart of an image processing method according to an embodiment of the present application
  • Fig. 5 shows a schematic block diagram of an image classification device according to an embodiment of the present application
  • Fig. 6 shows a schematic block diagram of a medical electronic device according to an embodiment of the present application.
  • Fig. 7 shows the architecture of a computing device according to an embodiment of the present application.
  • processing is usually performed only for the target image of interest.
  • image classification processing is usually performed only for the breast images on the side of interest.
  • the medical images of different individuals differ greatly in physiological characteristics such as tissue density and fat thickness, the visual effects of the breast medical images obtained for different individuals are also very different. Therefore, it is difficult to make accurate judgments if only image processing is performed on the medical image on the side of interest.
  • the comparison information of the medical images of the bilateral organs of the same person is considered when performing image classification, more accurate image classification results and disease screening results can be obtained.
  • Fig. 1 shows an exemplary scene diagram of the image classification system according to the present application.
  • the image classification system 100 may include a user terminal 110, a network 120, a server 130 and a database 140.
  • the user terminal 110 may be, for example, the computer 110-1 and the mobile phone 110-2 shown in FIG. 1. It is understandable that, in fact, the user terminal can be any other type of electronic device that can perform data processing, which can include but is not limited to desktop computers, laptops, tablets, smart phones, smart home devices, wearable devices, Car electronic equipment, monitoring equipment, etc.
  • the user terminal may also be any equipment provided with electronic equipment, such as vehicles, robots, and so on.
  • the user terminal provided according to the present application can be used to receive images to be processed, and use the method provided in the present application to realize image classification, thereby realizing disease screening.
  • the user terminal may collect the image to be processed through an image acquisition device (such as a camera, a video camera, etc.) provided on the user terminal.
  • the user terminal may also receive the image to be processed from an independently set image acquisition device.
  • the user terminal may also receive the image to be processed from the server via the network.
  • the image to be processed described here can be a single image or a frame in the video.
  • the user terminal may also receive the image to be processed from the medical acquisition device.
  • the medical image mentioned here can be, for example, CT (Computed Tomography), MRI (Magnetic Resonance Imaging, magnetic resonance imaging), ultrasound, X-ray, radionuclide imaging (such as SPECT (Single-Photon Emission computed Tomography) Medical images collected by methods such as single photon emission computed tomography, PET (Positron Emission Tomography, positron emission tomography), etc., can also be images that display human physiological information such as electrocardiogram, electroencephalogram, and optical photography.
  • the processing unit of the user terminal may be used to execute the image classification method provided in this application.
  • the user terminal may use an application built in the user terminal to execute the image classification method.
  • the user terminal may execute the image classification method provided in this application by calling an application program stored externally of the user terminal.
  • the user terminal sends the received image to be processed to the server 130 via the network 120, and the server 130 executes the image classification method.
  • the server 130 may use an application built in the server to execute the image classification method.
  • the server 130 may execute the image classification method by calling an application program stored outside the server.
  • the network 120 may be a single network, or a combination of at least two different networks.
  • the network 120 may include, but is not limited to, one or a combination of several of a local area network, a wide area network, a public network, and a private network.
  • the server 130 may be a single server or a server group, and each server in the group is connected through a wired or wireless network.
  • a server group can be centralized, such as a data center, or distributed.
  • the server 130 may be local or remote.
  • the database 140 can generally refer to a device having a storage function.
  • the database 140 is mainly used to store various data used, generated and output from the work of the user terminal 110 and the server 130.
  • the database 140 may be local or remote.
  • the database 140 may include various memories, such as Random Access Memory (RAM), Read Only Memory (ROM), etc.
  • RAM Random Access Memory
  • ROM Read Only Memory
  • the storage devices mentioned above are just a few examples, and the storage devices that can be used in the system are not limited to these.
  • the database 140 may be connected or communicated with the server 130 or a part thereof via the network 120, or directly connected or communicated with the server 130, or a combination of the above two methods.
  • the database 140 may be a stand-alone device. In other embodiments, the database 140 may also be integrated in at least one of the user terminal 110 and the server 130. For example, the database 140 may be set on the user terminal 110 or on the server 130. For another example, the database 140 may also be distributed, a part of which is set on the user terminal 110 and the other part is set on the server 130.
  • Fig. 2 shows a schematic flowchart of an image processing method according to an embodiment of the present application.
  • a target image and at least one reference image about the target image may be received.
  • the reference image may be an image of the same type as the target image.
  • the reference image may contain other target objects that are the same or the same type as the target object in the target image.
  • the target image may be a medical image.
  • the reference image may be a similar medical image of an organ on the other side of the same person.
  • the organs mentioned here can be any organs with two or more numbers in the human body such as breast, eyes, lungs, and teeth.
  • the target image mentioned here can also be any other type of image except medical images, as long as the reference image can contain the target object in the target image.
  • Target objects of the same type are sufficient.
  • the target image may be a face image.
  • the reference image may be a face image taken at other times (for example, in a different background, under different lighting, and at different ages).
  • the target image can be any animal or plant except humans.
  • the target image may include multiple images for the same target object.
  • the target image may include images of the target object obtained from at least two angles.
  • the target image may include a mammography detection image of the human breast taken at a CC (Craniocaudal) perspective and a mammography detection image of the human breast taken at a Mediolateral-Oblique (MLO) perspective.
  • MLO Mediolateral-Oblique
  • the target image may also include target objects acquired by at least two different devices.
  • the target image may include an image of the human breast acquired by an X-ray device and an image of the human breast acquired by an MRI device. It is understandable that when the target image includes other types of target objects, those skilled in the art can also arbitrarily set the manner of shooting the target object to obtain other target images obtained by different devices. For example, it is also possible to obtain a target image containing the target object through at least two cameras with different parameters.
  • the reference image may include multiple reference images, respectively corresponding to each of the multiple target images. Each reference image is obtained in the same way as the corresponding target image.
  • the first image feature of the target image and the second image feature of the reference image may be determined.
  • the first image feature of the target image and the second image feature of the reference image can be determined in the same manner.
  • a first neural network including at least one convolution layer may be used to perform convolution processing on the target image to obtain the first image feature. Further, the first neural network may be used to perform convolution processing on the reference image to obtain a second image feature. In other words, a neural network sharing parameters can be used to process the target image and the reference image.
  • the first image feature and the second image feature may each include a plurality of image features.
  • the first neural network may be any neural network that can obtain image features from an image.
  • the first neural network may be any network including at least one convolutional layer, such as any one of the Inception series network (such as Googlenet, etc.), the VGG series network, the Resnet series network, etc., or at least a part of any one of the foregoing networks.
  • At least one of the color feature, texture feature, shape feature, spatial relationship feature and other features in the target image can also be extracted as the first image feature. Further, the same method can be used to extract the features in the reference image as the second image feature.
  • step S206 the first image feature and the second image feature may be combined to determine the image feature to be classified.
  • the first image feature and the second image feature may be spliced to determine the image feature to be classified.
  • the first image feature and the second image feature may have multiple channels.
  • the first image feature can be a tensor of size H1*W1*C1, where H1 can be the size of the first image feature in the first direction (for example, the length direction), and W1 can be the size of the first image feature in the second
  • H1 and W1 may be the size in units of the number of pixels
  • C1 may be the number of channels of the first image feature.
  • the second image feature can be a tensor of size H2*W2*C2, where H2 can be the size of the second image feature in the first direction (for example, the length direction), and W2 can be the second image feature in the second direction ( For example, the size in the width direction, H2 and W2 may be the size in units of the number of pixels, and C2 may be the number of channels of the second image feature. Wherein C1, C2 are integers greater than 1.
  • fusing the first image feature and the second image feature to determine the image feature to be classified may include splicing the first image feature and the second image feature to determine the image feature to be classified .
  • the first image feature may have 1024 channels
  • the second image feature may also have 1024 channels.
  • a 2048 channel image feature to be classified can be obtained.
  • the image feature to be classified with 2048 channels is fused with the information of the first image feature and the second image feature, that is, the image information of the target image and the reference image are fused.
  • each element at the corresponding position in the corresponding channel of the first image feature and the second image feature may also be added to determine the image feature to be classified into which the image information of the target image and the reference image are fused.
  • step S208 the feature of the image to be classified obtained in step S206 may be used to determine the probability that the target image belongs to a preset category. For example, the fused image features to be classified are used to predict whether the left and right breasts are positive for breast cancer.
  • the first fully connected network may be used to process the features of the image to be classified to obtain the probability that the target image belongs to a preset category.
  • the first fully connected network may be configured such that the first fully connected network outputs a vector with multiple dimensions, and each element in the vector represents the confidence score that the target image and the reference image belong to a preset category.
  • the preset categories may include health categories and disease categories.
  • the target image may be a medical image of the breast on the left side of the human body
  • the reference image may be a medical image of the breast on the right side of the human body.
  • the vector output by the first fully connected network can be 4-dimensional. The four-dimensional elements of the vector respectively represent the confidence scores that the medical image of the left breast belongs to the health category and the disease category, and the medical image of the right breast belongs to the health category and the disease category.
  • the preset category may be any possible classification categories such as predefined animal categories, plant categories, and scene categories.
  • the first fully connected network can be configured to output vectors of predetermined dimensions. For example, when the number of input images is m and the number of preset categories is n, the vector output by the first fully connected network may have m n dimensions.
  • the probability that the target image and the reference image belong to a preset category may be determined according to the confidence scores for multiple dimensions output by the first fully connected network.
  • the softmax function can be used to normalize the two confidence scores for the left breast to obtain the medical image of the left breast that belongs to the health category. The probability and the probability that the medical image of the left breast belongs to the disease category. Similarly, the softmax function can be used to obtain the probability that the medical image of the right breast belongs to the healthy category and the probability that the medical image of the right breast belongs to the disease category.
  • the target image when the probability that the target image belongs to the preset category is greater than the preset probability threshold (such as 0.5), the target image can be considered to belong to the preset category.
  • the preset probability threshold such as 0.5
  • the target image can be classified according to the image information of the reference image.
  • the image information of the reference image and the target image can be fused during the image classification process, and the target image can be determined to belong to the predetermined image based on the image characteristics of the image information of the fused target image and the reference image. Set the probability of the category to achieve a more accurate classification of the target image.
  • the target image and the reference image are medical images, the accuracy of disease screening is improved.
  • the above-mentioned image classification method provided by the embodiments of the present application when a breast cancer patient has breast cancer and the contrast information of bilateral breast images is significantly different, it can be determined that one side of the person is positive for breast cancer with a high probability.
  • Fig. 3A shows a schematic process of the image classification method according to the present application.
  • the input image may include the cc-bit image and the mlo-bit image of the left breast of the human body, and the cc-bit image and the mlo-bit image of the right breast.
  • Figure 3B shows the cc-bit image and mlo-bit image of the left breast of the human body, and the cc-bit image and mlo-bit image of the right breast.
  • the above four input images can be respectively input to a googlenet network with shared parameters, so as to obtain the respective vectorized image characteristics of each image.
  • the output result of any layer in the googlenet network (such as a certain later layer, such as the pool5/7 ⁇ 7_s1 layer in the googlenet network) can be selected as the vectorized image feature of the image.
  • the pool5/7 ⁇ 7_s1 layer in the googlenet network can be used to generate a 1024-dimensional vectorized feature for each input image.
  • a 4096-dimensional fusion feature F [f cc l ,f mlo l ,f cc R ,f mlo R ].
  • a 4-dimensional vector can be obtained, representing the confidence scores s + l , s - l , s + R of the left breast and right breast respectively belonging to the health category and the disease category. , S - R .
  • the softmax function can be used to normalize s + l and s - l to obtain the probability p + l and p - l that the left breast belongs to the health category and the disease category.
  • the softmax function can be used to normalize s + R and s - R to obtain p + R and p - R of the probability that the right breast belongs to the health category and the disease category, where exp() is the exponential operation function .
  • the probability that the target image and the reference image belong to the health category and the disease category can be obtained using the process shown in FIG. 3A.
  • the probability p + l that the medical image of the left breast belongs to the healthy category is greater than the preset probability threshold (such as 0.5), it can be considered that the medical image of the left breast belongs to the healthy category.
  • the right breast can be determined The category to which the medical image belongs.
  • the image classification method provided by the embodiments of the present application can be used to identify whether the mammography scan of a patient suspected of breast cancer contains breast cancer. For example, based on the network structure used for the comparison of the left and right breasts, the multi-view scans of the left and right breasts can be received at the same time, and the features of each scan of each breast can be extracted separately, and then feature fusion is performed, and the fused features It is also used to predict whether the left and right breasts are positive for breast cancer.
  • the first neural network and the first fully connected network may be trained in the following manner.
  • a first training set of the first neural network may be determined, wherein the first training set includes a plurality of training images.
  • the multiple training images are images of the same type as the target image.
  • the multiple training images included in the first training set may be different breasts that are known to conform to the preset type.
  • the first training set may include a first training image.
  • a first reference training image for the first training image can be determined.
  • the first training image is a medical image of the left breast
  • the first reference training image may be a medical image of the right breast.
  • the first neural network may be used to perform convolution processing on the first training image and the first reference training image to obtain the first training image feature and the second training image feature.
  • the training image feature to be classified may be determined according to the first training image feature and the second training image feature.
  • the feature of the training image to be classified is fused with image information of the first training image and the first training reference image.
  • a first fully connected network may be used to process the characteristics of the training image to be classified to determine the probability that the first training image belongs to a preset category.
  • the parameters of the first neural network and the first fully connected network can be adjusted so that the probability that the first training image belongs to the preset category and the The loss between the real categories to which the first training image belongs is the smallest.
  • the cross-entropy loss function may be used to calculate the loss between the probability that the first training image belongs to the preset category and the real category to which the first training image belongs.
  • the first fully connected network can output the probability that the left breast image belongs to the health category and the disease category and the right breast image belongs to the health category and the disease category.
  • the loss of the left breast image can be calculated according to the following formula:
  • the loss of the right breast image can be calculated according to the following formula:
  • the first neural network and the first fully connected network are trained using multiple known types of bilateral breast images, the contrast information of the bilateral breast images can be learned during the training process. Therefore, even In the case of large differences in tissue density, fat thickness and other aspects of breast scans of different individuals, the first neural network and first fully connected network after training can also be based on the breast scans to determine whether there is breast cancer. Stable and accurate judgment results.
  • the proportion of positive samples and negative samples in training data is not balanced. For example, taking medical images as an example, there are fewer samples belonging to the disease category in the training images, but more samples belonging to the health category, so there is a problem of imbalanced samples.
  • the same method can be used to train at least two network models with different parameters (such as the aforementioned first neural network and first fully connected network) by using sample sets that include different proportions of positive samples and negative samples.
  • Network model to process the input image and determine the probability that the input image belongs to the preset category according to the output results of different networks.
  • the input image includes a target image and a reference image of the target image.
  • the first neural network is trained using a first training set
  • the second neural network is trained using a second training set
  • the positive values in the training images included in the first training set and the second training set are The negative sample ratio is different.
  • Fig. 4 shows a flowchart of a graph processing method according to an embodiment of the present application.
  • a target image can be received.
  • the target image may be a medical image or any other type of image.
  • the first neural network may be used to determine the first target image feature of the target image.
  • a first neural network including at least one convolutional layer may be used to perform convolution processing on the target image to obtain the first target image feature.
  • the first target image feature may include multiple image features.
  • a second neural network may be used to determine the second target image feature of the target image.
  • a second neural network including at least one convolutional layer may be used to perform convolution processing on the target image to obtain the second target image feature.
  • the second image target features may respectively include a plurality of image features.
  • the first neural network and the second neural network may be any neural networks that can obtain image features from an image.
  • the first neural network may be any network including at least one convolutional layer, such as any one of the Inception series network (such as Googlenet, etc.), the VGG series network, the Resnet series network, etc., or at least a part of any one of the foregoing networks.
  • the first neural network and the second neural network are different networks trained using the same training method, the first neural network is trained using the first training set, and the second neural network The network is trained using the second training set, and the ratios of positive and negative samples in the training images included in the first training set and the second training set are different.
  • step S408 the first image processing result and the second image processing result of the target image may be determined according to the first target image feature and the second target image feature.
  • the first image processing result and the second image processing result are of the same type.
  • the first image processing result and the second image processing result may be at least one of an image classification result, an image segmentation result, and a target detection result. This depends on the specific method and training set used by those skilled in the art to train the neural network.
  • step S410 the first image processing result and the second image processing result are merged to determine the image processing result of the target image.
  • the image processing result includes at least one of an image classification result, an image segmentation result, and a target detection result.
  • the proportion of positive samples and negative samples that make the network's positive sample loss and negative sample loss close can be determined through experiments. For example, by calculating the loss function of all positive samples and the loss function of all negative samples in the training set, the positive sample loss and the negative sample loss of the network trained by the training set can be determined. In the following, when the ratio of positive samples to negative samples is 1:2, the network's positive sample loss and negative sample loss are similar as an example to describe the principle of this application.
  • a first training set with a ratio of positive samples to negative samples of 1:1 and a second training set with a ratio of positive samples to negative samples of 1:3 can be determined by sampling.
  • the ratio of positive samples to negative samples is 1:2
  • the loss of positive samples and negative samples of the network are similar. Therefore, when the ratio of positive and negative samples in the training set is changed, the training generated The ratio of positive sample loss and negative sample loss of the network can also be changed accordingly. Therefore, since the positive sample ratio is increased in the first training set whose ratio of positive samples to negative samples is 1:1, the loss of positive samples of the network trained in the first training set is smaller than the loss of negative samples. Similarly, since the second training set with a ratio of positive samples to negative samples of 1:3 increases the ratio of negative samples, the loss of positive samples of the network trained in the second training set is greater than the loss of negative samples.
  • the network trained in the first training set has a better classification effect on positive samples
  • the network trained in the second training set has a better classification effect on negative samples.
  • the output of the network trained by the first training set indicates that the probability of the input image belonging to the preset category is closer to 0 or 1, that is It is easier to distinguish whether the input image belongs to a preset category.
  • the network trained in the first training set since the network trained in the first training set has a poor classification effect on negative samples, for input images of negative samples, the network trained in the first training set will output a probability closer to 0.5, that is, it is less easy to distinguish the input The category to which the image belongs. Based on the above characteristics, by fusing the output results of the network trained in the first training set and the network trained in the second training set, more accurate prediction results can be obtained.
  • the first training set with a ratio of positive samples to negative samples of 1:1, the second training set with a ratio of positive samples to negative samples of 1:3, and the ratio of positive samples to negative samples of 1:2 can also be determined by sampling.
  • the first training set, the second training set, and the third training set can be used to train the first neural network, the second neural network, and the third neural network, respectively.
  • the probability that the left breast medical image and the right breast medical image belong to the health category and the disease category can be determined by fusing (for example, weighted average) the above three network output results obtained by using different training sets. For example, the probability that the left breast medical image and the right breast medical image belong to the health category and the disease category can be determined by the following formula.
  • Is the probability that the breast medical image on the left belongs to the health category Is the probability that the left breast medical image belongs to the disease category
  • Is the probability that the breast medical image on the right belongs to the health category Is the probability that the breast medical image on the right belongs to the health category
  • first target reference image feature and the second target reference image feature of the reference image can be obtained through steps similar to steps S402-S410.
  • the first image feature includes a first target image feature and a second target image feature.
  • the first target image feature is obtained by convolution processing the target image using a first neural network including at least one convolutional layer
  • the second target image feature is obtained by using a second neural network to perform convolution processing on the target image. It is obtained by convolution processing.
  • the second image feature includes a first target reference image feature and a second target reference image feature, wherein the first target reference image feature uses a first neural network including at least one convolutional layer to convolve the reference image
  • the feature of the second target reference image is obtained by performing convolution processing on the reference image by using the second neural network.
  • the first neural network and the second neural network are different networks trained using the same training method, the first neural network is trained using the aforementioned first training set, and the second neural network is trained using the Obtained from the aforementioned second training set.
  • the proportions of positive and negative samples in the training images included in the first training set and the second training set are different. Therefore, the first neural network and the second neural network have different parameters and therefore have different outputs. result.
  • the image features to be classified include a first image feature to be classified and a second image feature to be classified.
  • the first image feature to be classified may be determined by splicing the first target image feature and the first target reference image feature
  • the second image feature to be classified may be determined by splicing the second target image feature.
  • the image feature and the second target reference image feature are determined.
  • the first fully connected network may be used to process the features of the first image to be classified to obtain the first probability that the target image belongs to a preset category.
  • a second fully connected network may be used to process the features of the second image to be classified to obtain the second probability that the target image belongs to a preset category.
  • the probability that the target image belongs to the preset category can be determined by fusing the first probability and the second probability. For example, the probability that the target image belongs to the preset category may be determined according to the weighted average of the first probability and the second probability.
  • the first fully connected network and the second fully connected network are different networks trained using the same training method, wherein the first fully connected network is trained using the first training set, and the second fully connected network
  • the connection network is trained by using the second training set, and the ratio of positive and negative samples in the training images included in the first training set and the second training set are different.
  • Fig. 5 shows a schematic block diagram of an image classification device according to an embodiment of the present application.
  • the image classification device 500 may include a receiving unit 510, an image feature determination unit 520, a fusion unit 530, and a classification result generation unit 540.
  • the receiving unit 510 may be configured to receive a target image and at least one reference image regarding the target image.
  • the reference image may be the same type of image as the target image.
  • the reference image may contain the same target object or other target objects of the same type as in the target image.
  • the target image may be a medical image.
  • the reference image may be a similar medical image of an organ on the other side of the same person.
  • the organs mentioned here can be any organs with two or more numbers in the human body such as breast, eyes, lungs, and teeth.
  • the target image mentioned here can also be any other type of image except medical images, as long as the reference image can contain the same type as the target image.
  • the target audience can be.
  • the target image may be a face image.
  • the reference image may be a face image taken at other times (for example, in a different background, under different lighting, and at different ages).
  • the target image can be any animal or plant except humans.
  • the target image may include multiple images for the same target object.
  • the target image may include images of the target object obtained from at least two angles.
  • the target image may include a mammography detection image of the human breast taken at a CC (Craniocaudal) perspective and a mammography detection image of the human breast taken at a Mediolateral-Oblique (MLO) perspective.
  • MLO Mediolateral-Oblique
  • the target image may also include target objects acquired by at least two different devices.
  • the target image may include an image of the human breast acquired by an X-ray device and an image of the human breast acquired by an MRI device. It is understandable that when the target image includes other types of target objects, those skilled in the art can also arbitrarily set the manner of shooting the target object to obtain other target images obtained by different devices. For example, it is also possible to obtain a target image containing the target object through at least two cameras with different parameters.
  • the reference image may include multiple reference images, respectively corresponding to each of the multiple target images. Each reference image is obtained in the same way as the corresponding target image.
  • the image feature determining unit 520 may be configured to determine the first image feature of the target image and the second image feature of the reference image. For example, the first image feature of the target image and the second image feature of the reference image can be determined in the same manner.
  • a first neural network including at least one convolution layer may be used to perform convolution processing on the target image to obtain the first image feature. Further, the first neural network may be used to perform convolution processing on the reference image to obtain a second image feature. In other words, a neural network sharing parameters can be used to process the target image and the reference image.
  • the first image feature and the second image feature may each include a plurality of image features.
  • the first neural network may be any neural network that can obtain image features from an image.
  • the first neural network may be any network including at least one convolutional layer, such as any one of the Inception series network (such as Googlenet, etc.), the VGG series network, the Resnet series network, etc., or at least a part of any one of the foregoing networks.
  • At least one of the color feature, texture feature, shape feature, spatial relationship feature and other features in the target image can also be extracted as the first image feature. Further, the same method can be used to extract the features in the reference image as the second image feature.
  • the first image feature includes a first target image feature and a second target image feature.
  • the first target image feature is obtained by convolution processing the target image using a first neural network including at least one convolutional layer
  • the second target image feature is obtained by using a second neural network to perform convolution processing on the target image. It is obtained by convolution processing.
  • the first target image feature and the second target image feature may each include a plurality of image features.
  • the second image feature includes a first target reference image feature and a second target reference image feature, wherein the first target reference image feature uses a first neural network including at least one convolutional layer to convolve the reference image
  • the feature of the second target reference image is obtained by performing convolution processing on the reference image by using the second neural network.
  • the first neural network and the second neural network are different networks trained using the same training method, the first neural network is trained using the aforementioned first training set, and the second neural network is trained using the Obtained from the aforementioned second training set.
  • the fusion unit 530 may be configured to fuse the first image feature and the second image feature to determine the image feature to be classified.
  • the first image feature and the second image feature may be spliced to determine the image feature to be classified.
  • fusing the first image feature and the second image feature to determine the image feature to be classified may include splicing the first image feature and the second image feature to determine the image feature to be classified.
  • the image feature to be classified includes a first image feature to be classified and a second image feature to be classified.
  • the first image feature to be classified may be determined by splicing the first target image feature and the first target reference image feature
  • the second image feature to be classified may be determined by splicing the second target image feature.
  • the image feature and the second target reference image feature are determined.
  • the classification result generating unit 540 may be configured to determine the probability that the target image belongs to a preset category by using the features of the image to be classified generated by the fusion unit 530.
  • the first fully connected network may be used to process the features of the image to be classified to obtain the probability that the target image belongs to a preset category.
  • the first fully connected network may be configured such that the first fully connected network outputs a vector with multiple dimensions, and each element in the vector represents the confidence score that the target image and the reference image belong to a preset category.
  • the probability that the target image and the reference image belong to a preset category may be determined according to the confidence scores for multiple dimensions output by the first fully connected network.
  • the softmax function can be used to normalize the two confidence scores for the left breast to obtain the medical image of the left breast that belongs to the health category. The probability and the probability that the medical image of the left breast belongs to the disease category. Similarly, the softmax function can be used to obtain the probability that the medical image of the right breast belongs to the healthy category and the probability that the medical image of the right breast belongs to the disease category.
  • the target image when the probability that the target image belongs to the preset category is greater than the preset probability threshold (such as 0.5), the target image can be considered to belong to the preset category.
  • the preset probability threshold such as 0.5
  • the classification result generating unit 540 may also use the first fully connected network to process the first image feature to be classified to obtain the first probability that the target image belongs to the preset category.
  • a second fully connected network may be used to process the features of the second image to be classified to obtain a second probability that the target image belongs to a preset category.
  • the probability that the target image belongs to a preset category can be determined by fusing the first probability and the second probability. For example, the probability that the target image belongs to a preset category may be determined according to a weighted average of the first probability and the second probability.
  • the target image can be classified according to the image information of the reference image.
  • the image information of the reference image and the target image can be merged in the image classification process, and the target image can be determined as belonging to the preset category according to the image characteristics of the image information of the fused target image and the reference image. Probability, so as to achieve a more accurate classification of the target image.
  • the target image and the reference image are medical images, the accuracy of disease screening is improved.
  • it can also overcome the problem of unbalanced training data proportion in related technologies, and further improve the accuracy of image classification and the accuracy of disease screening.
  • Fig. 6 shows a schematic block diagram of a medical electronic device according to an embodiment of the present application.
  • the medical electronic device 600 may include an image acquisition unit 610, an image feature determination unit 620, a fusion unit 630, and a classification result generation unit 640.
  • the image acquisition unit 610 may be used to acquire a target image and a reference image related to the target image.
  • the medical images mentioned here can be, for example, medical images collected by CT, MRI, ultrasound, X-ray, radionuclide imaging (such as SPECT, PET), etc., or can be displays such as electrocardiogram, electroencephalogram, optical photography, etc. Images of human body physiological information.
  • the image feature determination unit 620, the fusion unit 630, and the classification result generation unit 640 may be implemented as the image feature determination unit 520, the fusion unit 530, and the classification result generation unit 540 shown in FIG. 5, which will not be repeated here.
  • the medical electronic equipment provided in this application may be any medical imaging equipment such as CT, MRI, ultrasound, X-ray equipment.
  • the image acquisition unit 610 may be implemented as the imaging unit of the above-mentioned medical imaging device, and the image feature determination unit 620, the fusion unit 630, and the classification result generation unit 640 may be implemented by an internal processing unit (for example, a processor) of the medical imaging device.
  • FIG. 7 shows the architecture of the computing device.
  • the computing device 700 may include a bus 710, one or at least two CPUs 720, a read only memory (ROM) 730, a random access memory (RAM) 740, a communication port 750 connected to a network, input/output Components 760, hard disk 770, etc.
  • the storage device in the computing device 700 such as a ROM 730 or a hard disk 770, can store various data or files used in the processing and/or communication of the method for detecting targets in a video provided by this application, and programs executed by the CPU. instruction.
  • the computing device 700 may also include a user interface 780.
  • the architecture shown in FIG. 7 is only exemplary. When implementing different devices, one or at least two components in the computing device shown in FIG. 7 may be omitted according to actual needs.
  • the embodiments of the present application can also be implemented as a computer-readable storage medium.
  • the computer-readable storage medium stores computer-readable instructions.
  • the computer-readable instructions are executed by the processor, the method according to the embodiments of the present application described with reference to the above drawings can be executed.
  • the computer-readable storage medium includes, but is not limited to, for example, volatile memory and/or non-volatile memory.
  • the volatile memory may include random access memory (RAM) and/or cache memory (cache), for example.
  • the non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Multimedia (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Databases & Information Systems (AREA)
  • Quality & Reliability (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

一种图像分类方法、装置、设备、存储介质和医疗电子设备。图像分类方法，包括：接收目标图像以及关于目标图像的参考图像，其中目标图像是医学图像；采用相同的方式确定目标图像的第一图像特征和参考图像的第二图像特征；融合第一图像特征和第二图像特征确定待分类图像特征；以及利用待分类图像特征确定目标图像属于预设类别的概率。

Description

图像分类方法、装置、设备、存储介质和医疗电子设备
本申请要求于2019年06月28日提交的申请号为201910573560.9、发明名称为“图像分类方法、装置、设备、存储介质和医疗电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理领域，具体涉及一种图像分类方法、装置、设备、存储介质和医疗电子设备以及图像处理方法。
背景技术
图像分类是指根据一定的分类规则将输入图像自动分到一组预定义类别中。例如，根据图像中包含的语义信息，可以对输入图像进行对象分类、场景分类等。例如，可以识别输入图像中包含的预设的目标对象，并根据识别的对象进行分类。又例如，也可以根据输入图像中的语义信息将具有相似内容的图像划分成相同的类别。
发明内容
本申请的目的是提供一种图像分类方法、装置、设备、存储介质和医疗电子设备以及图像处理方法。
根据本申请的一个方面，提供了一种图像分类方法，由电子设备执行，所述方法包括：接收目标图像以及关于所述目标图像的参考图像，其中所述目标图像是医学图像；采用相同的方式确定所述目标图像的第一图像特征和所述参考图像的第二图像特征；融合所述第一图像特征和所述第二图像特征确定待分类图像特征；以及利用所述待分类图像特征确定所述目标图像属于预设类别的概率，该步骤包括：利用所述待分类图像特征，获得多个维度的向量，该向量中的元素分别表示所述目标图像和所述参考图像属于预设类别的置信分数，根据所述目标图像属于预设类别的置信分数，确定所述目标图像属于预设类别的概率。
根据本申请的另一方面，还提供了一种图像分类装置，包括：接收单元，配置成接收目标图像以及关于所述目标图像的参考图像，其中所述目标图像 是医学图像；图像特征确定单元，配置成采用相同的方式确定所述目标图像的第一图像特征和所述参考图像的第二图像特征；融合单元，配置成融合所述第一图像特征和所述第二图像特征确定待分类图像特征；以及分类结果生成单元，配置成利用所述待分类图像特征确定所述目标图像属于预设类别的概率，其中，分类结果生成单元用于利用所述待分类图像特征，获得多个维度的向量，该向量中的元素分别表示所述目标图像和所述参考图像属于预设类别的置信分数，根据所述目标图像属于预设类别的置信分数，确定所述目标图像属于预设类别的概率。
根据本申请的又一方面，还提供了一种图像处理方法，由电子设备执行，所述方法包括：接收目标图像；利用第一神经网络确定所述目标图像的第一目标图像特征；利用第二神经网络确定所述目标图像的第二目标图像特征；根据所述第一目标图像特征和所述第二目标图像特征确定所述目标图像的第一图像处理结果和第二图像处理结果；融合所述第一图像处理结果和所述第二图像处理结果以确定所述目标图像的图像处理结果，其中，所述第一神经网络与所述第二神经网络是采用相同训练方法训练的不同网络，所述第一神经网络是利用第一训练集训练得到的，所述第二神经网络是利用第二训练集训练得到的，第一训练集和第二训练集的包括的训练图像中的正负样本比例是不同的。
根据本申请的又一方面，还提供了一种医疗电子设备，包括：图像采集单元，配置成采集目标图像以及关于所述目标图像的参考图像，其中所述目标图像是医学图像；图像特征确定单元，配置成采用相同的方式确定所述目标图像的第一图像特征和所述参考图像的第二图像特征；融合单元，配置成融合所述第一图像特征和所述第二图像特征确定待分类图像特征；以及分类结果生成单元，配置成利用所述待分类图像特征确定所述目标图像属于预设类别的概率，其中，分类结果生成单元用于利用所述待分类图像特征，获得多个维度的向量，该向量中的元素分别表示所述目标图像和所述参考图像属于预设类别的置信分数，根据所述目标图像属于预设类别的置信分数，确定所述目标图像属于预设类别的概率。
根据本申请的又一方面，还提供了一种图像分类设备，所述设备包括存储器和处理器，其中所述存储器中存有指令，当利用所述处理器执行所述指令时，使得所述处理器执行如前所述的图像分类方法。
根据本申请的又一方面，还提供了一种计算机可读存储介质，其上存储有指令，所述指令在被处理器执行时，使得所述处理器执行如前所述的图像分类方法。
附图说明
为了更清楚地说明本申请实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员而言，在没有做出创造性劳动的前提下，还可以根据这些附图获得其他的附图。以下附图并未刻意按实际尺寸等比例缩放绘制，重点在于示出本申请的主旨。
图1示出了根据本申请的图像处理系统的示例性的场景图；
图2示出了根据本申请的实施例的一种图像分类方法的示意性的流程图；
图3A示出了根据本申请的图像分类方法的示意性的过程；
图3B中示出了人体左侧乳腺的cc位图像、mlo位图像以及右侧乳腺的cc位图像、mlo位图像；
图4示出了根据本申请的实施例的一种图像处理方法的示意性的流程图；
图5示出了根据本申请实施例的图像分类装置的示意性的框图；
图6示出了根据本申请实施例的医疗电子设备的示意性的框图；以及
图7示出了根据本申请的实施例的计算设备的架构。
具体实施方式
为使本发明实施例的目的、技术方案和优点更加清楚，下面将结合本发明实施例的附图，对本发明实施例的技术方案进行清楚、完整地描述。显然，所描述的实施例是本发明的一部分实施例，而不是全部的实施例。基于所描述的本发明的实施例，本领域普通技术人员在无需创造性劳动的前提下所获得的所有其他实施例，都属于本发明保护的范围。
除非另作定义，此处使用的技术术语或者科学术语应当为本发明所属领域内具有一般技能的人士所理解的通常意义。本申请中使用的“第一”、“第二”以及类似的词语并不表示任何顺序、数量或者重要性，而只是用来区分不同的组成部分。同样，“包括”或者“包含”等类似的词语意指出现该词前面的元件或者物件涵盖出现在该词后面列举的元件或者物件及其等同，而不 排除其他元件或者物件。“连接”或者“相连”等类似的词语并非限定于物理的或者机械的连接，而是可以包括电性的连接，不管是直接的还是间接的。“上”、“下”、“左”、“右”等仅用于表示相对位置关系，当被描述对象的绝对位置改变后，则该相对位置关系也可能相应地改变。
在相关技术使用的图像分类方法中，以医学图像为例，通常只针对感兴趣的目标图像进行处理。例如，针对人体乳腺的医学图像，通常仅针对感兴趣一侧的乳腺图像进行图像分类处理。然而，由于不同人个体的医学图像在组织密度、脂肪厚度等生理特征方面差异很大，导致针对不同人得到的乳腺的医学图像的视觉效果差别也很大。因此，如果仅针对感兴趣一侧的医学图像进行图像处理难以做出精确地判断。然而，如果在进行图像分类时考虑同一人的双侧器官的医学图像的对比信息，则能够得到更为精确的图像分类结果和疾病筛查结果。
图1示出了根据本申请的图像分类系统的示例性的场景图。如图1所示，该图像分类系统100可以包括用户终端110、网络120、服务器130以及数据库140。
用户终端110可以是例如图1中示出的电脑110-1、手机110-2。可以理解的是，事实上，用户终端可以是能够执行数据处理的任何其他类型的电子设备，其可以包括但不限于台式电脑、笔记本电脑、平板电脑、智能手机、智能家居设备、可穿戴设备、车载电子设备、监控设备等。用户终端也可以是设置有电子设备的任何装备，例如车辆、机器人等。
根据本申请提供的用户终端可以用于接收待处理的图像，并利用本申请提供的方法实现图像分类，进而实现疾病的筛查。例如，用户终端可以通过用户终端上设置的图像采集设备(例如照相机、摄像机等)采集待处理的图像。又例如，用户终端也可以从独立设置的图像采集设备接收待处理的图像。再例如，用户终端也可以经由网络从服务器接收待处理的图像。这里所述的待处理的图像可以是单独的图像，也可以是视频中的一帧。在待处理的图像是医学图像的情况下，用户终端也可以从医学采集设备接收待处理的图像。这里所说的医学图像可以是例如通过CT(Computed Tomography，计算机断层扫描)、MRI(Magnetic Resonance Imaging，磁共振成像)、超声、X光、核素显像(如SPECT(Single-Photon Emission computed Tomography，单光子 发射计算机断层扫描)、PET(Positron Emission Tomography，正电子发射断层扫描))等方法采集的医学图像，也可以是例如心电图、脑电图、光学摄影等显示人体生理信息的图像。
在一些实施例中，可以利用用户终端的处理单元执行本申请提供的图像分类方法。在一些实现方式中，用户终端可以利用用户终端内置的应用程序执行图像分类方法。在另一些实现方式中，用户终端可以通过调用用户终端外部存储的应用程序执行本申请提供的图像分类方法。
在另一些实施例中，用户终端将接收的待处理的图像经由网络120发送至服务器130，并由服务器130执行图像分类方法。在一些实现方式中，服务器130可以利用服务器内置的应用程序执行图像分类方法。在另一些实现方式中，服务器130可以通过调用服务器外部存储的应用程序执行图像分类方法。
网络120可以是单个网络，或至少两个不同网络的组合。例如，网络120可以包括但不限于局域网、广域网、公用网络、专用网络等中的一种或几种的组合。
服务器130可以是一个单独的服务器，或一个服务器群组，群组内的各个服务器通过有线的或无线的网络进行连接。一个服务器群组可以是集中式的，例如数据中心，也可以是分布式的。服务器130可以是本地的或远程的。
数据库140可以泛指具有存储功能的设备。数据库140主要用于存储从用户终端110和服务器130工作中所利用、产生和输出的各种数据。数据库140可以是本地的，或远程的。数据库140可以包括各种存储器、例如随机存取存储器(Random Access Memory(RAM))、只读存储器(Read Only Memory(ROM))等。以上提及的存储设备只是列举了一些例子，该系统可以使用的存储设备并不局限于此。
数据库140可以经由网络120与服务器130或其一部分相互连接或通信，或直接与服务器130相互连接或通信，或是上述两种方式的结合。
在一些实施例中，数据库140可以是独立的设备。在另一些实施例中，数据库140也可以集成在用户终端110和服务器130中的至少一个中。例如，数据库140可以设置在用户终端110上，也可以设置在服务器130上。又例如，数据库140也可以是分布式的，其一部分设置在用户终端110上，另一部分设置在服务器130上。
下文中将详细阐述本申请提供的图像处理方法的流程。
图2示出了根据本申请的实施例的一种图像处理方法的示意性的流程图。
如图2所示，在步骤S202中，可以接收目标图像以及关于所述目标图像的至少一个参考图像。在一些实施例中，参考图像可以是与目标图像类型相同的图像。例如，参考图像中可以包含与目标图像中的目标对象相同或同类型的其他目标对象。
在一些实施例中，目标图像可以是医学图像。例如，当目标图像是人体一侧器官的医学图像时，参考图像可以是同一人另一侧器官的同类医学图像。例如，这里所说的器官可以是乳腺、眼睛、肺部、牙齿等任何人体内存在两个或两个以上数量的器官。
可以理解的是，在不脱离本申请公开的原理的情况下，这里所说的目标图像也可以是除医学图像以外的任何其他类型的图像，只要参考图像中可以包含与目标图像中的目标对象的类型相同的目标对象即可。例如，目标图像可以是人脸图像，此时，参考图像可以是在其他时间(例如在不同的背景中、不同的光照下、不同的年龄阶段)拍摄的人脸图像。又例如，目标图像可以是除人以外的任何动物或植物。
在一些实施例中，目标图像可以包括针对同一目标对象的多个图像。
在一些实现方式中，目标图像可以包括从至少两个角度获取的目标对象的图像。例如，目标图像可以包括以上下夹俯视(CC，Craniocaudal)视角拍摄的人体乳腺的钼靶检测图像以及以左右夹侧视(MLO，Mediolateral-Oblique)视角拍摄的人体乳腺的钼靶检测图像。可以理解的是，当目标图像包括其他类型的目标对象时，本领域技术人员也可以任意设置拍摄目标对象的方式，以获得其他通过不同角度获取的目标图像。
在另一些实现方式中，目标图像也可以包括通过至少两个不同设备获取的目标对象。例如，目标图像可以包括通过X射线设备采集的人体乳腺的图像以及通过MRI设备采集的人体乳腺的图像。可以理解的是，当目标图像包括其他类型的目标对象时，本领域技术人员也可以任意设置拍摄目标对象的方式，以获得其他通过不同设备获取的目标图像。例如，也可以通过参数不同的至少两个相机分别获取包含目标对象的目标图像。
在目标图像包括多个图像的情况下，参考图像可以包括多个参考图像，分别对应于多个目标图像中的每一个目标图像。其中每个参考图像是通过与 对应的目标图像相同的方式获取的。
在步骤S204中，可以确定所述目标图像的第一图像特征和所述参考图像的第二图像特征。例如，可以采用相同的方式确定所述目标图像的第一图像特征和所述参考图像的第二图像特征。
在一些实施例中，可以利用包括至少一个卷积层的第一神经网络对所述目标图像进行卷积处理以得到第一图像特征。进一步地，可以利用所述第一神经网络对所述参考图像进行卷积处理以得到第二图像特征。也就是说，可以利用共享参数的神经网络对目标图像和参考图像进行处理。所述第一图像特征和所述第二图像特征可以分别包括多个图像特征。
在一些实现方式中，第一神经网络可以是任何能够从图像中获取图像特征的神经网络。例如，第一神经网络可以是包含至少一个卷积层的任何网络，如Inception系列网络(例如Googlenet等)、VGG系列网络、Resnet系列网络等中的任意一个或上述网络中任意一个的至少一部分。
在一些实施例中，也可以通过提取目标图像中的颜色特征、纹理特征、形状特征、空间关系特征等特征中的至少一种作为第一图像特征。进一步的，可以利用相同的方法提取参考图像中的特征作为第二图像特征。
在步骤S206中，可以融合所述第一图像特征和所述第二图像特征确定待分类图像特征。
在一些实施例中，可以通过拼接所述第一图像特征和所述第二图像特征，以确定所述待分类图像特征。
在一种实现方式中，第一图像特征和第二图像特征可以具有多个通道。例如，第一图像特征可以是尺寸为H1*W1*C1的张量，其中H1可以是第一图像特征在第一方向(例如长度方向)上的尺寸，W1可以是第一图像特征在第二方向(例如宽度方向)上的尺寸，H1、W1可以是以像素数量为单位的尺寸，C1可以是第一图像特征的通道数。第二图像特征可以是尺寸为H2*W2*C2的张量，其中H2可以是第二图像特征在第一方向(例如长度方向)上的尺寸，W2可以是第二图像特征在第二方向(例如宽度方向)上的尺寸，H2、W2可以是以像素数量为单位的尺寸，C2可以是第二图像特征的通道数。其中C1、C2是大于1的整数。
在这种情况下，融合所述第一图像特征和所述第二图像特征确定待分类图像特征可以包括拼接所述第一图像特征和所述第二图像特征，以确定所述 待分类图像特征。
例如，第一图像特征可以具有1024个通道，第二图像特征也可以具有1024个通道，通过拼接所述第一图像特征和所述第二图像特征可以得到一个2048个通道的待分类图像特征。该具有2048个通道的待分类图像特征融合有第一图像特征和第二图像特征的信息，即，融合有目标图像和参考图像的图像信息。
在一些实施例中，也可以将第一图像特征和第二图像特征对应通道中对应位置的每个元素进行相加，以确定融合有目标图像和参考图像的图像信息的待分类图像特征。
在步骤S208中，可以利用步骤S206中得到的待分类图像特征确定所述目标图像属于预设类别的概率。例如，融合后的待分类图像特征同时用于预测左右双侧乳腺是否为乳腺癌阳性。
在一些实施例中，可以利用第一全连接网络对所述待分类图像特征进行处理，以得到所述目标图像属于预设类别的概率。
例如，可以配置所述第一全连接网络使得所述第一全连接网络输出一个具有多个维度的向量，该向量中的每个元素表示目标图像和参考图像属于预设类别的置信分数。
以针对医学图像的分类过程为例，预设类别可以包括健康类别和疾病类别。在一种实现方式中，目标图像可以是人体左侧乳腺的医学图像，参考图像可以是人体右侧乳腺的医学图像。在这种情况下，第一全连接网络输出的向量可以是4维的。该向量的4个维度的元素分别代表左侧乳腺的医学图像属于健康类别、疾病类别以及右侧乳腺的医学图像属于健康类别、疾病类别的置信分数。
可以理解的是，针对不同的应用场景，本领域技术人员可以根据实际情况设置用于分类的预设类别的数量。例如，预设类别可以是预先定义的动物类别、植物类别、场景类别等任何可能的分类类别。根据输入图像的数量(即目标图像和参考图像的总数量)和预设类别的数量可以配置第一全连接网络以输出预定维度的向量。例如，当输入图像的数量是m，预设类别的数量是n时，第一全连接网络输出的向量可以具有m n个维度。
在一些实施例中，可以根据第一全连接网络输出的针对多个维度的置信分数，确定所述目标图像和所述参考图像属于预设类别的概率。
例如，针对左侧乳腺分别属于健康类别和疾病类别的两个置信分数，可以利用softmax函数对用于左侧乳腺的两个置信分数进行归一化以得到左侧乳腺的医学图像属于健康类别的概率和左侧乳腺的医学图像属于疾病类别的概率。类似地，可以利用softmax函数得到右侧乳腺的医学图像属于健康类别的概率和右侧乳腺的医学图像属于疾病类别的概率。
在一些实施例中，当目标图像属于预设类别的概率大于预设的概率阈值(如0.5)时，可以认为目标图像属于预设类别。
至此，可以根据参考图像的图像信息实现对于目标图像的图像分类。
利用本申请实施例提供的上述图像分类方法，可以在图像分类过程中融合参考图像和目标图像的图像信息，并可以根据融合了目标图像和参考图像的图像信息的图像特征，确定目标图像属于预设类别的概率，从而实现对目标图像的更准确的分类。在所述目标图像和参考图像为医学图像的情况下，提高了疾病筛查的准确性。例如，通过本申请实施例提供的上述图像分类方法，在乳腺癌病人因患有乳腺癌，而双侧乳腺图像对比信息显著不同时，可高概率判断此人某一侧为乳腺癌阳性。
图3A示出了根据本申请的图像分类方法的示意性的过程。如图3A所示，输入图像可以包括人体左侧乳腺的cc位图像、mlo位图像以及右侧乳腺的cc位图像、mlo位图像。图3B中示出了人体左侧乳腺的cc位图像、mlo位图像以及右侧乳腺的cc位图像、mlo位图像。
如图3A所示，可以将上述四个输入图像分别输入一个共享参数的googlenet网络，从而得到每个图像的各自的向量化的图像特征。在一些实施例中，可以选择googlenet网络中的任一层(如某一靠后层，例如googlenet网络中的第pool5/7×7_s1层)输出的结果作为图像的向量化的图像特征。
例如，可以利用googlenet网络中第pool5/7×7_s1层，为每个输入图像生成一个1024维度的向量化特征。通过融合这四个1024维的图像特征f cc l、f mlo l、f cc R、f mlo R，可以得到一个4096维的融合特征F＝[f cc l,f mlo l,f cc R,f mlo R]。将该融合特征再经过一个全连接层处理，可以得到一个4维的向量，分别代表左侧乳腺和右侧乳腺分别属于健康类别和疾病类别的置信分数s + l、s - l、s + R、s - R。可以利用softmax函数对s + l、s - l进行归一化，以得到左侧乳腺属于健康类别和疾病类别的概率p + l和p - l。
Figure PCTCN2020098407-appb-000001
Figure PCTCN2020098407-appb-000002
类似地，可以利用softmax函数对s + R、s - R进行归一化，以得到右侧乳腺属于健康类别和疾病类别概率的p + R和p - R，其中，exp()为指数运算函数。
在左侧乳腺的医学图像作为目标图像，右侧乳腺的医学图像作为参考图像的情况下，利用图3A中示出的过程可以得到目标图像和参考图像分别属于健康类别和疾病类别的概率。当左侧乳腺的医学图像属于健康类别的概率p + l的值大于预设的概率阈值(如0.5)时，可以认为左侧乳腺的医学图像属于健康类别，类似地，可以确定右侧乳腺的医学图像所属的类别。
本申请实施例提供的图像分类方法可用于识别疑似乳腺癌病人的乳腺钼靶扫描片是否含有乳腺癌。例如，可以基于用于左右双侧乳腺对比的网络结构，同时接收左右双侧乳腺的多视角扫描片，对每侧乳腺的每个扫描片单独进行特征提取，而后进行特征融合，融合后的特征同时用于预测左右双侧乳腺是否为乳腺癌阳性。
在利用神经网络提取图像特征并得到最后的分类结果的情况下，为了针对不同类型的输入图像实现准确的分类效果，需要利用相应类型的训练集对使用的神经网络(例如上文中提及的第一神经网络和/或第一全连接网络)进行训练。
在一些实施例中，所述第一神经网络和所述第一全连接网络可以是通过以下方式训练的。例如，可以确定所述第一神经网络的第一训练集，其中所述第一训练集中包括多个训练图像。该多个训练图像是与目标图像相同类型的图像。例如，以医学图像为例，当上述第一神经网络和第一全连接网络是用于乳腺图像的分类时，第一训练集中包括的多个训练图像可以是已知符合预设类型的不同乳腺图像的样本。例如，所述第一训练集可以包括第一训练图像。根据图2中所示的方法，可以确定用于所述第一训练图像的第一参考训练图像。例如，在第一训练图像是左侧乳腺的医学图像的情况下，第一参考训练图像可以是右侧乳腺的医学图像。
参考图2中所示的方法，可以利用所述第一神经网络对所述第一训练图像和所述第一参考训练图像分别进行卷积处理以得到第一训练图像特征和第二训练图像特征。进一步地，可以根据所述第一训练图像特征和所述第二训练图像特征确定待分类训练图像特征。其中所述待分类训练图像特征融合有第一训练图像和第一训练参考图像的图像信息。进一步地，可以利用第一全 连接网络对所述待分类训练图像特征进行处理，以确定所述第一训练图像属于预设类别的概率。
为了实现对于第一神经网络和第一全连接网络的训练，可以调整所述第一神经网络和所述第一全连接网络的参数，使得所述第一训练图像属于预设类别的概率与所述第一训练图像所属的真实类别之间的损失最小。
可以利用交叉熵损失函数，计算第一训练图像属于预设类别的概率与所述第一训练图像所属的真实类别之间的损失。
如前所述，以乳腺医学图像为例，第一全连接网络可以输出表示左侧乳腺图像属于健康类别和疾病类别以及右侧乳腺图像属于健康类别和疾病类别的概率。在这种情况下，可以根据下式计算左侧乳腺图像的损失：
Figure PCTCN2020098407-appb-000003
其中
Figure PCTCN2020098407-appb-000004
是左侧乳腺图像属于健康类别的概率，
Figure PCTCN2020098407-appb-000005
是左侧乳腺图像属于疾病类别的概率，d是标注真值，当左侧乳腺图像所属的真实类别是健康类别时，d＝0，当左侧乳腺图像所属的真实类别是疾病类别时，d＝1。
类似地，可以根据下式计算右侧乳腺图像的损失：
Figure PCTCN2020098407-appb-000006
其中
Figure PCTCN2020098407-appb-000007
是右侧乳腺图像属于健康类别的概率，
Figure PCTCN2020098407-appb-000008
是右侧乳腺图像属于疾病类别的概率，d是标注真值，当右侧乳腺图像所属的真实类别是健康类别时，d＝0，当右侧乳腺图像所属的真实类别是疾病类别时，d＝1。
根据本申请实施例，由于第一神经网络和第一全连接网络是使用多个已知类别的双侧乳腺图像进行训练的，在训练过程中可以学习双侧乳腺图像的对比信息，因此，即使在不同人个体的乳腺扫描片在组织密度、脂肪厚度等方面差异比较大的情况下，训练后的第一神经网络和第一全连接网络也可以基于乳腺扫描片，做出是否有乳腺癌的稳定的、精确的判断结果。
由于在一些情况下，训练数据正样本和负样本的比例不均衡。例如，以医学图像为例，训练图像中属于疾病类别的样本较少，而属于健康类别的样本较多，因此存在样本不均衡的问题。
为了解决上述样本不均衡的问题，可以通过包括不同比例的正样本和负样本的样本集，采用相同方法训练至少两个参数不同的网络模型(如包括前述第一神经网络和第一全连接网络的网络模型)以对输入图像进行处理，并根据不同的网络的输出结果确定输入图像属于预设类别的概率。所述输入图 像包括目标图像和目标图像的参考图像。例如，所述第一神经网络是利用第一训练集训练得到的，所述第二神经网络是利用第二训练集训练得到的，第一训练集和第二训练集包括的训练图像中的正负样本比例是不同的。
图4示出了根据本申请实施例的一种图处理方法的流程图。如图4所示，在步骤S402中，可以接收目标图像。其中所述目标图像可以是医学图像或其它任何类型的图像。
在步骤S404中，可以利用第一神经网络确定所述目标图像的第一目标图像特征。例如，可以利用包括至少一个卷积层的第一神经网络对所述目标图像进行卷积处理以得到第一目标图像特征。所述第一目标图像特征可以包括多个图像特征。
在步骤S406中，可以利用第二神经网络确定所述目标图像的第二目标图像特征。例如，可以利用包括至少一个卷积层的第二神经网络对所述目标图像进行卷积处理以得到第二目标图像特征。所述第二图像目标特征可以分别包括多个图像特征。
在一些实现方式中，第一神经网络和第二神经网络可以是任何能够从图像中获取图像特征的神经网络。例如，第一神经网络可以是包含至少一个卷积层的任何网络，如Inception系列网络(例如Googlenet等)、VGG系列网络、Resnet系列网络等中的任意一个或上述网络中任意一个的至少一部分。
在一些实施例中，所述第一神经网络与所述第二神经网络是采用相同训练方法训练的不同网络，所述第一神经网络是利用第一训练集训练得到的，所述第二神经网络是利用第二训练集训练得到的，第一训练集和第二训练集的包括的训练图像中的正负样本比例是不同的。
在步骤S408中，可以根据所述第一目标图像特征和所述第二目标图像特征确定所述目标图像的第一图像处理结果和第二图像处理结果。
可以理解的是，由于第一神经网络和第二神经网络是采用相同训练方法得到的不同网络，因此第一图像处理结果和第二图像处理结果是相同类型的。例如，第一图像处理结果和第二图像处理结果可以是图像分类结果、图像分割结果、目标检测结果中的至少一个。这取决于本领域技术人员训练神经网络所采用的具体方式和训练集。
在步骤S410中，融合所述第一图像处理结果和所述第二图像处理结果，以确定所述目标图像的图像处理结果。其中，对应于第一图像处理结果和第 二图像处理结果的类型，所述图像处理结果包括图像分类结果、图像分割结果、目标检测结果中的至少一个。
可以通过实验确定使得网络的正样本损失和负样本损失接近的正样本和负样本的比例。例如，可以通过计算训练集中所有正样本的损失函数和所有负样本的损失函数，确定该训练集训练的网络的正样本损失和负样本损失。下面将以当正样本和负样本比例为1:2时，网络的正样本损失和负样本损失相近为例，描述本申请的原理。
在一种实现方式中，可以通过采样确定正样本和负样本比例为1:1的第一训练集以及正样本和负样本比例为1:3的第二训练集。
根据本申请的原理，由于当正样本和负样本比例为1:2时，网络的正样本损失和负样本损失相近，因此，当改变训练集中的正样本和负样本的比例时，训练生成的网络的正样本损失和负样本损失的比例也可以相应改变。因此，由于正样本和负样本比例为1:1的第一训练集中提高了正样本比例，因此第一训练集训练的网络的正样本损失小于负样本损失。类似地，由于正样本和负样本比例为1:3的第二训练集中提高了负样本比例，因此第二训练集训练的网络的正样本损失大于负样本损失。
在这种情况下，第一训练集训练的网络对正样本的分类效果更好，第二训练集训练的网络对负样本的分类效果更好。例如，当利用第一训练集训练的网络进行图像分类时，对于正样本的输入图像，第一训练集训练的网络输出的表示该输入图像属于预设类别的概率更接近于0或1，即更容易分辨该输入图像是否属于预设类别。相应地，由于第一训练集训练的网络对负样本的分类效果较差，对于负样本的输入图像，第一训练集训练的网络将输出更接近于0.5的概率，即较不容易分辨该输入图像所属的类别。基于以上特性，通过融合第一训练集训练的网络和第二训练集训练的网络的输出结果，能够得到更为精确的预测结果。
类似地，也可以通过采样确定正样本和负样本比例为1:1的第一训练集、正样本和负样本比例为1:3的第二训练集以及正样本和负样本比例为1:2的第三训练集。可以利用上述第一训练集、第二训练集以及第三训练集，分别训练第一神经网络、第二神经网络以及第三神经网络。以前述的人体乳腺医学图像为例，可以根据利用第一神经网络、第二神经网络以及第三神经网络分别输出左侧乳腺医学图像属于健康类别的三个输出结果PA + l、PB + l、PC + l、左 侧乳腺医学图像属于疾病类别的三个输出结果PA - l、PB - l、PC - l、右侧乳腺医学图像属于健康类别的三个输出结果PA + r、PB + r、PC + r、右侧乳腺医学图像属于疾病类别的三个输出结果PA - r、PB - r、PC - r。可以通过融合(例如加权平均)上述三个利用不同训练集得到的网络输出的结果，确定左侧乳腺医学图像和右侧乳腺医学图像属于健康类别和疾病类别的概率。例如，可以通过下式确定左侧乳腺医学图像和右侧乳腺医学图像属于健康类别和疾病类别的概率。
Figure PCTCN2020098407-appb-000009
Figure PCTCN2020098407-appb-000010
Figure PCTCN2020098407-appb-000011
Figure PCTCN2020098407-appb-000012
其中，
Figure PCTCN2020098407-appb-000013
是左侧乳腺医学图像属于健康类别的概率，
Figure PCTCN2020098407-appb-000014
是左侧乳腺医学图像属于疾病类别的概率，
Figure PCTCN2020098407-appb-000015
是右侧乳腺医学图像属于健康类别的概率，
Figure PCTCN2020098407-appb-000016
是右侧乳腺图像属于疾病类别的概率。
类似地，可以通过与步骤S402-S410类似的步骤获得参考图像的第一目标参考图像特征和第二目标参考图像特征。
返回参考图2，在步骤S204中，在一些实施例中，所述第一图像特征包括第一目标图像特征和第二目标图像特征。所述第一目标图像特征是利用包括至少一个卷积层的第一神经网络对所述目标图像进行卷积处理得到的，所述第二目标图像特征是利用第二神经网络对所述目标图像进行卷积处理得到的。
所述第二图像特征包括第一目标参考图像特征以及第二目标参考图像特征，其中所述第一目标参考图像特征是利用包括至少一个卷积层的第一神经网络对所述参考图像进行卷积处理得到的，所述第二目标参考图像特征是利用所述第二神经网络对所述参考图像进行卷积处理得到的。
其中，所述第一神经网络与所述第二神经网络是采用相同训练方法训练的不同网络，所述第一神经网络是利用前述第一训练集训练得到的，所述第二神经网络是利用前述第二训练集训练得到的。如前所述，第一训练集和第二训练集的包括的训练图像中的正负样本比例是不同的，因此第一神经网络和第二神经网络具有不同的参数，也因此具有不同的输出结果。
在这种情况下，在步骤S206中，所述待分类图像特征包括第一待分类图像特征和第二待分类图像特征。例如，所述第一待分类图像特征可以是通过 拼接所述第一目标图像特征和所述第一目标参考图像特征确定的，所述第二待分类图像特征可以是通过拼接所述第二目标图像特征和所述第二目标参考图像特征确定的。
在步骤S208中，可以利用第一全连接网络对所述第一待分类图像特征进行处理，以得到所述目标图像属于预设类别的第一概率。可以利用第二全连接网络对所述第二待分类图像特征进行处理，以得到所述目标图像属于预设类别的第二概率。通过融合所述第一概率和第二概率可以确定所述目标图像属于预设类别的概率。例如，可以根据所述第一概率和所述第二概率的加权平均值确定所述目标图像属于预设类别的概率。其中，所述第一全连接网络与所述第二全连接网络是采用相同训练方法训练的不同网络，其中所述第一全连接网络是利用第一训练集训练得到的，所述第二全连接网络是利用第二训练集训练得到的，第一训练集和第二训练集的包括的训练图像中的正负样本比例是不同的。
尽管本申请中仅以两个或三个不同的训练集为例解释了本申请的原理，然而，本申请的实施例不限于此。本领域技术人员可以根据实际情况选取更多不同的训练集，并得到更多个用于执行图像分类的网络模型。
利用本申请提供的上述基于不同训练集得到的网络模型实现的图像分类方法，能够克服相关技术中训练数据比例不均衡的问题，提高图像分类的准确性，以及疾病筛查的准确性。
图5示出了根据本申请实施例的图像分类装置的示意性的框图。如图5所示，图像分类装置500可以包括接收单元510、图像特征确定单元520、融合单元530以及分类结果生成单元540。
接收单元510可以配置成接收目标图像以及关于所述目标图像的至少一个参考图像。在一些实施例中，参考图像可以使与目标图像相同类型的图像。例如，参考图像中可以包含与目标图像中相同的目标对象或相同类型的其他目标对象。
其中，所述目标图像可以是医学图像。例如，当目标图像是人体一侧器官的医学图像时，参考图像可以是同一人另一侧器官的同类医学图像。例如，这里所说的器官可以是乳腺、眼睛、肺部、牙齿等任何人体内存在两个或两个以上数量的器官。
可以理解的是，在不脱离本申请公开的原理的情况下，这里所说的目标 图像也可以是除医学图像以外的任何其他类型的图像，只要参考图像中可以包含与目标图像中的相同类型的目标对象即可。例如，目标图像可以是人脸图像，此时，参考图像可以是在其他时间(例如在不同的背景中、不同的光照下、不同的年龄阶段)拍摄的人脸图像。又例如，目标图像可以是除人以外的任何动物或植物。
在一些实施例中，目标图像可以包括针对同一目标对象的多个图像。
在一些实现方式中，目标图像可以包括从至少两个角度获取目标对象的图像。例如，目标图像可以包括以上下夹俯视(CC，Craniocaudal)视角拍摄的人体乳腺的钼靶检测图像以及以左右夹侧视(MLO，Mediolateral-Oblique)视角拍摄的人体乳腺的钼靶检测图像。可以理解的是，当目标图像包括其他类型的目标对象时，本领域技术人员也可以任意设置拍摄目标对象的方式，以获得其他通过不同角度获取的目标图像。
在另一些实现方式中，目标图像也可以包括通过至少两个不同设备获取的目标对象。例如，目标图像可以包括通过X射线设备采集的人体乳腺的图像以及通过MRI设备采集的人体乳腺的图像。可以理解的是，当目标图像包括其他类型的目标对象时，本领域技术人员也可以任意设置拍摄目标对象的方式，以获得其他通过不同设备获取的目标图像。例如，也可以通过参数不同的至少两个相机分别获取包含目标对象的目标图像。
在目标图像包括多个图像的情况下，参考图像可以包括多个参考图像，分别对应于多个目标图像中的每一个目标图像。其中每个参考图像是通过与对应的目标图像相同的方式获取的。
图像特征确定单元520可以配置成确定所述目标图像的第一图像特征和所述参考图像的第二图像特征。例如，可以采用相同的方式确定所述目标图像的第一图像特征和所述参考图像的第二图像特征。
在一些实施例中，可以利用包括至少一个卷积层的第一神经网络对所述目标图像进行卷积处理以得到第一图像特征。进一步地，可以利用所述第一神经网络对所述参考图像进行卷积处理以得到第二图像特征。也就是说，可以利用共享参数的神经网络对目标图像和参考图像进行处理。所述第一图像特征和所述第二图像特征可以分别包括多个图像特征。
在一些实现方式中，第一神经网络可以是任何能够从图像中获取图像特征的神经网络。例如，第一神经网络可以是包含至少一个卷积层的任何网络， 如Inception系列网络(例如Googlenet等)、VGG系列网络、Resnet系列网络等中的任意一个或上述网络中任意一个的至少一部分。
在一些实施例中，也可以通过提取目标图像中的颜色特征、纹理特征、形状特征、空间关系特征等特征中的至少一种作为第一图像特征。进一步的，可以利用相同的方法提取参考图像中的特征作为第二图像特征。
在一些实施例中，所述第一图像特征包括第一目标图像特征和第二目标图像特征。所述第一目标图像特征是利用包括至少一个卷积层的第一神经网络对所述目标图像进行卷积处理得到的，所述第二目标图像特征是利用第二神经网络对所述目标图像进行卷积处理得到的。所述第一目标图像特征和所述第二目标图像特征可以分别包括多个图像特征。
所述第二图像特征包括第一目标参考图像特征以及第二目标参考图像特征，其中所述第一目标参考图像特征是利用包括至少一个卷积层的第一神经网络对所述参考图像进行卷积处理得到的，所述第二目标参考图像特征是利用所述第二神经网络对所述参考图像进行卷积处理得到的。
其中，所述第一神经网络与所述第二神经网络是采用相同训练方法训练的不同网络，所述第一神经网络是利用前述第一训练集训练得到的，所述第二神经网络是利用前述第二训练集训练得到的。
融合单元530可以配置成融合所述第一图像特征和所述第二图像特征确定待分类图像特征。
在一些实施例中，可以通过拼接所述第一图像特征和所述第二图像特征，以确定所述待分类图像特征。
其中，融合所述第一图像特征和所述第二图像特征确定待分类图像特征可以包括拼接所述第一图像特征和所述第二图像特征，以确定所述待分类图像特征。
在一些实施例中，所述待分类图像特征包括第一待分类图像特征和第二待分类图像特征。例如，所述第一待分类图像特征可以是通过拼接所述第一目标图像特征和所述第一目标参考图像特征确定的，所述第二待分类图像特征可以是通过拼接所述第二目标图像特征和所述第二目标参考图像特征确定的。
分类结果生成单元540可以配置成利用融合单元530生成的待分类图像特征确定所述目标图像属于预设类别的概率。在一些实施例中，可以利用第 一全连接网络对所述待分类图像特征进行处理，以得到所述目标图像属于预设类别的概率。
例如，可以配置所述第一全连接网络使得所述第一全连接网络输出一个具有多个维度的向量，该向量中的每个元素表示目标图像和参考图像属于预设类别的置信分数。
可以理解的是，针对不同的应用场景，本领域技术人员可以根据实际情况设置用于分类的预设类别的数量。
在一些实施例中可以根据第一全连接网络输出的针对多个维度的置信分数确定所述目标图像和所述参考图像属于预设类别的概率。
例如，针对左侧乳腺分别属于健康类别和疾病类别的两个置信分数，可以利用softmax函数对用于左侧乳腺的两个置信分数进行归一化以得到左侧乳腺的医学图像属于健康类别的概率和左侧乳腺的医学图像属于疾病类别的概率。类似地，可以利用softmax函数得到右侧乳腺的医学图像属于健康类别的概率和右侧乳腺的医学图像属于疾病类别的概率。
在一些实施例中，当目标图像属于预设类别的概率大于预设的概率阈值(如0.5)时，可以认为目标图像属于预设类别。
在一些实施例中，分类结果生成单元540还可以利用第一全连接网络对所述第一待分类图像特征进行处理，以得到所述目标图像属于预设类别的第一概率。可以利用第二全连接网络对所述第二待分类图像特征进行处理，以得到所述目标图像属于预设类别的第二概率。通过融合所述第一概率和第二概率可以确定所述目标图像属于预设类别的概率。例如，可以根据所述第一概率和所述第二概率的加权平均值确定所述目标图像属于预设类别的概率。
至此，可以根据参考图像的图像信息实现对于目标图像的图像分类。
利用本申请提供的上述图像分类装置，可以在图像分类过程中融合参考图像和目标图像的图像信息，并可以根据融合了目标图像和参考图像的图像信息的图像特征确定目标图像属于预设类别的概率，从而实现对目标图像的更准确的分类。在所述目标图像和参考图像为医学图像的情况下，提高了疾病筛查的准确性。此外，还能够克服相关技术中训练数据比例不均衡的问题，进一步提高图像分类的准确性，以及疾病筛查的准确性。
图6示出了根据本申请实施例的医疗电子设备的示意性的框图。如图6所示，医疗电子设备600可以包括图像采集单元610、图像特征确定单元620、 融合单元630以及分类结果生成单元640。
图像采集单元610可以用于采集目标图像以及关于所述目标图像的参考图像。这里所说的医学图像可以是例如通过CT、MRI、超声、X光、核素显像(如SPECT、PET)等方法采集的医学图像，也可以是例如心电图、脑电图、光学摄影等显示人体生理信息的图像。
图像特征确定单元620、融合单元630以及分类结果生成单元640可以实现为图5中示出的图像特征确定单元520、融合单元530以及分类结果生成单元540，在此不再加以赘述。
在一些实现方式中，本申请提供的医疗电子设备可以是CT、MRI、超声、X光仪器等任何医学成像设备。图像采集单元610可以实现为上述医学成像设备的成像单元，图像特征确定单元620、融合单元630以及分类结果生成单元640可以通过医学成像设备的内部处理单元(例如处理器)实现。
此外，根据本申请实施例的方法或装置也可以借助于图7所示的计算设备的架构来实现。图7示出了该计算设备的架构。如图7所示，计算设备700可以包括总线710、一个或至少两个CPU 720、只读存储器(ROM)730、随机存取存储器(RAM)740、连接到网络的通信端口750、输入/输出组件760、硬盘770等。计算设备700中的存储设备，例如ROM 730或硬盘770可以存储本申请提供的用于在视频中对目标进行检测的方法的处理和/或通信使用的各种数据或文件以及CPU所执行的程序指令。计算设备700还可以包括用户界面780。当然，图7所示的架构只是示例性的，在实现不同的设备时，根据实际需要，可以省略图7示出的计算设备中的一个或至少两个组件。
本申请的实施例也可以被实现为计算机可读存储介质。根据本申请实施例的计算机可读存储介质上存储有计算机可读指令。当所述计算机可读指令由处理器运行时，可以执行参照以上附图描述的根据本申请实施例的方法。所述计算机可读存储介质包括但不限于例如易失性存储器和/或非易失性存储器。所述易失性存储器例如可以包括随机存取存储器(RAM)和/或高速缓冲存储器(cache)等。所述非易失性存储器例如可以包括只读存储器(ROM)、硬盘、闪存等。
本领域技术人员能够理解，本申请所披露的内容可以出现多种变型和改进。例如，以上所描述的各种设备或组件可以通过硬件实现，也可以通过软件、固件、或者三者中的一些或全部的组合实现。
此外，如本申请和权利要求书中所示，除非上下文明确提示例外情形，“一”、“一个”、“一种”和/或“该”等词并非特指单数，也可包括复数。一般说来，术语“包括”与“包含”仅提示包括已明确标识的步骤和元素，而这些步骤和元素不构成一个排它性的罗列，方法或者设备也可能包含其他的步骤或元素。
此外，虽然本申请对根据本申请的实施例的系统中的某些单元做出了各种引用，然而，任何数量的不同单元可以被使用并运行在客户端和/或服务器上。所述单元仅是说明性的，并且所述系统和方法的不同方面可以使用不同单元。
此外，本申请中使用了流程图用来说明根据本申请的实施例的系统所执行的操作。应当理解的是，前面或下面操作不一定按照顺序来精确地执行。相反，可以按照倒序或同时处理各种步骤。同时，也可以将其他操作添加到这些过程中，或从这些过程移除某一步或数步操作。
除非另有定义，这里使用的所有术语(包括技术和科学术语)具有与本发明所属领域的普通技术人员共同理解的相同含义。还应当理解，诸如在通常字典里定义的那些术语应当被解释为具有与它们在相关技术的上下文中的含义相一致的含义，而不应用理想化或极度形式化的意义来解释，除非这里明确地这样定义。
上面是对本发明的说明，而不应被认为是对其的限制。尽管描述了本发明的若干示例性实施例，但本领域技术人员将容易地理解，在不背离本发明的新颖教学和优点的前提下可以对示例性实施例进行许多修改。因此，所有这些修改都意图包含在权利要求书所限定的本发明范围内。应当理解，上面是对本发明的说明，而不应被认为是限于所公开的特定实施例，并且对所公开的实施例以及其他实施例的修改意图包含在所附权利要求书的范围内。本发明由权利要求书及其等效物限定。

Claims (15)

  1. 一种图像分类方法，由电子设备执行，所述方法包括：
    接收目标图像以及关于所述目标图像的参考图像，其中所述目标图像是医学图像；
    采用相同的方式确定所述目标图像的第一图像特征和所述参考图像的第二图像特征；
    融合所述第一图像特征和所述第二图像特征确定待分类图像特征；以及
    利用所述待分类图像特征确定所述目标图像属于预设类别的概率，该步骤包括：利用所述待分类图像特征，获得多个维度的向量，该向量中的元素分别表示所述目标图像和所述参考图像属于预设类别的置信分数，根据所述目标图像属于预设类别的置信分数，确定所述目标图像属于预设类别的概率。
  2. 根据权利要求1所述的图像分类方法，其中，确定所述目标图像的第一图像特征和所述参考图像的第二图像特征包括：
    利用包括至少一个卷积层的第一神经网络对所述目标图像进行卷积处理以得到第一图像特征；以及
    利用所述第一神经网络对所述参考图像进行卷积处理以得到第二图像特征。
  3. 根据权利要求1所述的方法，其中，融合所述第一图像特征和所述第二图像特征确定待分类图像特征包括：
    拼接所述第一图像特征和所述第二图像特征，以确定所述待分类图像特征。
  4. 根据权利要求2所述的图像分类方法，其中，利用所述待分类图像特征确定所述目标图像属于预设类别的概率包括：
    利用第一全连接网络对所述待分类图像特征进行处理，以得到所述目标图像属于预设类别的概率。
  5. 根据权利要求1所述的图像分类方法，其中，
    所述第一图像特征包括第一目标图像特征和第二目标图像特征，所述第一目标图像特征是利用包括至少一个卷积层的第一神经网络对所述目标图像进行卷积处理得到的，所述第二目标图像特征是利用第二神经网络对所述目标图像进行卷积处理得到的，
    所述第二图像特征包括第一目标参考图像特征以及第二目标参考图像特 征，其中所述第一目标参考图像特征是利用所述包括至少一个卷积层的第一神经网络对所述参考图像进行卷积处理得到的，所述第二目标参考图像特征是利用所述第二神经网络对所述参考图像进行卷积处理得到的，
    所述第一神经网络与所述第二神经网络是采用相同训练方法训练的不同网络，其中所述第一神经网络是利用第一训练集训练得到的，所述第二神经网络是利用第二训练集训练得到的，第一训练集和第二训练集的包括的训练图像中的正负样本比例是不同的。
  6. 根据权利要求5所述的图像分类方法，其中
    所述待分类图像特征包括第一待分类图像特征和第二待分类图像特征，其中所述第一待分类图像特征是通过拼接所述第一目标图像特征和所述第一目标参考图像特征确定的，所述第二待分类图像特征是通过拼接所述第二目标图像特征和所述第二目标参考图像特征确定的。
  7. 根据权利要求6所述的图像分类方法，其中，利用所述待分类图像特征确定所述目标图像的图像分类结果包括：
    利用第一全连接网络对所述第一待分类图像特征进行处理，以得到所述目标图像属于预设类别的第一概率；
    利用第二全连接网络对所述第二待分类图像特征进行处理，以得到所述目标图像属于预设类别的第二概率；以及
    根据所述第一概率和所述第二概率的加权平均值确定所述目标图像属于预设类别的概率，
    其中，所述第一全连接网络与所述第二全连接网络是采用相同训练方法训练的不同网络，其中所述第一全连接网络是利用第一训练集训练得到的，所述第二全连接网络是利用第二训练集训练得到的，第一训练集和第二训练集的包括的训练图像中的正负样本比例是不同的。
  8. 根据权利要求4或7所述的图像分类方法，其中所述第一神经网络和所述第一全连接网络是通过以下方式训练的：
    确定所述第一神经网络的第一训练集，其中所述第一训练集包括第一训练图像；
    确定所述第一训练图像的第一参考训练图像；
    利用所述第一神经网络对所述第一训练图像和所述第一参考训练图像分别进行卷积处理以得到第一训练图像特征和第二训练图像特征；
    根据所述第一训练图像特征和所述第二训练图像特征确定待分类训练图像特征；
    利用第一全连接网络对对所述待分类训练图像特征进行处理以确定所述第一训练图像属于预设类别的概率；
    调整所述第一神经网络和所述第一全连接网络的参数使得所述第一训练图像属于预设类别的概率与所述第一训练图像所属的真实类别之间的损失最小。
  9. 一种图像分类装置，包括：
    接收单元，配置成接收目标图像以及关于所述目标图像的参考图像，其中所述目标图像是医学图像；
    图像特征确定单元，配置成采用相同的方式确定所述目标图像的第一图像特征和所述参考图像的第二图像特征；
    融合单元，配置成融合所述第一图像特征和所述第二图像特征确定待分类图像特征；以及
    分类结果生成单元，配置成利用所述待分类图像特征确定所述目标图像属于预设类别的概率，其中，分类结果生成单元用于利用所述待分类图像特征，获得多个维度的向量，该向量中的元素分别表示所述目标图像和所述参考图像属于预设类别的置信分数，根据所述目标图像属于预设类别的置信分数，确定所述目标图像属于预设类别的概率。
  10. 根据权利要求9所述的图像分类装置，其中，图像特征确定单元进一步配置成：
    利用包括至少一个卷积层的第一神经网络对所述目标图像进行卷积处理以得到第一图像特征；以及
    利用所述第一神经网络对所述参考图像进行卷积处理以得到第二图像特征。
  11. 一种图像处理方法，由电子设备执行，所述方法包括：
    接收目标图像；
    利用第一神经网络确定所述目标图像的第一目标图像特征；
    利用第二神经网络确定所述目标图像的第二目标图像特征；
    根据所述第一目标图像特征和所述第二目标图像特征确定所述目标图像的第一图像处理结果和第二图像处理结果；
    融合所述第一图像处理结果和所述第二图像处理结果以确定所述目标图像的图像处理结果，
    其中，所述第一神经网络与所述第二神经网络是采用相同训练方法训练的不同网络，所述第一神经网络是利用第一训练集训练得到的，所述第二神经网络是利用第二训练集训练得到的，第一训练集和第二训练集的包括的训练图像中的正负样本比例是不同的。
  12. 如权利要求11所述的图像处理方法，其中所述图像处理结果包括图像分类结果、图像分割结果、目标检测结果中的至少一个。
  13. 一种医疗电子设备，包括：
    图像采集单元，配置成采集目标图像以及关于所述目标图像的参考图像，其中所述目标图像是医学图像；
    图像特征确定单元，配置成采用相同的方式确定所述目标图像的第一图像特征和所述参考图像的第二图像特征；
    融合单元，配置成融合所述第一图像特征和所述第二图像特征确定待分类图像特征；以及
    分类结果生成单元，配置成利用所述待分类图像特征确定所述目标图像属于预设类别的概率，其中，分类结果生成单元用于利用所述待分类图像特征，获得多个维度的向量，该向量中的元素分别表示所述目标图像和所述参考图像属于预设类别的置信分数，根据所述目标图像属于预设类别的置信分数，确定所述目标图像属于预设类别的概率。
  14. 一种图像分类设备，所述设备包括存储器和处理器，其中所述存储器中存有指令，当利用所述处理器执行所述指令时，使得所述处理器执行如权利要求1-8和11-12中任一项所述的图像分类方法。
  15. 一种计算机可读存储介质，其上存储有指令，所述指令在被处理器执行时，使得所述处理器执行如权利要求1-8和11-12中任一项所述的图像分类方法。
PCT/CN2020/098407 2019-06-28 2020-06-28 图像分类方法、装置、设备、存储介质和医疗电子设备 WO2020259666A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP20831068.0A EP3992851A4 (en) 2019-06-28 2020-06-28 METHOD, APPARATUS AND DEVICE FOR IMAGE CLASSIFICATION, STORAGE MEDIA AND MEDICAL ELECTRONIC DEVICE
KR1020217028280A KR102605969B1 (ko) 2019-06-28 2020-06-28 이미지 분류 방법, 장치 및 디바이스, 저장 매체, 그리고 의료 전자 디바이스
JP2021548679A JP7297081B2 (ja) 2019-06-28 2020-06-28 画像分類方法、画像分類装置、医療用電子機器、画像分類機器、及びコンピュータプログラム
US17/459,920 US11900647B2 (en) 2019-06-28 2021-08-27 Image classification method, apparatus, and device, storage medium, and medical electronic device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910573560.9A CN110276411B (zh) 2019-06-28 2019-06-28 图像分类方法、装置、设备、存储介质和医疗电子设备
CN201910573560.9 2019-06-28

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/459,920 Continuation US11900647B2 (en) 2019-06-28 2021-08-27 Image classification method, apparatus, and device, storage medium, and medical electronic device

Publications (1)

Publication Number Publication Date
WO2020259666A1 true WO2020259666A1 (zh) 2020-12-30

Family

ID=67963631

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/098407 WO2020259666A1 (zh) 2019-06-28 2020-06-28 图像分类方法、装置、设备、存储介质和医疗电子设备

Country Status (6)

Country Link
US (1) US11900647B2 (zh)
EP (1) EP3992851A4 (zh)
JP (1) JP7297081B2 (zh)
KR (1) KR102605969B1 (zh)
CN (1) CN110276411B (zh)
WO (1) WO2020259666A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114139031A (zh) * 2021-10-28 2022-03-04 马上消费金融股份有限公司 数据分类方法、装置、电子设备及存储介质
CN114582469A (zh) * 2022-03-11 2022-06-03 无锡祥生医疗科技股份有限公司 医疗影像分类方法、装置、设备及存储介质
EP4350575A4 (en) * 2021-06-30 2024-09-18 Huawei Tech Co Ltd IMAGE CLASSIFICATION METHOD AND ASSOCIATED DEVICE

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110276411B (zh) * 2019-06-28 2022-11-18 腾讯科技（深圳）有限公司 图像分类方法、装置、设备、存储介质和医疗电子设备
CN110827294A (zh) * 2019-10-31 2020-02-21 北京推想科技有限公司 网络模型训练方法及装置、病灶区域确定方法及装置
CN111028310B (zh) * 2019-12-31 2023-10-03 上海联影医疗科技股份有限公司 乳腺断层扫描的扫描参数确定方法、装置、终端及介质
CN111311578B (zh) * 2020-02-17 2024-05-03 腾讯科技（深圳）有限公司 基于人工智能的对象分类方法以及装置、医学影像设备
CN111598131B (zh) * 2020-04-17 2023-08-25 北京百度网讯科技有限公司 图像处理方法、装置、电子设备及存储介质
CN112149748B (zh) * 2020-09-28 2024-05-21 商汤集团有限公司 图像分类方法及装置、电子设备和存储介质
CN112138394B (zh) * 2020-10-16 2022-05-03 腾讯科技（深圳）有限公司 图像处理方法、装置、电子设备及计算机可读存储介质
CN113313063A (zh) * 2021-06-21 2021-08-27 暨南大学 麦穗检测方法、电子装置和存储介质
CN113283552A (zh) * 2021-07-22 2021-08-20 深圳百胜扬工业电子商务平台发展有限公司 图像的分类方法、装置、存储介质及电子设备
CN113569953A (zh) * 2021-07-29 2021-10-29 中国工商银行股份有限公司 分类模型的训练方法、装置及电子设备
KR102415616B1 (ko) * 2021-10-18 2022-07-01 라이트하우스(주) 예술품의 이미지 표준화 기반 교육 및 거래 서비스 제공 방법, 장치 및 시스템
CN114220063B (zh) * 2021-11-17 2023-04-07 浙江大华技术股份有限公司 目标检测方法及装置
CN115130539A (zh) * 2022-04-21 2022-09-30 腾讯科技（深圳）有限公司 分类模型训练、数据分类方法、装置和计算机设备
CN114820592B (zh) * 2022-06-06 2023-04-07 北京医准智能科技有限公司 图像处理装置、电子设备及介质
CN117036894B (zh) * 2023-10-09 2024-03-26 之江实验室 基于深度学习的多模态数据分类方法、装置及计算机设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020067857A1 (en) * 2000-12-04 2002-06-06 Hartmann Alexander J. System and method for classification of images and videos
CN106650550A (zh) * 2015-10-28 2017-05-10 中通服公众信息产业股份有限公司 一种融合车标与车头图像特征的车辆型号识别方法及系统
CN109146848A (zh) * 2018-07-23 2019-01-04 东北大学 一种融合多模态乳腺图像的计算机辅助参考系统及方法
CN109447065A (zh) * 2018-10-16 2019-03-08 杭州依图医疗技术有限公司 一种乳腺影像识别的方法及装置
CN110276411A (zh) * 2019-06-28 2019-09-24 腾讯科技（深圳）有限公司 图像分类方法、装置、设备、存储介质和医疗电子设备
CN110321920A (zh) * 2019-05-08 2019-10-11 腾讯科技（深圳）有限公司 图像分类方法、装置、计算机可读存储介质和计算机设备

Family Cites Families (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6778705B2 (en) * 2001-02-27 2004-08-17 Koninklijke Philips Electronics N.V. Classification of objects through model ensembles
JP5652250B2 (ja) * 2011-02-24 2015-01-14 富士ゼロックス株式会社 画像処理プログラム及び画像処理装置
US8498491B1 (en) * 2011-08-10 2013-07-30 Google Inc. Estimating age using multiple classifiers
US10055551B2 (en) * 2013-10-10 2018-08-21 Board Of Regents Of The University Of Texas System Systems and methods for quantitative analysis of histopathology images using multiclassifier ensemble schemes
CN104573708A (zh) * 2014-12-19 2015-04-29 天津大学 组合降采样极限学习机
CN106156807B (zh) * 2015-04-02 2020-06-02 华中科技大学 卷积神经网络模型的训练方法及装置
CN106874279B (zh) * 2015-12-11 2021-01-15 腾讯科技（深圳）有限公司 生成应用类别标签的方法及装置
US9779492B1 (en) * 2016-03-15 2017-10-03 International Business Machines Corporation Retinal image quality assessment, error identification and automatic quality correction
CN106096670B (zh) * 2016-06-17 2019-07-30 深圳市商汤科技有限公司 级联卷积神经网络训练和图像检测方法、装置及系统
CN106682435B (zh) * 2016-12-31 2021-01-29 西安百利信息科技有限公司 一种多模型融合自动检测医学图像中病变的系统及方法
CN108771530B (zh) * 2017-05-04 2021-03-30 深圳硅基智能科技有限公司 基于深度神经网络的眼底病变筛查系统
CN107688823B (zh) * 2017-07-20 2018-12-04 北京三快在线科技有限公司 一种图像特征获取方法及装置，电子设备
US20190034734A1 (en) * 2017-07-28 2019-01-31 Qualcomm Incorporated Object classification using machine learning and object tracking
JP2019028887A (ja) * 2017-08-02 2019-02-21 国立研究開発法人産業技術総合研究所 画像処理方法
CN109426858B (zh) * 2017-08-29 2021-04-06 京东方科技集团股份有限公司 神经网络、训练方法、图像处理方法及图像处理装置
CN107665352A (zh) * 2017-09-07 2018-02-06 浙江工业大学 一种基于多通道残差网络的珍珠分类方法
CN107680088A (zh) * 2017-09-30 2018-02-09 百度在线网络技术（北京）有限公司 用于分析医学影像的方法和装置
US20190130188A1 (en) * 2017-10-26 2019-05-02 Qualcomm Incorporated Object classification in a video analytics system
US20190130191A1 (en) * 2017-10-30 2019-05-02 Qualcomm Incorporated Bounding box smoothing for object tracking in a video analytics system
CN109934369A (zh) * 2017-12-15 2019-06-25 北京京东尚科信息技术有限公司 用于信息推送的方法及装置
CN109461495B (zh) * 2018-11-01 2023-04-14 腾讯科技（深圳）有限公司 一种医学图像的识别方法、模型训练的方法及服务器
WO2020102584A2 (en) * 2018-11-14 2020-05-22 Intuitive Surgical Operations, Inc. Convolutional neural networks for efficient tissue segmentation
CN110516620B (zh) * 2019-08-29 2023-07-28 腾讯科技（深圳）有限公司 目标跟踪方法、装置、存储介质及电子设备
US11544495B2 (en) * 2020-07-10 2023-01-03 Adobe Inc. Attributionally robust training for weakly supervised localization and segmentation
US11776129B2 (en) * 2020-12-16 2023-10-03 Qualcomm Incorporated Semantic refinement of image regions
WO2023164145A1 (en) * 2022-02-25 2023-08-31 Regeneron Pharmaceuticals, Inc. Size exclusion chromatography for characterizing host cell proteins

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020067857A1 (en) * 2000-12-04 2002-06-06 Hartmann Alexander J. System and method for classification of images and videos
CN106650550A (zh) * 2015-10-28 2017-05-10 中通服公众信息产业股份有限公司 一种融合车标与车头图像特征的车辆型号识别方法及系统
CN109146848A (zh) * 2018-07-23 2019-01-04 东北大学 一种融合多模态乳腺图像的计算机辅助参考系统及方法
CN109447065A (zh) * 2018-10-16 2019-03-08 杭州依图医疗技术有限公司 一种乳腺影像识别的方法及装置
CN110321920A (zh) * 2019-05-08 2019-10-11 腾讯科技（深圳）有限公司 图像分类方法、装置、计算机可读存储介质和计算机设备
CN110276411A (zh) * 2019-06-28 2019-09-24 腾讯科技（深圳）有限公司 图像分类方法、装置、设备、存储介质和医疗电子设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3992851A4

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4350575A4 (en) * 2021-06-30 2024-09-18 Huawei Tech Co Ltd IMAGE CLASSIFICATION METHOD AND ASSOCIATED DEVICE
CN114139031A (zh) * 2021-10-28 2022-03-04 马上消费金融股份有限公司 数据分类方法、装置、电子设备及存储介质
CN114139031B (zh) * 2021-10-28 2024-03-19 马上消费金融股份有限公司 数据分类方法、装置、电子设备及存储介质
CN114582469A (zh) * 2022-03-11 2022-06-03 无锡祥生医疗科技股份有限公司 医疗影像分类方法、装置、设备及存储介质

Also Published As

Publication number Publication date
EP3992851A1 (en) 2022-05-04
CN110276411A (zh) 2019-09-24
CN110276411B (zh) 2022-11-18
KR20210119526A (ko) 2021-10-05
JP7297081B2 (ja) 2023-06-23
JP2022537866A (ja) 2022-08-31
KR102605969B1 (ko) 2023-11-23
US20210390695A1 (en) 2021-12-16
EP3992851A4 (en) 2022-08-10
US11900647B2 (en) 2024-02-13

Similar Documents

Publication Publication Date Title
WO2020259666A1 (zh) 图像分类方法、装置、设备、存储介质和医疗电子设备
Han et al. Learning more with less: Conditional PGGAN-based data augmentation for brain metastases detection using highly-rough annotation on MR images
WO2020238734A1 (zh) 图像分割模型的训练方法、装置、计算机设备和存储介质
US12118739B2 (en) Medical image processing method, apparatus, and device, medium, and endoscope
Pang et al. CTumorGAN: a unified framework for automatic computed tomography tumor segmentation
WO2020125498A1 (zh) 心脏磁共振图像分割方法、装置、终端设备及存储介质
CN107563434B (zh) 一种基于三维卷积神经网络的脑部mri图像分类方法、装置
WO2020248898A1 (zh) 图像处理方法、装置、设备和存储介质
TW202125415A (zh) 三維目標檢測及模型的訓練方法、設備、儲存媒體
CN110458829A (zh) 基于人工智能的图像质控方法、装置、设备及存储介质
WO2021159811A1 (zh) 青光眼辅助诊断装置、方法及存储介质
CN113989293A (zh) 图像分割方法及相关模型的训练方法和装置、设备
Cai et al. Identifying architectural distortion in mammogram images via a se-densenet model and twice transfer learning
Wen et al. Pulmonary nodule detection based on convolutional block attention module
Guo et al. Blind image quality assessment for pathological microscopic image under screen and immersion scenarios
Wen et al. A-PSPNet: A novel segmentation method of renal ultrasound image
Saednia et al. A cascaded deep learning framework for segmentation of nuclei in digital histology images
CN118262121A (zh) 图像特征提取方法、装置、电子设备及存储介质
CN104766068A (zh) 一种多规则融合的随机游走舌像提取方法
Sibille et al. Whole-body tumor segmentation of 18f-fdg pet/ct using a cascaded and ensembled convolutional neural networks
CN114972297A (zh) 口腔健康监测方法及装置
CN113808130A (zh) 肿瘤图像智能分类方法、装置和医学图像处理设备
Tawfeeq et al. Predication of Most Significant Features in Medical Image by Utilized CNN and Heatmap.
CN110415239B (zh) 图像处理方法、装置、设备、医疗电子设备以及介质
CN114708973B (zh) 一种用于对人体健康进行评估的设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20831068

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021548679

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 20217028280

Country of ref document: KR

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2020831068

Country of ref document: EP