WO2021032062A1 - 图像处理模型生成方法、图像处理方法、装置及电子设备 - Google Patents

图像处理模型生成方法、图像处理方法、装置及电子设备 Download PDF

Info

Publication number
WO2021032062A1
WO2021032062A1 PCT/CN2020/109611 CN2020109611W WO2021032062A1 WO 2021032062 A1 WO2021032062 A1 WO 2021032062A1 CN 2020109611 W CN2020109611 W CN 2020109611W WO 2021032062 A1 WO2021032062 A1 WO 2021032062A1
Authority
WO
WIPO (PCT)
Prior art keywords
layer
image
lesion
image processing
training sample
Prior art date
Application number
PCT/CN2020/109611
Other languages
English (en)
French (fr)
Inventor
史永明
吴琼
欧歌
王纯
Original Assignee
京东方科技集团股份有限公司
北京京东方技术开发有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司, 北京京东方技术开发有限公司 filed Critical 京东方科技集团股份有限公司
Priority to US17/423,439 priority Critical patent/US11887303B2/en
Publication of WO2021032062A1 publication Critical patent/WO2021032062A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/20—ICT specially adapted for the handling or processing of medical images for handling medical images, e.g. DICOM, HL7 or PACS
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion

Definitions

  • the present disclosure relates to the field of computer technology, in particular to an image processing model generation method, an image processing method, an image processing model generation device, an image processing device, a computer-readable storage medium, and an electronic device .
  • an image processing model generation method includes: inputting at least one training sample focus image of a known disease type into an initial image processing model, wherein the initial image processing model includes a classification layer and an annotation layer, and the training The sample lesion image includes the initial center point coordinates, initial length, and initial width corresponding to the lesion area; the classification layer is called to classify the training sample lesion image to obtain the training sample lesion image corresponding to each known disease category Classification probability; calling the labeling layer to process the training sample lesion image to obtain the predicted center point coordinates, predicted length, and predicted width of the lesion area contained in the training sample lesion image; according to the at least one training The classification probability, the initial center point coordinates, the initial length, the initial width, the predicted center point coordinates, the predicted length, and the predicted width corresponding to the sample lesion image, and the at least one training The loss value of the sample lesion image in the initial image processing model; determine whether the loss value is within the preset range, and if the loss value is not within the prese
  • the invoking the classification layer to perform classification processing on the training sample lesion image to obtain the classification probability of the training sample lesion image corresponding to each known disease category includes: At least one set of one-dimensional feature maps of the image is input to the classification layer; the classification layer is called, the one-dimensional feature maps are classified, and the classification probabilities of the training sample lesion images corresponding to each known disease category are output.
  • the calling the annotation layer to process the training sample lesion image to obtain the predicted center point coordinates, predicted length, and predicted width of the lesion area contained in the training sample lesion image includes : Input the processed groups of two-dimensional feature maps of the training sample lesion image into the annotation layer; call the annotation layer to determine the largest feature point in the two-dimensional feature map, and the largest feature point corresponding The maximum feature value and two-dimensional coordinates of the; according to the maximum feature point, the maximum feature value, the two-dimensional coordinates and a preset feature threshold, determine the prediction center corresponding to the focus area contained in the training sample focus image Point coordinates, predicted length and predicted width.
  • the two-dimensional coordinates include a first coordinate value in the horizontal axis direction and a second coordinate value in the vertical axis direction.
  • the width includes: determining the two-dimensional coordinates as the predicted center point coordinates; calculating the absolute value of the characteristic difference between the maximum characteristic value and the preset characteristic threshold; according to the absolute value of the characteristic difference And the two-dimensional coordinates of the largest feature point, acquiring the first feature point and the second feature point in the horizontal axis direction in the two-dimensional feature map, and the third feature point in the vertical axis direction And a fourth feature point; obtaining a first coordinate value of the first feature point in the horizontal axis direction, and a second coordinate value of the second feature point in the horizontal axis direction; obtaining the first feature point The third coordinate value of the three characteristic points in the longitudinal axis direction, and the fourth coordinate value of the fourth characteristic point in the longitudinal axis direction.
  • the predicted width is calculated; based on the third coordinate value and the fourth coordinate value, the predicted length is calculated.
  • the initial image processing model further includes: a global average pooling layer.
  • the method further includes: inputting multiple sets of two-dimensional feature maps of the training sample lesion image into the global average pooling layer; invoking the global average The pooling layer performs global average pooling on the multiple sets of two-dimensional feature maps to obtain multiple sets of one-dimensional feature maps corresponding to the multiple sets of two-dimensional feature maps.
  • the initial image processing model further includes: a fully connected layer. After the invoking the global average pooling layer, and before the invoking the classification layer, the method further includes: invoking the fully connected layer, and comparing the multiple groups of one obtained after the global average pooling. Perform feature extraction on the one-dimensional feature map to obtain at least one set of one-dimensional feature maps of the training sample lesion image, and input the at least one set of one-dimensional feature maps into the classification layer.
  • the initial image processing model further includes: a feature weighted summation layer and an upsampling layer.
  • the method further includes: converting multiple sets of two-dimensional feature maps of the training sample lesion image, and after the global average pooling The multiple sets of one-dimensional feature maps obtained later are input to the feature weighted summation layer; the feature weighted summation layer is called, and the multiple sets of one-dimensional feature maps are compared to the multiple sets of two-dimensional feature maps.
  • the initial image processing model further includes a neural network.
  • the neural network includes at least one layer, and each layer includes a convolutional layer, an activation function, and a pooling layer in turn.
  • the method further includes: inputting the training sample lesion image into the neural network; calling the neural network, The training sample lesion image sequentially passes through the convolutional layer, activation function, and pooling layer of each layer of the neural network to obtain multiple sets of two-dimensional feature maps of the training sample lesion image, and the multiple sets of two
  • the dimensional feature map is input to the global average pooling layer; in the case that the initial image processing model further includes a feature weighted summation layer, the multiple sets of two-dimensional feature maps are also input to the feature weighted summation layer.
  • the classification probability, the initial center point coordinates, the initial length, the initial width, the predicted center point coordinates, the The prediction length and the prediction width, and obtaining the loss value of the at least one training sample lesion image in the initial image processing model includes: calculating the classification loss value according to the classification probability corresponding to the training sample lesion image Calculate the position loss value according to the initial center point coordinates, the initial length, the initial width, the predicted center point coordinates, the predicted length, and the predicted width corresponding to the training sample lesion image; According to the classification loss value and the location loss value, a loss value of the training sample lesion image in the initial image processing model is obtained.
  • an image processing method including: inputting an image of a lesion to be processed into a target image processing model, the target image processing model being trained by the methods described in some of the above embodiments, the target image processing model including classification Layer and labeling layer; call the classification layer to classify the image of the lesion to be processed to obtain the disease classification probability corresponding to the image of the lesion to be processed; call the label layer to process the image of the lesion to be processed , Obtain the regional center point coordinates, region length, and region width of the lesion area included in the to-be-processed lesion image; determine the type of disease corresponding to the to-be-processed lesion image according to the disease classification probability; according to the regional center point The coordinates, the length of the area and the width of the area determine the lesion marking area in the image of the lesion to be processed.
  • the invoking the classification layer to classify the image of the to-be-processed lesion to obtain the disease classification probability corresponding to the image of the to-be-processed lesion includes: comparing at least one of the image of the to-be-processed lesion Group one-dimensional feature maps into the classification layer; call the classification layer, perform classification processing on the one-dimensional feature maps, and output disease classification probabilities corresponding to the to-be-processed lesion image.
  • the call the annotation layer to process the image of the lesion to be processed to obtain the coordinates of the center point of the area, the length of the area, and the width of the area corresponding to the lesion area contained in the image of the lesion to be processed.
  • the method includes: inputting the processed multiple sets of two-dimensional feature maps of the to-be-processed lesion image into the annotation layer; calling the annotation layer to determine the largest feature point in the two-dimensional feature map, and the largest feature point Corresponding maximum feature value and two-dimensional coordinates; according to the maximum feature point, the maximum feature value, the two-dimensional coordinates and a preset feature threshold, determine the area corresponding to the lesion area contained in the to-be-processed lesion image Center point coordinates, area length and area width.
  • the two-dimensional coordinates include a first coordinate value in the horizontal axis direction and a second coordinate value in the vertical axis direction. Said determining, according to the maximum feature point, the maximum feature value, the two-dimensional coordinates and a preset feature threshold, the coordinates of the center point of the area, the length of the area, and the width of the area of the lesion area included in the image of the to-be-processed lesion , Including: determining the two-dimensional coordinates as the coordinates of the center point of the area; calculating the absolute value of the characteristic difference between the maximum characteristic value and the preset characteristic threshold; and according to the sum of the absolute value of the characteristic difference.
  • an image processing model generation device including: a sample image input component, a classification probability acquisition component, a prediction data acquisition component, a loss value acquisition component, and a target model generation component.
  • the sample image input component is configured to input at least one training sample lesion image of a known disease type into an initial image processing model; wherein, the initial image processing model includes a classification layer and an annotation layer, and the training sample lesion image Including the initial center point coordinates, initial length and initial width corresponding to the lesion area.
  • the classification probability acquisition component is configured to call the classification layer, perform classification processing on the training sample lesion image, and obtain the classification probability of the training sample lesion image corresponding to each known disease category.
  • the prediction data acquisition component is configured to call the annotation layer to process the training sample lesion image to obtain the predicted center point coordinates, predicted length, and predicted width of the lesion area included in the training sample lesion image.
  • the loss value acquiring component is configured to, according to the classification probability, the initial center point coordinates, the initial length, the initial width, the predicted center point coordinates, and the corresponding to the at least one training sample lesion image
  • the prediction length and the prediction width are obtained, and the loss value of the at least one training sample lesion image in the initial image processing model is acquired.
  • the target model generating component is configured to determine whether the loss value is within a preset range, and if the loss value is not within the preset range, update the parameters of the initial image processing model, and update the parameterized image
  • the processing model is used as the initial image processing model for the next training; continue to train the image processing model with updated parameters until the loss value is within the preset range, and the image processing model trained last time is used as the target image processing model .
  • the initial image processing model further includes a neural network, a global average pooling layer, a fully connected layer, a feature weighted summation layer, and an upsampling layer; wherein the neural network includes at least one layer, each The layers include convolutional layer, activation function and pooling layer in turn.
  • the image processing model generation device further includes: a first two-dimensional feature map acquisition component coupled to the sample image input component, configured to invoke the neural network, and sequentially pass the training sample lesion image through the nerve
  • the convolutional layer, activation function and pooling layer of each layer of the network obtain multiple sets of two-dimensional feature maps of the training sample lesion image.
  • the image processing model generation device further includes: a first global average pooling component coupled to the first two-dimensional feature map acquisition component, configured to call the global average pooling layer, and perform a comparison of the two groups of two Performing global average pooling on the dimensional feature map to obtain multiple sets of one-dimensional feature maps corresponding to the multiple sets of two-dimensional feature maps;
  • the image processing model generating device further includes: a first fully connected component coupled to the first global average pooling component, the first fully connected component is also coupled to the classification probability acquisition component, and the second A fully connected component is configured to call the fully connected layer, perform feature extraction on the multiple sets of one-dimensional feature maps to obtain at least one set of one-dimensional feature maps of the training sample lesion image, and combine the at least one set of The one-dimensional feature map is input to the classification probability acquisition component.
  • the image processing model generating device further includes: a first feature weighted summation component coupled to the first two-dimensional feature map acquisition component and the first global average pooling component, and is configured to invoke the feature weighting
  • the summation layer performs feature weighted summation on each group of two-dimensional feature maps in the multiple sets of two-dimensional feature maps according to the multiple sets of one-dimensional feature maps.
  • the image processing model generation device further includes: a first up-sampling component coupled to the first feature weighted summation component, the first up-sampling component is also coupled to the predicted data acquisition component, the first An up-sampling component is configured to call the up-sampling layer, perform up-sampling processing on the multiple sets of two-dimensional feature maps that have undergone the feature weighted summation, and input the multiple sets of two-dimensional feature maps that have undergone the up-sampling processing The prediction data acquisition component.
  • an image processing device which includes: an image input component to be processed, a disease classification probability acquisition component, a focus area data acquisition component, a disease type determination component, and a focus area determination component.
  • the to-be-processed image input component is configured to input the to-be-processed lesion image into a target image processing model; wherein, the target image processing model is obtained by training the methods described in some of the above examples, and the target image processing model includes classification Layer and label layer.
  • the disease classification probability acquisition component is configured to call the classification layer to perform classification processing on the to-be-processed lesion image to obtain the disease classification probability corresponding to the to-be-processed lesion image.
  • the lesion area data acquisition component is configured to call the annotation layer to process the to-be-processed lesion image to obtain the area center point coordinates, the area length and the area width of the lesion area contained in the to-be-processed lesion image.
  • the disease type determining component is configured to determine the disease type corresponding to the to-be-processed focus image according to the disease classification probability.
  • the lesion area determining component is configured to determine the lesion marking area in the image of the to-be-processed lesion according to the coordinates of the center point of the area, the length of the area, and the width of the area.
  • the target image processing model further includes a neural network, a global average pooling layer, a fully connected layer, a feature-weighted summation layer, and an upsampling layer; wherein the neural network includes at least one layer, each The layers include convolutional layer, activation function and pooling layer in turn.
  • the image processing device further includes: a second two-dimensional feature map acquiring component coupled to the to-be-processed image input component, configured to call the neural network, and sequentially pass the to-be-processed lesion image through the neural network
  • the convolutional layer, activation function and pooling layer of each layer of obtain multiple sets of two-dimensional feature maps of the image of the to-be-processed lesion.
  • the image processing device further includes: a second global average pooling component coupled to the second two-dimensional feature map acquiring component, configured to call the global average pooling layer, and compare the multiple sets of two-dimensional features
  • the graph is globally averaged pooled to obtain multiple sets of one-dimensional feature maps corresponding to the multiple sets of two-dimensional feature maps.
  • the image processing device further includes: a second fully connected component coupled to the second global average pooling component, the second fully connected component is also coupled to the disease classification probability acquisition component, the second The fully connected component is configured to call the fully connected layer, perform feature extraction on the multiple sets of one-dimensional feature maps, obtain at least one set of one-dimensional feature maps of the to-be-processed lesion image, and combine the at least one set of one-dimensional feature maps.
  • the dimensional feature map is input to the disease classification probability acquisition component.
  • the image processing device further includes: a second feature weighted summation unit coupled to the second two-dimensional feature map acquisition unit and the second global average pooling unit, and is configured to call the feature weighted summation Layer, according to the multiple sets of one-dimensional feature maps, perform feature weighted summation on each set of two-dimensional feature maps in the multiple sets of two-dimensional feature maps.
  • a second feature weighted summation unit coupled to the second two-dimensional feature map acquisition unit and the second global average pooling unit, and is configured to call the feature weighted summation Layer, according to the multiple sets of one-dimensional feature maps, perform feature weighted summation on each set of two-dimensional feature maps in the multiple sets of two-dimensional feature maps.
  • the image processing device further includes a second up-sampling component coupled to the second feature weighted sum component, the second up-sampling component is also coupled to the lesion area data acquisition component, and the second The up-sampling component is configured to call the up-sampling layer, perform up-sampling processing on the multiple sets of two-dimensional feature maps that have undergone the feature weighted summation, and input the multiple sets of two-dimensional feature maps that have undergone the up-sampling processing into the office.
  • the data acquisition component of the lesion area is configured to call the up-sampling layer, perform up-sampling processing on the multiple sets of two-dimensional feature maps that have undergone the feature weighted summation, and input the multiple sets of two-dimensional feature maps that have undergone the up-sampling processing into the office.
  • the data acquisition component of the lesion area is configured to call the up-sampling layer, perform up-sampling processing on the multiple sets of two-dimensional feature maps that have undergone the feature weighted summ
  • a computer-readable storage medium stores computer program instructions.
  • the processor executes some of the above-mentioned embodiments.
  • a computer program product in another aspect, includes computer program instructions, and when the computer program instructions are executed on a computer, the computer program instructions cause the computer to execute one or more steps in the image processing model generation method described in some of the above embodiments, And/or, one or more steps in the image processing method described in some of the above embodiments.
  • a computer program When the computer program is executed on a computer, the computer program causes the computer to execute one or more steps in the image processing model generation method described in some of the above embodiments, and/or, as described in some of the above embodiments One or more steps in the image processing method.
  • an electronic device which includes a processor, a memory, and a computer program stored on the memory and capable of being run on the processor.
  • the processor executes the computer program, it implements one or more steps in the image processing model generation method described in some of the above embodiments, and/or, as in the image processing method described in some of the above embodiments One or more steps.
  • Fig. 1 is a flowchart of an image processing model generation method according to some embodiments
  • Figure 2 is a schematic diagram of a training sample lesion image according to some embodiments.
  • Figure 3 is a structural diagram of an initial image processing model according to some embodiments.
  • Fig. 4 is a flowchart of another image processing model generation method according to some embodiments.
  • 5 to 8 are flowcharts of S2, S3, S33, and S4 in the image processing model generation method according to some embodiments;
  • Fig. 9 is a structural diagram of a target image processing model according to some embodiments.
  • Fig. 10 is a flowchart of an image processing method according to some embodiments.
  • FIG. 11 is a schematic diagram of a focus image to be processed according to some embodiments.
  • FIG. 12 is a flowchart of another image processing method according to some embodiments.
  • FIG. 13-15 are flowcharts of S200, S300, and S303 in image processing methods according to some embodiments.
  • Fig. 16 is a structural diagram of an image processing model generating apparatus according to some embodiments.
  • FIG. 17 is a structural diagram of another image processing model generating apparatus according to some embodiments.
  • Fig. 18 is a structural diagram of an image processing apparatus according to some embodiments.
  • Fig. 19 is a structural diagram of another image processing apparatus according to some embodiments.
  • first and second are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined with “first” and “second” may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present disclosure, unless otherwise specified, “plurality” means two or more.
  • the expressions “coupled” and “connected” and their extensions may be used.
  • the term “connected” may be used when describing some embodiments to indicate that two or more components are in direct physical or electrical contact with each other.
  • the term “coupled” may be used when describing some embodiments to indicate that two or more components have direct physical or electrical contact.
  • the term “coupled” or “communicatively coupled” may also mean that two or more components are not in direct contact with each other, but still cooperate or interact with each other.
  • the embodiments disclosed herein are not necessarily limited to the content herein.
  • a and/or B includes the following three combinations: A only, B only, and the combination of A and B.
  • the term “if” is optionally interpreted to mean “when” or “when” or “in response to determination” or “in response to detection.”
  • the phrase “if it is determined" or “if [the stated condition or event] is detected” is optionally interpreted to mean “when determining" or “in response to determining" Or “when [the stated condition or event] is detected” or “in response to the detection of [stated condition or event]”.
  • classification algorithms based on deep learning can well diagnose the types of diseases.
  • the inventors of the present disclosure have discovered in research that these classification algorithms have certain differences between the regions of interest when identifying a picture as a certain type of disease in the process of medical image processing. Although some algorithms can be used to roughly mark the location of the lesion on medical images, this marking method is too rough to reflect the specific location of the lesion on the diseased tissue.
  • some embodiments of the present disclosure provide a method for generating an image processing model. As shown in FIG. 1, the method includes the following S1 to S7.
  • S1 Input at least one training sample focus image of a known disease type into the initial image processing model 1 (as shown in FIG. 2).
  • the initial image processing model 1 includes a classification layer 1007 (Classification) and a label layer 1010 (Attention Location).
  • the training sample lesion image 1001 includes the initial center point B coordinates (G x , G y ), initial length G h and initial width G w corresponding to the lesion area.
  • the initial image processing model 1 refers to an image processing model that has not been trained or is in the process of training.
  • the known disease category refers to the disease category corresponding to the training sample lesion image 1001.
  • the types of known diseases may be diseases such as pneumonia, heart disease, cirrhosis, etc., or other types of diseases, depending on specific needs.
  • the initial image processing model 1 includes a classification layer 1007 and an annotation layer 1010, and both the classification layer 1007 and the annotation layer 1010 are network layers that can process lesion images.
  • the training sample lesion image 1001 is used to train the initial image processing model 1, at least one training sample lesion image 1001 of a known disease type can be obtained, and at least one training sample lesion image 1001 is input into the initial image processing model 1 to correct The initial image processing model 1 is trained.
  • the number of training sample lesion images 1001 input to the initial image processing model 1 can be one or more, for example, 500, 800, or 1000, etc., and can also be determined according to the actual situation. No specific restrictions.
  • the training sample lesion image 1001 includes a lesion area, and the lesion area refers to the position of the lesion in the training sample lesion image 1001, and its position is represented in a certain area range in the training sample lesion image 1001.
  • the lesion area may be represented as a rectangular area, and the lesion area may be a real lesion area previously marked in the training sample lesion image 1001 by a doctor or the like.
  • the real lesion area in the training sample lesion image 1001 is the area shown in the real box TB (Truth box).
  • the real box TB is based on the initial center point B coordinates (G x , G y ), initial length G h and The initial width G w is obtained.
  • the initial center point B coordinates (G x , G y ) are the coordinates of the center point corresponding to the real lesion area in the image coordinate system corresponding to the training sample lesion image 1001; the initial length G h and the initial width G w are real respectively The length and width of the lesion area.
  • the training sample lesion image 1001 can be sequentially input to the initial image processing model 1 to train the initial image processing model 1 and execute S2.
  • S2 Call the classification layer 1007, perform classification processing on the training sample lesion image 1001, and obtain the classification probability of the training sample lesion image 1001 corresponding to each known disease category.
  • the classification layer 1007 is a network layer that can be used to classify the disease type corresponding to the lesion image.
  • the classification layer 1007 can obtain the classification probability of each disease type corresponding to the lesion image, thereby obtaining the disease type corresponding to the lesion image, such as Pneumonia, heart disease, etc.
  • the training sample lesion image 1001 may only include lesions of one disease, or may include lesions of multiple diseases.
  • the training sample lesion image 1001 includes lesions of three diseases, and the classification layer 1002 can obtain training The classification probabilities of the three diseases corresponding to the sample lesion image 1001 are determined according to the classification probabilities of each disease to determine the types of diseases included in the training sample lesion image 1001.
  • S3 Call the annotation layer 1010, process the training sample lesion image 1001, and obtain the predicted center point coordinates, predicted length, and predicted width of the lesion area contained in the training sample lesion image 1001.
  • the labeling layer 1010 is a network layer that can be used to label the specific location of the lesion on the diseased tissue.
  • the labeling layer 1010 can add a labeling frame to the lesion area on the lesion image.
  • the labeling frame can be a rectangular frame or a circle. Shape frame and so on.
  • the training sample lesion image 1001 is input to the initial image processing model 1, and the label layer 1010 is processed to output a prediction box PB (Pred box).
  • the prediction box PB is based on the coordinates of the predicted center point A (P x , P y ) , Predicted length P h and predicted width P w .
  • the predicted center point A coordinates (P x , P y ) are the coordinates of the center point corresponding to the predicted lesion area in the image coordinate system corresponding to the training sample lesion image 1001;
  • the predicted length P h and predicted width P w are respectively the predicted The length and width of the lesion area.
  • S2 and S3 can be executed simultaneously; S2 and S3 can also be executed first, or S3 and S2 can be executed first.
  • the classification probability of the disease type of the training sample lesion image 1001 and the prediction data of the lesion area are obtained from the above S2 and S3, and S4 can be executed.
  • the loss value L can be based on the classification probability corresponding to the training sample lesion image 1001, the initial center point B coordinates (G x , G y ), the initial length G h , the initial width G w , and the predicted center point A coordinates (P x , P y ) , The predicted length P h and the predicted width P w are calculated.
  • a training sample lesion image 1001 is input in a training process, and a training sample lesion image 1001 can obtain a loss value L for one disease. If a training sample lesion image 1001 includes multiple diseases, Then, according to the weights of multiple diseases and the corresponding loss values, the loss values of multiple diseases in a training sample lesion image 1001 are averaged to obtain the loss value L of a training sample lesion image 1001.
  • multiple training sample lesion images 1001 are input in one training process, and the loss value L of each training sample lesion image 1001 is obtained by the method of the above embodiment, and the loss value of multiple training sample lesion images 1001 L is averaged to obtain the loss value L of multiple training sample lesion images 1001.
  • the loss values of the training sample lesion images are obtained from the above S4, and S5 can be executed, see FIG. 3.
  • S5 Determine whether the loss value L is within the preset range; if the loss value L is not within the preset range, execute S6; if the loss value L is within the preset range, execute S7.
  • S6 Update the parameters of the initial image processing model 1, and use the updated image processing model as the initial image processing model 1 for the next training.
  • S7 Use the currently trained image processing model (that is, the last trained image processing model) as the target image processing model 2 after training.
  • the preset range refers to the range to be compared with the loss value L, which is a numerical range set in advance.
  • loss value L is within the preset range, it means that the training result of the current training image processing model has reached the expected result, and the currently trained image processing model (that is, the last trained image processing model) can be used as the target after training Image processing model 2.
  • the loss value L is not within the preset range, it means that the training result of the initial image processing model 1 did not meet expectations. Based on the result of the loss value L, the parameters of the initial image processing model 1 need to be updated, and the image processing after the updated parameters The model is used as the initial image processing model 1 for the next training, and continues to input at least one training sample lesion image 1001 of a known disease type to train the initial image processing model 1.
  • the image processing model generated by the image processing model generation method provided by the embodiments of the present disclosure can accurately identify the lesion area in the lesion image, and add corresponding annotation coordinates, which can reflect the specific location of the lesion on the lesion tissue, where
  • the classification layer 1007 and the annotation layer 1010 train the area of interest (ie the marked lesion area) of the image processing model to make the position of the marked lesion area more accurate.
  • the trained image processing model can automatically identify the medical images contained The type of disease and the location of the lesion area of each type of disease are marked, which saves manpower and improves efficiency, and the generated image processing model has high detection accuracy for classifying and marking the type of disease.
  • the initial image processing model 1 further includes: Neural Networks 1000 (Neural Networks), the neural network 1000 includes at least one layer, and each layer in turn includes a convolutional layer 1002 (Convolutional Layer), Activation function 1003 (Activation Function) and pooling layer 1007 (Pooling Layer).
  • Neural Networks 1000 Neural Networks
  • the neural network 1000 includes at least one layer, and each layer in turn includes a convolutional layer 1002 (Convolutional Layer), Activation function 1003 (Activation Function) and pooling layer 1007 (Pooling Layer).
  • the image processing model generation method in the embodiment of the present disclosure further includes: performing S11 after S1.
  • S11 Call the neural network 1000, pass the training sample lesion image 1001 through the convolutional layer 1002, activation function 1003, and pooling layer 1004 of each layer of the neural network 1000 in turn to obtain multiple sets of two-dimensional feature maps of the training sample lesion 1001 image .
  • the convolutional layer 1002 is used to extract features; the activation function 1003 is used to activate the extracted features; the pooling layer 1004 is used to down-sample the activated features; through multi-layer superposition, training sample lesions can be obtained Multiple sets of two-dimensional feature maps of the image 1001.
  • the neural network 1000 in the above-mentioned embodiments of the present disclosure may be a convolutional neural network (Convolutional Neural Networks, CNN for short), a Recurrent Neural Networks (RNN for short), etc.
  • CNN convolutional Neural Networks
  • RNN Recurrent Neural Networks
  • the neural network 1000 in the above-mentioned embodiments of the present disclosure may be a long short-term memory network (Long Short-Term Memory, LSTM), artificial neural network (Artificial Neural Networks, ANNs), etc.
  • LSTM Long Short-Term Memory
  • ANNs Artificial Neural Networks
  • the initial image processing model 1 further includes: a global average pooling layer 1005 (Global Average Pooling, GAP for short).
  • GAP Global Average Pooling
  • the image processing model generation method in the embodiment of the present disclosure further includes: performing S11A after S11.
  • S11A Call the global average pooling layer 1005 to perform global average pooling on multiple sets of two-dimensional feature maps to obtain multiple sets of one-dimensional feature maps corresponding to multiple sets of two-dimensional feature maps.
  • the multiple sets of two-dimensional feature maps obtained by the training sample lesion image 1001 through the neural network 1000 are input into the global average pooling layer 1005, and the global average pooling layer 1005 is called to perform global average pooling on multiple sets of two-dimensional feature maps to obtain the corresponding multiple Multiple groups of one-dimensional feature maps of two-dimensional feature maps.
  • the initial image processing model 1 further includes: a fully connected layer 1006 (Fully Connected Layers).
  • the image processing model generation method in the embodiment of the present disclosure further includes: performing S12 after S11A.
  • S12 Call the fully connected layer 1006 to perform feature extraction on multiple sets of one-dimensional feature maps obtained after global average pooling, to obtain at least one set of one-dimensional feature maps of the training sample lesion image 1001.
  • the multiple sets of one-dimensional feature maps obtained by processing the training sample lesion image 1001 through the global average pooling layer 1005 are input into the fully connected layer 1006, the fully connected layer 1006 is called, and the multiple sets of one-dimensional feature maps obtained after the global average pooling are processed By feature extraction, at least one set of one-dimensional feature maps of the training sample lesion image 1001 is obtained, and the at least one set of one-dimensional feature maps are input into the classification layer 1007.
  • S2 calls the classification layer 1007 to classify the training sample lesion image 1001 to obtain the classification probability of the training sample lesion image 1001 corresponding to each known disease category, including:
  • S21 Input at least one set of one-dimensional feature maps of the training sample lesion image 1001 into the classification layer 1007.
  • S22 Call the classification layer 1007, perform classification processing on the one-dimensional feature map, and output the classification probability of the training sample lesion image 1001 corresponding to each known disease type.
  • the initial image processing model 1 further includes: a feature weighted summation layer 1008 (English name: FWS) and an upsampling layer 1009 (Upsampling Layers).
  • FWS feature weighted summation layer
  • upsampling layer 1009 Upsampling Layers
  • the image processing model generation method in the embodiment of the present disclosure further includes: performing S11B after S11.
  • S11B Call the feature weighted summation layer 1008, and perform feature weighted summation on each group of two-dimensional feature maps in the multiple sets of two-dimensional feature maps according to multiple sets of one-dimensional feature maps.
  • the multiple sets of two-dimensional feature maps of the training sample lesion image 1001 and the multiple sets of one-dimensional feature maps obtained after global average pooling are input to the feature weighted summation layer 1008, and the feature weighted summation layer 1008 is called.
  • One-dimensional feature map which performs feature weighted summation on each group of two-dimensional feature maps in multiple sets of two-dimensional feature maps.
  • S13 is executed after S11B.
  • S13 Invoke the up-sampling layer 1009, perform up-sampling processing on multiple sets of two-dimensional feature maps that have been feature-weighted and summed, to obtain multiple sets of two-dimensional feature maps processed by the training sample lesion image 1001.
  • S3 calls the annotation layer 1010 to process the training sample lesion image 1001 to obtain the predicted center point A coordinate (P) of the lesion area contained in the training sample lesion image 1001 x , P y ), the predicted length P h and the predicted width P w , including S31 to S33.
  • S31 Input the processed multiple sets of two-dimensional feature maps of the training sample lesion image 1001 into the annotation layer 1010;
  • S32 Call the labeling layer 1010 to determine the largest feature point in the two-dimensional feature map, and the largest feature value and two-dimensional coordinates corresponding to the largest feature point.
  • S33 Determine the predicted center point A coordinates (P x , P y ) and predicted length P corresponding to the lesion area contained in the training sample lesion image 1001 according to the maximum feature point, maximum feature value, two-dimensional coordinates and preset feature threshold h and the predicted width P w .
  • the preset feature threshold can be adjusted according to the expected range of the predicted length P h and the predicted width P w of the lesion area. Generally, when the preset feature threshold is larger, the obtained lesion area prediction The greater the length P h and the predicted width P w , and vice versa.
  • S33 determines the predicted center point A coordinate (P) corresponding to the lesion area contained in the training sample lesion image 1001 according to the maximum feature point, the maximum feature value, two-dimensional coordinates, and a preset feature threshold.
  • x , P y ), prediction length P h, and prediction width P w including S331 to S335.
  • S331 Determine the two-dimensional coordinates of the largest feature point as the predicted center point A coordinates (P x , P y ).
  • S332 Calculate the absolute value of the characteristic difference between the maximum characteristic value and the preset characteristic threshold.
  • S334 Obtain the first coordinate value P x -t1 of the first feature point A1 in the horizontal axis X direction , and the second coordinate value P x +t2 of the second feature point A2 in the horizontal axis X direction, where t1, t2 None is less than zero.
  • S4 is based on the classification probability corresponding to at least one training sample lesion image 1001, the initial center point B coordinate (G x , G y ), the initial length G h , the initial width G w , and the prediction
  • the center point A coordinates (P x , P y ), the predicted length P h and the predicted width P w , and the loss value L of at least one training sample lesion image 1001 in the initial image processing model 1 is obtained, including S41-S43.
  • the loss value L includes a classification loss value and a location loss value.
  • the classification loss value means that the initial image processing model 1 classifies the training sample lesion image 1001 to obtain the classification loss with the classification probability, and the classification loss value is recorded as L cls .
  • the classification loss value L cls can be obtained according to the classification probabilities.
  • softmax and SVM full English name: Support Vector Machine, Chinese name: The classification loss value L cls is calculated by means of support vector machine) or sigmoid.
  • the position loss value means that the initial image processing model 1 processes the training sample lesion image 1001 to determine the predicted center point A coordinates (P x , P y ) corresponding to the lesion area, the predicted length P h and the predicted width P w and the initial center Point B coordinates (G x , G y ), the initial length G h and the initial width G w exist position loss, the value of the position loss is recorded as L al .
  • the position loss value L al can be based on the initial center point B coordinates (G x , G y ), initial length G h , initial width G w , predicted center point A coordinates (P x , P y ), predicted length P h , and predicted The width P w is calculated.
  • the position loss value L al can be calculated according to the following formula (1):
  • the calculation method of the position loss value L al may also adopt the L1 norm or the L2 norm.
  • Some embodiments of the present disclosure provide an image processing method. As shown in FIG. 10, the method includes S100-S500.
  • S100 Input the to-be-processed lesion image 2001 into the target image processing model 2 (as shown in FIG. 9).
  • the target image processing model 2 is obtained by training the image processing model generation method of some of the above embodiments, and it can be used to process the lesion image in the medical field to determine the corresponding image of the lesion.
  • the focus image 2001 to be processed refers to an image in the medical field, such as an image taken by a medical instrument.
  • the target image processing model 2 includes a classification layer 2007 and an annotation layer 2010, where the classification layer 2007 can be used to classify the to-be-processed lesion image 2001 to obtain the classification probability corresponding to the to-be-processed lesion image 2001 to determine the lesion to be processed
  • the disease corresponding to the image 2001; the annotation layer 2010 can be used to determine the area where the lesion in the lesion image 2001 to be processed is located in the lesion image. This process will be described in detail in the following steps.
  • the to-be-processed lesion image 2001 After the to-be-processed lesion image 2001 is obtained, the to-be-processed lesion image 2001 can be input into the target image processing model 2, and S200 is executed.
  • S200 Call the classification layer 2007, perform classification processing on the lesion image 2001 to be processed, and obtain the disease classification probability corresponding to the lesion image 2001 to be processed.
  • the classification layer 2007 is a network layer that can be used to classify the disease type corresponding to the lesion image.
  • the classification layer 2007 can obtain the classification probability of each disease type corresponding to the lesion image, thereby obtaining the disease type corresponding to the lesion image, such as Pneumonia, heart disease, etc.
  • S300 Invoke the label layer 2010, process the lesion image 2001 to be processed, and obtain the coordinates of the center point of the area, the length of the area, and the width of the lesion area included in the lesion image 2001 to be processed.
  • the labeling layer 2010 is a network layer that can be used to label the specific location of the lesion on the diseased tissue.
  • the labeling layer 2010 can add a labeling frame to the lesion area on the lesion image.
  • the labeling frame can be a rectangular frame or a circle. Shape frame and so on.
  • the to-be-processed lesion image 2001 is input into the target image processing model 2, and is processed by the annotation layer 2010 to output the lesion frame Z.
  • the lesion frame Z is based on the coordinates of the center point B of the area (D x , D y ) and the area length D h and the area width D w . Then the coordinates of the center point B of the area (D x , D y ) are the coordinates of the center point corresponding to the focus area in the image coordinate system corresponding to the focus image 2001 to be processed; the area length D h and the area width D w are the focus area respectively Corresponding length and width.
  • S200 and S300 may be executed at the same time; S200 may be executed first, and then S300 may be executed, or S300 may be executed first, and then S200 may be executed.
  • the classification probability of the disease type of the to-be-processed focus image 2001 and the area data of the focus area are obtained respectively, and S400 may be executed.
  • S400 Determine the type of disease corresponding to the focus image 2001 to be processed according to the disease classification probability.
  • the disease classification probability is obtained in S200. If the disease classification probability for a specific disease type in the to-be-processed focus image 2001 is within a certain range, it means that the to-be-processed focus image 2001 contains the specific disease type. If the disease classification probability for a specific disease type in the to-be-processed focus image 2001 is not within a certain range, it means that the to-be-processed focus image 2001 does not include the specific disease type.
  • S500 Determine the lesion marking area in the lesion image 2001 to be processed according to the coordinates (D x , D y ) of the area center point D, the area length D h and the area width D w.
  • the coordinates (D x , D y ) of the area center point B obtained in S300 are the coordinates of the center point corresponding to the focus area in the image coordinate system corresponding to the focus image 2001 to be processed.
  • the area length D h and the area width D w are respectively Is the length and width corresponding to the lesion area, and the lesion marking area is marked in the lesion image 2001 to be processed, as shown in the lesion frame Z shown in FIG. 11.
  • the image processing method provided by the embodiments of the present disclosure utilizes a trained image processing model to automatically identify the types of diseases contained in medical images and to mark the location of the lesion area of each type of disease, which saves manpower and improves efficiency.
  • the image processing model has high detection accuracy for classification and labeling of diseases.
  • the target image processing model 2 further includes: a neural network 2000, the neural network 2000 includes at least one layer, and each layer includes a convolutional layer 2002, an activation function 2003, and a pooling layer 2004 in turn .
  • the image processing method in the embodiment of the present disclosure further includes: performing S101 after S100.
  • S101 Invoke the neural network 2000, pass the to-be-processed lesion image 2001 through the convolutional layer 2002, activation function 2003, and pooling layer 2004 of each layer of the neural network 2000 in turn to obtain multiple sets of two-dimensional feature maps of the to-be-processed lesion image 2001 .
  • the convolutional layer 2002 is used to extract features; the activation function 2003 is used to activate the extracted features; the pooling layer 2004 is used to down-sample the activated features; through multi-layer superposition, the to-be-treated lesions can be obtained Multiple sets of two-dimensional feature maps of the image 2001.
  • the neural network 2000 in the foregoing embodiments of the present disclosure may be a convolutional neural network, a cyclic neural network, or the like.
  • the neural network 2000 in the above-mentioned embodiments of the present disclosure may be a Long Short-Term Memory (LSTM), artificial neural networks (Artificial Neural Networks, ANNs), etc.
  • LSTM Long Short-Term Memory
  • ANNs Artificial Neural Networks
  • the target image processing model 2 further includes: a global average pooling layer.
  • the image processing method in the embodiment of the present disclosure further includes: performing S101A after S101.
  • S101A Call the global average pooling layer 2005 to perform global average pooling on multiple sets of two-dimensional feature maps to obtain multiple sets of one-dimensional feature maps corresponding to multiple sets of two-dimensional feature maps.
  • the target image processing model 2 further includes: a fully connected layer 2006.
  • the image processing method in the embodiment of the present disclosure further includes: performing S102 after S101A.
  • S102 Call the fully connected layer 2006 to perform feature extraction on multiple sets of one-dimensional feature maps obtained after global average pooling, to obtain at least one set of one-dimensional feature maps of the lesion image 2001 to be processed.
  • S200 calls the classification layer 2007 to classify the to-be-processed lesion image 2001 to obtain the disease classification probability corresponding to the to-be-processed lesion image 2001, including:
  • S201 Input at least one set of one-dimensional feature maps of the lesion image 2001 to be processed into the classification layer 2007.
  • S202 Call the classification layer 2007, perform classification processing on the one-dimensional feature map, and output the disease classification probability corresponding to the focus image 2001 to be processed.
  • the target image processing model 2 further includes: a feature weighted summation layer 2007 and an upsampling layer 2009.
  • the image processing method in the embodiment of the present disclosure further includes: performing S101B after S101.
  • S101B Call the feature weighted summation layer 2008, and perform feature weighted summation on each group of two-dimensional feature maps in the multiple sets of two-dimensional feature maps according to multiple sets of one-dimensional feature maps.
  • the multiple sets of two-dimensional feature maps of the lesion image 2001 to be processed and the multiple sets of one-dimensional feature maps obtained after global average pooling are input to the feature weighted summation layer 2008, and the feature weighted summation layer 2008 is called.
  • One-dimensional feature map which performs feature weighted summation on each group of two-dimensional feature maps in multiple sets of two-dimensional feature maps.
  • S103 is executed after S101B.
  • S103 Call the up-sampling layer 2009 to perform up-sampling processing on the multiple sets of two-dimensional feature maps that have been feature-weighted and summed to obtain the processed multiple sets of two-dimensional feature maps of the to-be-processed lesion image 2001, and combine the processed multiple sets of two The dimensional feature map is input to the annotation layer 2010.
  • S300 calls the annotation layer 2010 to process the to-be-processed lesion image 2001 to obtain the regional center point D coordinate (D x , D y ) of the lesion area contained in the to-be-processed lesion image 2001 , The area length D h and the area width D w , including S301 to S303.
  • S301 Input the processed multiple sets of two-dimensional feature maps of the to-be-processed lesion image 2001 into the annotation layer 2010.
  • S302 Call the annotation layer 2010 to determine the maximum feature point in the two-dimensional feature map, and the maximum feature value and two-dimensional coordinates corresponding to the maximum feature point.
  • the preset feature threshold can be adjusted according to the desired range of the area length D h and the area width D w of the lesion area.
  • the preset feature threshold is larger, the obtained area of the lesion area The greater the length D h and the area width D w , and vice versa.
  • S303 according to the maximum feature point, the maximum feature value, the two-dimensional coordinates, and the preset feature threshold, determine the regional center point D corresponding to the lesion area contained in the lesion image 2001 to be processed
  • the coordinates (D x , D y ), area length D h and area width D w include S3031 to S3035.
  • S3031 Use the two-dimensional coordinates of the largest feature point as the D coordinates (D x , D y ) of the center point of the area.
  • S3032 Calculate the absolute value of the characteristic difference between the maximum characteristic value and the preset characteristic threshold.
  • S3034 Obtain the first coordinate value D x- t11 of the first feature point D1 in the horizontal axis X direction , and the second coordinate value D x + t22 of the second feature point D2 in the horizontal axis X direction, where t11 and t22 None is less than zero.
  • some embodiments of the present disclosure provide an image processing model generating device 3, and the image processing model generating device 3 includes:
  • the sample image input component 10 is configured to input at least one training sample lesion image 1001 of a known disease type into the initial image processing model 1.
  • the initial image processing model 1 includes a classification layer 1007 and an annotation layer 1010, and the training sample lesion image 1001 includes the initial center point B coordinates (G x , G y ), initial length G h and initial width G w corresponding to the lesion area.
  • the classification probability acquisition component 20 is coupled with the sample image input component 10 and is configured to call the classification layer 1007 to classify the training sample lesion image 1001 to obtain the classification probability of the training sample lesion image 1001 corresponding to each known disease category.
  • the prediction data acquisition component 30, coupled with the sample image input component 10, is configured to call the annotation layer 1010 to process the training sample lesion image 1001 to obtain the predicted center point A coordinate of the lesion area contained in the training sample lesion image 1001 ( P x , P y ), predicted length P h, and predicted width P w .
  • the loss value acquisition component 40 is coupled to the classification probability acquisition component 20 and the prediction data acquisition component 30, and is configured according to the classification probability corresponding to the at least one training sample lesion image 1001, the initial center point B coordinates (G x , G y ), Initial length G h , initial width G w , predicted center point A coordinates, (P x , P y ) predicted length P h and predicted width P w , and obtain at least one training sample lesion image 1001. Loss in the initial image processing model 1 Value L.
  • the target model generating component 50 is coupled to the loss value acquiring component 40, and is configured to determine whether the loss value L is within a preset range, and if the loss value L is not within the preset range, update the parameters of the initial image processing model 1 , The image processing model after the updated parameters is used as the initial image processing model 1 for the next training; continue to train the image processing model after the updated parameters until the loss value L is within the preset range, and the last trained image processing model As the target image processing model 2.
  • the initial image processing model 1 further includes a neural network 1000, a global average pooling layer 1005, a fully connected layer 1006, a feature weighted summation layer 1008, and an upsampling layer 1009;
  • the network 1000 includes at least one layer, and each layer includes a convolutional layer 1002, an activation function 1003, and a pooling layer 1004 in turn.
  • the image processing model generating device 3 in the embodiment of the present disclosure further includes:
  • the first two-dimensional feature map acquiring component 11 is coupled to the sample image input component 10, and is configured to call the neural network 1000, and sequentially pass the training sample lesion image 1001 through the convolutional layer 1002 of each layer of the neural network 1000, and the activation function 1003 and the pooling layer 1004 to obtain multiple sets of two-dimensional feature maps of the training sample lesion image 1001.
  • the first global average pooling component 11A is coupled to the first two-dimensional feature map acquiring component 11, and is configured to call the global average pooling layer 1005 to perform global average pooling on multiple sets of two-dimensional feature maps to obtain corresponding multiple groups Multiple sets of one-dimensional feature maps of two-dimensional feature maps.
  • the first fully connected component 12 is coupled to the first global average pooling component 11A, the first fully connected component 12 is also coupled to the classification probability acquisition component 20, the first fully connected component 12 is configured to call the fully connected layer 1006, Perform feature extraction on multiple sets of one-dimensional feature maps to obtain at least one set of one-dimensional feature maps of the training sample lesion image 1001, and input the at least one set of one-dimensional feature maps to the classification probability acquisition component 20.
  • the first feature weighted summation component 11B is coupled to the first two-dimensional feature map acquisition component 11 and the first global average pooling component 11A, and is configured to call the feature weighted summation layer 1008, according to multiple sets of one-dimensional feature maps, Perform feature weighted summation on each set of two-dimensional feature maps in multiple sets of two-dimensional feature maps.
  • the first upsampling component 13 is coupled to the first feature weighted summation component 11B, the first upsampling component 13 is also coupled to the prediction data acquisition component 30, and the first upsampling component 13 is configured to call the upsampling layer 1009,
  • the up-sampling process is performed on the multiple sets of two-dimensional feature maps that have undergone feature weighted summation, and the multiple sets of up-sampling two-dimensional feature maps are input to the prediction data acquisition component 30.
  • the functions of the components included in the image processing model generating device 3 provided in the foregoing embodiment of the present disclosure may refer to the description of the corresponding steps in the image processing model generating method described in the foregoing embodiment.
  • the image processing model generating device 3 provided in the above-mentioned embodiments of the present disclosure can generate a target image processing model 2, which can be applied to the medical image recognition process to enable the target image processing model 2 to determine the type of disease and the location of the lesion in the medical image.
  • Automatic detection saves manpower and improves the efficiency of disease diagnosis, and the identification of the type of disease in the medical image and the labeling of the lesion location through the target image processing model 2 are more accurate.
  • some embodiments of the present disclosure further provide an image processing device 4, and the image processing device 4 includes:
  • the to-be-processed image input component 100 is configured to input the to-be-processed lesion image 2001 into the target image processing model 2.
  • the target image processing model 2 is obtained by training the image processing model generation method in some embodiments of the disclosure above, and the target image processing model 2 includes the classification layer 2007 and the label layer 2010 (as shown in Figure 9).
  • the disease classification probability acquisition component 200 is coupled to the to-be-processed image input component 100 and is configured to call the classification layer 2007 to perform classification processing on the to-be-processed focus image 2001 to obtain the disease classification probability corresponding to the to-be-processed focus image 2001.
  • the lesion area data acquisition component 300 is coupled to the to-be-processed image input component 100, and is configured to call the annotation layer 2010 to process the to-be-processed lesion image 2001 to obtain the coordinate of the regional center point D of the lesion area contained in the to-be-processed lesion image 2001 (D x , D y ), area length D h and area width D w (as shown in Figure 11).
  • the disease type determination component 400 is coupled to the disease classification probability acquisition component 200 and is configured to determine the disease type corresponding to the focus image 2001 to be processed according to the disease classification probability.
  • the lesion area determining component 500 is coupled to the lesion area data acquiring component 300, and is configured to determine the focus image to be processed according to the coordinates (D x , D y ) of the area center point D, the area length D h and the area width D w Mark the area of the lesion.
  • the target image processing model 2 further includes a neural network 2000, a global average pooling layer 2005, a fully connected layer 2006, a feature weighted summation layer 2008, and an upsampling layer 2009;
  • the network 2000 includes at least one layer, and each layer includes a convolutional layer 2002, an activation function 2003, and a pooling layer 2004 in turn.
  • the image processing device 4 in the embodiment of the present disclosure further includes:
  • the second two-dimensional feature map acquiring component 101 is coupled to the image input component 100 to be processed, and is configured to call the neural network 2000, and sequentially pass the to-be-processed lesion image 2001 through the convolutional layer 2002 of each layer of the neural network 2000 and activate Function 2003 and pooling layer 2004 to obtain multiple sets of two-dimensional feature maps of the focus image 2001 to be processed.
  • the second global average pooling component 101A is coupled to the second two-dimensional feature map acquiring component 101, and is configured to call the global average pooling layer 2005 to perform global average pooling on multiple sets of two-dimensional feature maps to obtain corresponding multiple groups Multiple sets of one-dimensional feature maps of two-dimensional feature maps.
  • the second fully connected component 102 is coupled to the second global average pooling component 101A and the disease classification probability acquisition component 200, and is configured to call the fully connected layer 2006 to perform feature extraction on multiple sets of one-dimensional feature maps to obtain the pending processing At least one set of one-dimensional feature maps of the lesion image 2001 is inputted into the disease classification probability acquisition component 200.
  • the second feature weighted summation component 101B is coupled to the second two-dimensional feature map acquisition component 101 and the second global average pooling component 101A, and is configured to call the feature weighted summation layer 2008, according to multiple sets of one-dimensional feature maps, Perform feature weighted summation on each set of two-dimensional feature maps in multiple sets of two-dimensional feature maps.
  • the second up-sampling component 103 is coupled to the second feature-weighted summation component 101B and the lesion area data acquisition component 300, and is configured to call the up-sampling layer 2009 to perform on the multiple sets of feature-weighted summation of the two-dimensional feature maps. Sampling processing is performed, and multiple sets of two-dimensional feature maps that have undergone up-sampling processing are input to the lesion area data acquiring component 300.
  • the image processing device 4 provided in the above-mentioned embodiment of the present disclosure applies the target image processing model 2 to the recognition of medical images, so that the target image processing model 2 can automatically detect the disease type and focus position of the medical image, saving Manpower improves the efficiency of disease diagnosis, and the identification of disease types in medical images and the labeling of lesion locations are more accurate through the target image processing model 2.
  • Some embodiments of the present disclosure provide a computer-readable storage medium (for example, a non-transitory computer-readable storage medium) in which computer program instructions are stored, and when the computer program instructions run on a processor , Causing the processor to execute one or more steps in the image processing model generation method described in some of the above embodiments, and/or, one of the image processing methods described in some of the above embodiments Or multiple steps.
  • a computer-readable storage medium for example, a non-transitory computer-readable storage medium
  • the foregoing computer-readable storage medium may include, but is not limited to: magnetic storage devices (for example, hard disks, floppy disks, or tapes, etc.), optical disks (for example, CD (Compact Disk), DVD (Digital Versatile Disk, Digital universal disk), etc.), smart cards and flash memory devices (for example, EPROM (Erasable Programmable Read-Only Memory), cards, sticks or key drives, etc.).
  • magnetic storage devices for example, hard disks, floppy disks, or tapes, etc.
  • optical disks for example, CD (Compact Disk), DVD (Digital Versatile Disk, Digital universal disk), etc.
  • smart cards and flash memory devices for example, EPROM (Erasable Programmable Read-Only Memory), cards, sticks or key drives, etc.
  • the various computer-readable storage media described in this disclosure may represent one or more devices and/or other machine-readable storage media for storing information.
  • the term "machine-readable storage medium” may include, but is not limited to, wireless channels and various other media capable of storing, containing, and/or carrying instructions and/or data.
  • Some embodiments of the present disclosure also provide a computer program product.
  • the computer program product includes computer program instructions, and when the computer program instructions are executed on a computer, the computer program instructions cause the computer to execute one or more steps in the image processing model generation method described in some of the above embodiments, And/or, one or more steps in the image processing method described in some of the above embodiments.
  • Some embodiments of the present disclosure also provide a computer program.
  • the computer program When the computer program is executed on a computer, the computer program causes the computer to execute one or more steps in the image processing model generation method described in some of the above embodiments, and/or, as some of the above embodiments One or more steps in the image processing method described in the embodiment.
  • Some embodiments of the present disclosure also provide an electronic device, including: a processor, a memory, and a computer program stored on the memory and capable of running on the processor.
  • the processor executes the computer program when the computer program is executed.
  • the processor is used to support the image processing model generation device 3, and/or the image processing device 4 executes one or more steps in the image processing model generation method described in some of the above embodiments, and/or, as One or more steps in the image processing method described in some of the above embodiments, and/or other processes used in the techniques described herein.
  • the processor can be a central processing unit (Central Processing Unit, CPU for short), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other Programming logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
  • CPU Central Processing Unit
  • DSP digital signal processors
  • ASIC application-specific integrated circuits
  • FPGA field programmable gate arrays
  • Programming logic devices discrete gates or transistor logic devices, discrete hardware components, etc.
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor.
  • the memory is used to store the program code and data of the image processing model generating device 3 and/or the program code and data of the image processing device 4 provided by the embodiment of the present disclosure.
  • the processor can execute various functions of the image processing model generating device 3 and/or various functions of the image processing device 4 by running or executing a software program stored in the memory and calling data stored in the memory.
  • the memory can be a read-only memory (Read-Only Memory, ROM) or other types of static storage devices that can store static information and instructions, random access memory (Random Access Memory, RAM), or other types that can store information and instructions
  • Dynamic storage devices can also be Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM (Compact Disc Read-Only Memory, CD-ROM) or other optical disc storage, optical disc storage ( Including compact discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program codes in the form of instructions or data structures and can be stored by a computer Any other media taken, but not limited to this.
  • the memory can exist independently and is connected to the processor through a communication bus.
  • the memory can also be integrated with the processor.

Abstract

一种图像处理模型生成方法，包括：将特定病症的多个训练样本病灶图像输入初始图像处理模型，初始图像处理模型包括分类层和标注层，训练样本病灶图像包括病灶区域对应的初始中心点坐标、初始长度和初始宽度；分类层对训练样本病灶图像进行分类处理，得到训练样本病灶图像对应各已知疾病种类的分类概率；标注层对训练样本病灶图像进行处理，得到训练样本病灶图像中所包含病灶区域的预测中心点坐标、预测长度和预测宽度；根据分类层和标注层获取的数据，获取初始图像处理模型的损失值，在损失值处于预设范围内的情况下，将最后一次训练的初始图像处理模型作为训练后的目标图像处理模型。

Description

图像处理模型生成方法、图像处理方法、装置及电子设备
本申请要求于2019年8月22日提交的、申请号为201910778807.0的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本公开涉及计算机技术领域，特别是涉及一种图像处理模型生成方法、一种图像处理方法、一种图像处理模型生成装置、一种图像处理装置、一种计算机可读存储介质及一种电子设备。
背景技术
近年来，随着医学成像采集设备的不断完善，以及图像处理、模式识别、机器学习等学科的不断发展，多学科交叉的医学图像处理和分析成为研究热点。
公开内容
一方面，提供一种图像处理模型生成方法，包括：将已知疾病种类的至少一个训练样本病灶图像输入初始图像处理模型，其中，所述初始图像处理模型包括分类层和标注层，所述训练样本病灶图像包括病灶区域对应的初始中心点坐标、初始长度和初始宽度；调用所述分类层，对所述训练样本病灶图像进行分类处理，得到所述训练样本病灶图像对应各已知疾病种类的分类概率；调用所述标注层，对所述训练样本病灶图像进行处理，得到所述训练样本病灶图像中所包含的病灶区域的预测中心点坐标、预测长度和预测宽度；根据所述至少一个训练样本病灶图像对应的所述分类概率、所述初始中心点坐标、所述初始长度、所述初始宽度、所述预测中心点坐标、所述预测长度和所述预测宽度，获取所述至少一个训练样本病灶图像在所述初始图像处理模型中的损失值；判断所述损失值是否处于预设范围内，若所述损失值不在预设范围内，则对所述初始图像处理模型的参数进行更新，将更新参数后的图像处理模型作为下一次训练的初始图像处理模型；重复上述步骤，直至所述损失值处于所述预设范围内，将最后一次训练的图像处理模型作为训练后的目标图像处理模型。
在一些实施例中，所述调用所述分类层，对所述训练样本病灶图像进行分类处理，得到所述训练样本病灶图像对应各已知疾病种类的分类概率，包括：将所述训练样本病灶图像的至少一组一维特征图输入所述分类层；调用所述分类层，对所述一维特征图进行分类处理，输出所述训练样本病灶图像对应各已知疾病种类的分类概率。
在一些实施例中，所述调用所述标注层，对所述训练样本病灶图像进行处理，得到所述训练样本病灶图像中所包含的病灶区域的预测中心点坐标、预测长度和预测宽度，包括：将所述训练样本病灶图像的经过处理的多组二维特征图输入所述标注层；调用所述标注层，确定所述二维特征图中的最大特征点，及所述最大特征点对应的最大特征值和二维坐标；根据所述最大特征点、所述最大特征值、所述二维坐标和预设特征阈值，确定所述训练样本病灶图像中所包含的病灶区域对应的预测中心点坐标、预测长度和预测宽度。
在一些实施例中，所述二维坐标包括横轴方向上的第一坐标值和纵轴方向上的第二坐标值。
所述根据所述最大特征点、所述最大特征值、所述二维坐标和预设特征阈值，确定所述训练样本病灶图像中所包含的病灶区域对应的预测中心点坐标、预测长度和预测宽度，包括：将所述二维坐标确定为所述预测中心点坐标；计算得到所述最大特征值和所述预设特征阈值之间的特征差值绝对值；根据所述特征差值绝对值和所述最大特征点的二维坐标，获取所述二维特征图中在所述横轴方向上的第一特征点和第二特征点，及在所述纵轴方向上的第三特征点和第四特征点；获取所述第一特征点在所述横轴方向上的第一坐标值，及所述第二特征点在所述横轴方向上的第二坐标值；获取所述第三特征点在所述纵轴方向上的第三坐标值，及所述第四特征点在所述纵轴方向上的第四坐标值。
基于所述第一坐标值和所述第二坐标值，计算得到所述预测宽度；基于所述第三坐标值和所述第四坐标值，计算得到所述预测长度。
在一些实施例中，所述初始图像处理模型还包括：全局平均池化层。在所述调用所述分类层和在所述调用所述标注层之前，还包括：将所述训练样本病灶图像的多组二维特征图输入所述全局平均池化层；调用所述全局平均池化层，对所述多组二维特征图进行全局平均池化，得到对应所述多组二维特征图的多组一维特征图。
在一些实施例中，所述初始图像处理模型还包括：全连接层。在所述调用所述全局平均池化层之后，以及在所述调用所述分类层之前，还包括：调用所述全连接层，对经过所述全局平均池化后得到的所述多组一维特征图进行特征提取，得到所述训练样本病灶图像的至少一组一维特征图，并将所述至少一组一维特征图输入所述分类层。
在一些实施例中，所述初始图像处理模型还包括：特征加权求和层和上采样层。在所述调用所述全局平均池化层之后，以及在所述调用所述标注层 之前，还包括：将所述训练样本病灶图像的多组二维特征图，以及经过所述全局平均池化后得到的所述多组一维特征图，输入所述特征加权求和层；调用所述特征加权求和层，根据所述多组一维特征图，对所述多组二维特征图中的每组二维特征图进行特征加权求和；调用所述上采样层，对经过所述特征加权求和的多组二维特征图进行上采样处理，得到所述训练样本病灶图像经过处理的多组二维特征图，并将所述经过处理的多组二维特征图输入所述标注层。
在一些实施例中，所述初始图像处理模型还包括：神经网络。所述神经网络包括至少一个层，每个层依次包括卷积层、激活函数和池化层。在所述将所述训练样本病灶图像的多组二维特征图输入所述全局平均池化层之前，还包括：将所述训练样本病灶图像输入所述神经网络；调用所述神经网络，将所述训练样本病灶图像依次通过所述神经网络的每个层的卷积层、激活函数和池化层，得到所述训练样本病灶图像的多组二维特征图，并将所述多组二维特征图输入所述全局平均池化层；在所述初始图像处理模型还包括特征加权求和层的情况下，所述多组二维特征图还被输入至所述特征加权求和层。
在一些实施例中，所述根据所述至少一个训练样本病灶图像对应的所述分类概率、所述初始中心点坐标、所述初始长度、所述初始宽度、所述预测中心点坐标、所述预测长度和所述预测宽度，获取所述至少一个训练样本病灶图像在所述初始图像处理模型中的损失值，包括：根据所述训练样本病灶图像对应的所述分类概率，计算得到分类损失值；根据所述训练样本病灶图像对应的所述初始中心点坐标、所述初始长度、所述初始宽度、所述预测中心点坐标、所述预测长度和所述预测宽度，计算得到位置损失值；根据所述分类损失值和所述位置损失值，获取所述训练样本病灶图像在所述初始图像处理模型中的损失值。
另一方面，提供一种图像处理方法，包括：将待处理病灶图像输入目标图像处理模型，所述目标图像处理模型通过上面一些实施例所述的方法训练得到，所述目标图像处理模型包括分类层和标注层；调用所述分类层，对所述待处理病灶图像进行分类处理，得到所述待处理病灶图像对应的疾病分类概率；调用所述标注层，对所述待处理病灶图像进行处理，得到所述待处理病灶图像中所包含病灶区域的区域中心点坐标、区域长度和区域宽度；根据所述疾病分类概率，确定所述待处理病灶图像对应的疾病种类；根据所述区域中心点坐标、所述区域长度和所述区域宽度，确定所述待处理病灶图像中的病灶标注区域。
在一些实施例中，所述调用所述分类层，对所述待处理病灶图像进行分类处理，得到所述待处理病灶图像对应的疾病分类概率，包括：将所述待处理病灶图像的至少一组一维特征图输入所述分类层；调用所述分类层，对所述一维特征图进行分类处理，输出所述待处理病灶图像对应的疾病分类概率。
在一些实施例中，所述调用所述标注层，对所述待处理病灶图像进行处理，得到所述待处理病灶图像中所包含的病灶区域对应的区域中心点坐标、区域长度和区域宽度，包括：将所述待处理病灶图像的经过处理的多组二维特征图输入所述标注层；调用所述标注层，确定所述二维特征图中的最大特征点，及所述最大特征点对应的最大特征值和二维坐标；根据所述最大特征点、所述最大特征值、所述二维坐标和预设特征阈值，确定所述待处理病灶图像中所包含的病灶区域对应的区域中心点坐标、区域长度和区域宽度。
在一些实施例中，所述二维坐标包括横轴方向上的第一坐标值和纵轴方向上的第二坐标值。所述根据所述最大特征点、所述最大特征值、所述二维坐标和预设特征阈值，确定所述待处理病灶图像中所包含的病灶区域的区域中心点坐标、区域长度和区域宽度，包括：将所述二维坐标确定为所述区域中心点坐标；计算得到所述最大特征值和所述预设特征阈值之间的特征差值绝对值；根据所述特征差值绝对值和所述最大特征点的二维坐标，获取所述二维特征图中在所述横轴方向上的第一特征点和第二特征点，及在所述纵轴方向上的第三特征点和第四特征点；获取所述第一特征点在所述横轴方向上的第一坐标值，及所述第二特征点在所述横轴方向上的第二坐标值；获取所述第三特征点在所述纵轴方向上的第三坐标值，及所述第四特征点在所述纵轴方向上的第四坐标值；基于所述第一坐标值和所述第二坐标值，计算得到所述区域宽度；基于所述第三坐标值和所述第四坐标值，计算得到所述区域长度。
又一方面，提供一种图像处理模型生成装置，包括：样本图像输入部件、分类概率获取部件、预测数据获取部件、损失值获取部件、以及目标模型生成部件。
其中，样本图像输入部件，被配置为将已知疾病种类的的至少一个训练样本病灶图像输入初始图像处理模型；其中，所述初始图像处理模型包括分类层和标注层，所述训练样本病灶图像包括病灶区域对应的初始中心点坐标、初始长度和初始宽度。
分类概率获取部件，被配置为调用所述分类层，对所述训练样本病灶图像进行分类处理，得到所述训练样本病灶图像对应各已知疾病种类的分类概 率。
预测数据获取部件，被配置为调用所述标注层，对所述训练样本病灶图像进行处理，得到所述训练样本病灶图像中所包含病灶区域的预测中心点坐标、预测长度和预测宽度。
损失值获取部件，被配置为根据所述至少一个训练样本病灶图像对应的所述分类概率、所述初始中心点坐标、所述初始长度、所述初始宽度、所述预测中心点坐标、所述预测长度和所述预测宽度，获取所述至少一个训练样本病灶图像在所述初始图像处理模型中的损失值。
目标模型生成部件，被配置为判断所述损失值是否处于预设范围内，若所述损失值不在预设范围内，则对所述初始图像处理模型的参数进行更新，将更新参数后的图像处理模型作为下一次训练的初始图像处理模型；继续对更新参数后的图像处理模型进行训练，直至所述损失值处于所述预设范围内，将最后一次训练的图像处理模型作为目标图像处理模型。
在一些实施例中，所述初始图像处理模型还包括神经网络、全局平均池化层、全连接层、特征加权求和层和上采样层；其中，所述神经网络包括至少一个层，每个层依次包括卷积层、激活函数和池化层。
所述图像处理模型生成装置还包括：与所述样本图像输入部件耦接的第一二维特征图获取部件，被配置为调用所述神经网络，将所述训练样本病灶图像依次通过所述神经网络的每个层的卷积层、激活函数和池化层，得到所述训练样本病灶图像的多组二维特征图。
所述图像处理模型生成装置还包括：与所述第一二维特征图获取部件耦接的第一全局平均池化部件，被配置为调用所述全局平均池化层，对所述多组二维特征图进行全局平均池化，得到对应所述多组二维特征图的多组一维特征图；
所述图像处理模型生成装置还包括：与所述第一全局平均池化部件耦接的第一全连接部件，所述第一全连接部件还与所述分类概率获取部件耦接，所述第一全连接部件被配置为调用所述全连接层，对所述多组一维特征图进行特征提取，得到所述训练样本病灶图像的至少一组一维特征图，并将所述至少一组一维特征图输入所述分类概率获取部件。
所述图像处理模型生成装置还包括：与所述第一二维特征图获取部件以及所述第一全局平均池化部件耦接的第一特征加权求和部件，被配置为调用所述特征加权求和层，根据所述多组一维特征图，对所述多组二维特征图中的每组二维特征图进行特征加权求和。
所述图像处理模型生成装置还包括：与所述第一特征加权求和部件耦接的第一上采样部件，所述第一上采样部件还与所述预测数据获取部件耦接，所述第一上采样部件被配置为调用所述上采样层，对经过所述特征加权求和的多组二维特征图进行上采样处理，并将经过所述上采样处理的多组二维特征图输入所述预测数据获取部件。
再一方面，提供一种图像处理装置，包括：待处理图像输入部件、疾病分类概率获取部件、病灶区域数据获取部件、疾病种类确定部件、以及病灶区域确定部件。
其中，待处理图像输入部件，被配置为将待处理病灶图像输入目标图像处理模型；其中，所述目标图像处理模型通过上面一些事实例所述的方法训练得到，所述目标图像处理模型包括分类层和标注层。
疾病分类概率获取部件，被配置为调用所述分类层，对所述待处理病灶图像进行分类处理，得到所述待处理病灶图像对应的疾病分类概率。
病灶区域数据获取部件，被配置为调用所述标注层，对所述待处理病灶图像进行处理，得到所述待处理病灶图像中所包含病灶区域的区域中心点坐标、区域长度和区域宽度。
疾病种类确定部件，被配置为根据所述疾病分类概率，确定所述待处理病灶图像对应的疾病种类。
病灶区域确定部件，被配置为根据所述区域中心点坐标、所述区域长度和所述区域宽度，确定所述待处理病灶图像中的病灶标注区域。
在一些实施例中，所述目标图像处理模型还包括神经网络、全局平均池化层、全连接层、特征加权求和层和上采样层；其中，所述神经网络包括至少一个层，每个层依次包括卷积层、激活函数和池化层。
所述图像处理装置还包括：与所述待处理图像输入部件耦接的第二二维特征图获取部件，被配置为调用所述神经网络，将所述待处理病灶图像依次通过所述神经网络的每个层的卷积层、激活函数和池化层，得到所述待处理病灶图像的多组二维特征图。
所述图像处理装置还包括：与所述第二二维特征图获取部件耦接的第二全局平均池化部件，被配置为调用所述全局平均池化层，对所述多组二维特征图进行全局平均池化，得到对应所述多组二维特征图的多组一维特征图。
所述图像处理装置还包括：与所述第二全局平均池化部件耦接的第二全连接部件，所述第二全连接部件还与所述疾病分类概率获取部件耦接，所述第二全连接部件被配置为调用所述全连接层，对所述多组一维特征图进行特 征提取，得到所述待处理病灶图像的至少一组一维特征图，并将所述至少一组一维特征图输入所述疾病分类概率获取部件。
所述图像处理装置还包括：与所述第二二维特征图获取部件以及所述第二全局平均池化部件耦接的第二特征加权求和部件，被配置为调用所述特征加权求和层，根据所述多组一维特征图，对所述多组二维特征图中的每组二维特征图进行特征加权求和。
所述图像处理装置还包括：与所述第二特征加权求和部件耦接的第二上采样部件，所述第二上采样部件还与所述病灶区域数据获取部件耦接，所述第二上采样部件被配置为调用所述上采样层，对经过所述特征加权求和的多组二维特征图进行上采样处理，并将经过所述上采样处理的多组二维特征图输入所述病灶区域数据获取部件。
又一方面，提供一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机程序指令，所述计算机程序指令在处理器上运行时，使得所述处理器执行如上述一些实施例所述的图像处理模型生成方法中的一个或多个步骤，和/或，如上述一些实施例所述的图像处理方法中的一个或多个步骤。
又一方面，提供一种计算机程序产品。所述计算机程序产品包括计算机程序指令，在计算机上执行所述计算机程序指令时，所述计算机程序指令使计算机执行如上述一些实施例所述的图像处理模型生成方法中的一个或多个步骤，和/或，如上述一些实施例所述的图像处理方法中的一个或多个步骤。
又一方面，提供一种计算机程序。当所述计算机程序在计算机上执行时，所述计算机程序使计算机执行如上述一些实施例所述的图像处理模型生成方法中的一个或多个步骤，和/或，如上述一些实施例所述的图像处理方法中的一个或多个步骤。
又一方面，提供一种电子设备，包括处理器、存储器、以及存储在所述存储器上并可在所述处理器上运行的计算机程序。所述处理器执行所述计算机程序时，实现如上述一些实施例所述的图像处理模型生成方法中的一个或多个步骤，和/或，如上述一些实施例所述的图像处理方法中的一个或多个步骤。
附图说明
为了更清楚地说明本公开中的技术方案，下面将对本公开一些实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本公开的一些实施例的附图，对于本领域普通技术人员来讲，还可以根据这些附图获得其他的附图。此外，以下描述中的附图可以视作 示意图，并非对本公开实施例所涉及的产品的实际尺寸、方法的实际流程、信号的实际时序等的限制。
图1为根据一些实施例的一种图像处理模型生成方法的流程图；
图2为根据一些实施例的训练样本病灶图像的示意图；
图3为根据一些实施例的初始图像处理模型的结构图；
图4为根据一些实施例的另一种图像处理模型生成方法的流程图；
图5～图8分别为根据一些实施例的图像处理模型生成方法中的S2、S3、S33和S4的流程图；
图9为根据一些实施例的目标图像处理模型的结构图；
图10为根据一些实施例的一种图像处理方法的流程图；
图11为根据一些实施例的待处理病灶图像的示意图；
图12为根据一些实施例的另一种图像处理方法的流程图；
图13～图15分别为根据一些实施例的图像处理方法中的S200、S300和S303的流程图；
图16为根据一些实施例的一种图像处理模型生成装置的结构图；
图17为根据一些实施例的另一种图像处理模型生成装置的结构图；
图18为根据一些实施例的一种图像处理装置的结构图；
图19为根据一些实施例的另一种图像处理装置的结构图。
具体实施方式
下面将结合附图，对本公开一些实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本公开一部分实施例，而不是全部的实施例。基于本公开所提供的实施例，本领域普通技术人员所获得的所有其他实施例，都属于本公开保护的范围。
除非上下文另有要求，否则，在整个说明书和权利要求书中，术语“包括(comprise)”及其其他形式例如第三人称单数形式“包括(comprises)”和现在分词形式“包括(comprising)”被解释为开放、包含的意思，即为“包含，但不限于”。在说明书的描述中，术语“一个实施例(one embodiment)”、“一些实施例(some embodiments)”、“示例性实施例(exemplary embodiments)”、“示例(example)”、“特定示例(specific example)”或“一些示例(some examples)”等旨在表明与该实施例或示例相关的特定特征、结构、材料或特性包括在本公开的至少一个实施例或示例中。上述术语的示意性表示不一定是指同一实施例或示例。此外，所述的特定特征、结构、材料或特点可以以任何适当方式包括在任何一个或多个实施例或示例中。
以下，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本公开实施例的描述中，除非另有说明，“多个”的含义是两个或两个以上。
在描述一些实施例时，可能使用了“耦接”和“连接”及其衍伸的表达。例如，描述一些实施例时可能使用了术语“连接”以表明两个或两个以上部件彼此间有直接物理接触或电接触。又如，描述一些实施例时可能使用了术语“耦接”以表明两个或两个以上部件有直接物理接触或电接触。然而，术语“耦接”或“通信耦合(communicatively coupled)”也可能指两个或两个以上部件彼此间并无直接接触，但仍彼此协作或相互作用。这里所公开的实施例并不必然限制于本文内容。
“A和/或B”，包括以下三种组合：仅A，仅B，及A和B的组合。
如本文中所使用，根据上下文，术语“如果”任选地被解释为意思是“当……时”或“在……时”或“响应于确定”或“响应于检测到”。类似地，根据上下文，短语“如果确定……”或“如果检测到[所陈述的条件或事件]”任选地被解释为是指“在确定……时”或“响应于确定……”或“在检测到[所陈述的条件或事件]时”或“响应于检测到[所陈述的条件或事件]”。
本文中“适用于”或“被配置为”的使用意味着开放和包容性的语言，其不排除适用于或被配置为执行额外任务或步骤的设备。
另外，“基于”的使用意味着开放和包容性，因为“基于”一个或多个所述条件或值的过程、步骤、计算或其他动作在实践中可以基于额外条件或超出所述的值。
相关技术中，基于深度学习的分类算法可以很好的对疾病种类进行诊断。但是，本公开的发明人在研究中发现，这些分类算法在医疗影像处理过程中，将图片判为某类疾病时的关注区域，和医学上的判别区域存在一定的差异。虽然可以利用某些算法在医疗影像上大致标出病灶位置，此种标注方式太过粗糙，不能体现病灶在病变组织上的具体位置。
基于此，本公开的一些实施例提供了一种图像处理模型生成方法，如图1所示，该方法包括如下S1～S7。
S1：将已知疾病种类的至少一个训练样本病灶图像输入初始图像处理模型1(如图2所示)。
如图2和图3所示，其中，初始图像处理模型1包括分类层1007(Classification)和标注层1010(Attention Location)。训练样本病灶图像1001 包括病灶区域对应的初始中心点B坐标(G x，G y)、初始长度G h和初始宽度G w。
在本公开实施例中，初始图像处理模型1是指还未进行训练或者正在训练过程中的图像处理模型。已知疾病种类是指训练样本病灶图像1001对应的疾病种类。示例性地，已知疾病种类可以为肺炎、心脏病、肝硬化等疾病，也可以是其他疾病种类，具体可以视具体需求而定。
初始图像处理模型1中包含有分类层1007和标注层1010，分类层1007和标注层1010都是可以对病灶图像进行处理的网络层。
在采用训练样本病灶图像1001对初始图像处理模型1进行训练时，可以获取已知疾病种类的至少一个训练样本病灶图像1001，将至少一个训练样本病灶图像1001输入初始图像处理模型1中，以对初始图像处理模型1进行训练。
输入至初始图像处理模型1的训练样本病灶图像1001的数量可以是一个也可以是多个，例如为500个、800个或1000个等，也可以视实际情况而定，本公开实施例对此不做具体限定。
训练样本病灶图像1001包括病灶区域，病灶区域是指病灶在训练样本病灶图像1001中的位置，其位置在训练样本病灶图像1001中是以一定区域范围进行表示的。
示例性地，病灶区域可以表示为一个矩形区域，该病灶区域可以是由医生等人员预先在训练样本病灶图像1001中标注的真实病灶区域。如图2所示，训练样本病灶图像1001中真实病灶区域为真实框TB(Truth box)所示区域，真实框TB是根据初始中心点B坐标(G x，G y)、初始长度G h和初始宽度G w而得到的。则初始中心点B坐标(G x，G y)即为真实病灶区域对应的中心点在训练样本病灶图像1001对应的图像坐标系中的坐标；初始长度G h和初始宽度G w即分别为真实病灶区域对应的长和宽。
上述示例仅是为了更好地理解本公开实施例的技术方案而列举的示例，不作为对本公开实施例的唯一限制。
在得到已知疾病种类的至少一个训练样本病灶图像1001之后，可以将训练样本病灶图像1001依次输入至初始图像处理模型1，以对初始图像处理模型1进行训练，执行S2。
S2：调用分类层1007，对训练样本病灶图像1001进行分类处理，得到所述训练样本病灶图像1001对应各已知疾病种类的分类概率。
分类层1007是可以用于对病灶图像所对应的疾病种类进行分类的网络 层，通过分类层1007可以获取病灶图像所对应的各疾病种类的分类概率，从而获取病灶图像所对应的疾病种类，如肺炎、心脏病等。
其中，训练样本病灶图像1001中可以只包括一种疾病的病灶，也可以为包括多种疾病的病灶，示例性地，训练样本病灶图像1001中包括三种疾病的病灶，分类层1002可以获取训练样本病灶图像1001所对应的三种疾病的分类概率，根据各疾病的分类概率，进而判断训练样本病灶图像1001中所包括的疾病种类。
S3：调用标注层1010，对训练样本病灶图像1001进行处理，得到训练样本病灶图像1001中所包含的病灶区域的预测中心点坐标、预测长度和预测宽度。
标注层1010是可以用于对病灶在病变组织上的具体位置进行标注的网络层，标注层1010可以在病灶图像上的病灶区域添加标注框，示例性地，标注框可以为矩形框，或者圆形框等。
如图2所示，训练样本病灶图像1001输入初始图像处理模型1，经过标注层1010的处理输出预测框PB(Pred box)，预测框PB是根据预测中心点A坐标(P x，P y)、预测长度P h和预测宽度P w而得到的。则预测中心点A坐标(P x，P y)即为预测病灶区域对应的中心点在训练样本病灶图像1001对应的图像坐标系中的坐标；预测长度P h和预测宽度P w即分别为预测病灶区域对应的长和宽。
需要说明的是，对上述S2和S3的执行次序并不设限，可以是同时执行S2和S3；也可以是先执行S2，再执行S3，还可以是先执行S3，再执行S2。
由上述S2和S3分别获得训练样本病灶图像1001的疾病种类的分类概率和病灶区域的预测数据，可执行S4。
S4：根据至少一个训练样本病灶图像1001对应的分类概率、初始中心点B坐标(G x，G y)、初始长度G h、初始宽度G w、预测中心点A坐标(P x，P y)、预测长度P h、预测宽度P w，获取至少一个训练样本病灶图像1001在初始图像处理模型1中的损失值，损失值记为L。
损失值L可以根据训练样本病灶图像1001对应的分类概率、初始中心点B坐标(G x，G y)、初始长度G h、初始宽度G w、预测中心点A坐标(P x，P y)、预测长度P h、预测宽度P w计算得到。
在一些实施例中，在一次训练过程中输入一个训练样本病灶图像1001，一个训练样本病灶图像1001中针对一种疾病可得到一个损失值L，若一个训练样本病灶图像1001中包括多种疾病，则根据多种疾病的权重分别与对应的 损失值加权，对一个训练样本病灶图像1001中多种疾病的损失值进行求平均，得到一个训练样本病灶图像1001的损失值L。
在一些实施例中，在一次训练过程中输入多个训练样本病灶图像1001，每个训练样本病灶图像1001的损失值L采用上述实施例的方法获得，对多个训练样本病灶图像1001的损失值L进行求平均，可得到多个训练样本病灶图像1001的损失值L。
由上述S4分别获得训练样本病灶图像的损失值，可执行S5，参见图3。
S5：判断损失值L是否处于预设范围内；若损失值L不在预设范围内，则执行S6；若损失值L在预设范围内，则执行S7。
S6：对初始图像处理模型1的参数进行更新，将更新参数后的图像处理模型作为下一次训练的初始图像处理模型1。
重复上述S1～S5，直至损失值L处于预设范围内，将最后一次训练的图像处理模型作为训练后的目标图像处理模型2(如图8所示)。
S7：将当前训练的图像处理模型(即最后一次训练的图像处理模型)作为训练后的目标图像处理模型2。
在上述S6和S7中，预设范围是指与损失值L进行比较的范围，是预先进行设定的一个数值范围。
若损失值L在预设范围内，则说明当前训练图像处理模型的训练结果已经达到了预期结果，可以将经过当前训练的图像处理模型(即最后一次训练的图像处理模型)作为训练后的目标图像处理模型2。
在损失值L不在预设范围内时，表示初始图像处理模型1训练的结果未达到预期，基于损失值L的结果，需要对初始图像处理模型1的参数进行更新，将更新参数后的图像处理模型作为下一次训练的初始图像处理模型1，继续输入已知疾病种类的至少一个训练样本病灶图像1001，对初始图像处理模型1进行训练。
本公开实施例提供的图像处理模型生成方法生成的图像处理模型能够准确的识别出病灶图像中的病灶区域，并添加相应的标注坐标，能够体现出病灶在病变组织上的具体位置，其中，通过分类层1007与标注层1010对图像处理模型的关注区域(即标注的病灶区域)进行了训练，使得标注的病灶区域位置更加精确，利用训练好的图像处理模型，能够自动识别医疗影像中所包含的疾病种类以及对各疾病种类的病灶区域位置进行标注，节省了人力提高效率，且生成的图像处理模型进行疾病种类的分类和标注具有较高的检测精度。
在一些实施例中，如图3所示，初始图像处理模型1还包括：神经网络1000(Neural Networks)，神经网络1000包括至少一个层，每个层依次包括卷积层1002(Convolutional Layer)、激活函数1003(Activation Function)和池化层1007(Pooling Layer)。
如图4所示，本公开实施例中图像处理模型生成方法，还包括：在S1之后执行S11。
S11：调用神经网络1000，将训练样本病灶图像1001依次通过神经网络1000的每个层的卷积层1002、激活函数1003和池化层1004，得到训练样本病灶1001图像的多组二维特征图。
其中，卷积层1002用于提取特征；激活函数1003用于对提取到的特征进行激活操作；池化层1004用于对激活后的特征进行下采样；通过多层叠加，可以得到训练样本病灶图像1001的多组二维特征图。
在一些实施例中，本公开上述实施例中神经网络1000可以为卷积神经网络(Convolutional Neural Networks，简称CNN)、循环神经网络(Recurrent Neural Networks，简称RNN)等。
在一些实施例中，本公开上述实施例中神经网络1000可以为长短期记忆网络(Long Short-Term Memory，简称LSTM)、人工神经网络(Artificial Neural Networks，简称ANNs)等。
在一些实施例中，如图3所示，初始图像处理模型1还包括：全局平均池化层1005(Global Average Pooling，简称GAP)。
如图4所示，本公开实施例中图像处理模型生成方法，还包括：在S11之后执行S11A。
S11A：调用全局平均池化层1005，对多组二维特征图进行全局平均池化，得到对应多组二维特征图的多组一维特征图。
将训练样本病灶图像1001通过神经网络1000得到的多组二维特征图输入全局平均池化层1005，调用全局平均池化层1005，对多组二维特征图进行全局平均池化，得到对应多组二维特征图的多组一维特征图。
在一些实施例中，如图3所示，初始图像处理模型1还包括：全连接层1006(Fully Connected Layers)。
如图4所示，本公开实施例中图像处理模型生成方法，还包括：在S11A之后执行S12。
S12：调用全连接层1006，对全局平均池化后得到的多组一维特征图进行特征提取，得到训练样本病灶图像1001的至少一组一维特征图。
将训练样本病灶图像1001通过全局平均池化层1005处理得到的多组一维特征图输入全连接层1006，调用全连接层1006，对经过全局平均池化后得到的多组一维特征图进行特征提取，得到训练样本病灶图像1001的至少一组一维特征图，并将至少一组一维特征图输入分类层1007。
在一些实施例中，如图5所示，S2调用分类层1007，对训练样本病灶图像1001进行分类处理，得到训练样本病灶图像1001对应各已知疾病种类的分类概率，包括：
S21：将训练样本病灶图像1001的至少一组一维特征图输入分类层1007。
S22：调用分类层1007，对一维特征图进行分类处理，输出训练样本病灶图像1001对应各已知疾病种类的分类概率。
在一些实施例中，如图3所示，初始图像处理模型1还包括：特征加权求和层1008(英文名称为：FWS)和上采样层1009(Upsampling Layers)。
如图4所示，本公开实施例中图像处理模型生成方法，还包括：在S11之后执行S11B。
S11B：调用特征加权求和层1008，根据多组一维特征图，对多组二维特征图中的每组二维特征图进行特征加权求和。
将训练样本病灶图像1001的多组二维特征图，以及经过全局平均池化后得到的多组一维特征图，输入特征加权求和层1008，调用特征加权求和层1008，根据多组一维特征图，对多组二维特征图中的每组二维特征图进行特征加权求和。
在S11B之后执行S13。
S13：调用上采样层1009，对经过特征加权求和的多组二维特征图进行上采样处理，得到训练样本病灶图像1001经过处理的多组二维特征图。
在一些实施例中，如图2和图6所示，S3调用标注层1010，对训练样本病灶图像1001进行处理，得到训练样本病灶图像1001中所包含的病灶区域的预测中心点A坐标(P x，P y)、预测长度P h和预测宽度P w，包括S31～S33。
S31：将训练样本病灶图像1001的经过处理的多组二维特征图输入标注层1010；
S32：调用标注层1010，确定二维特征图中的最大特征点，及最大特征点对应的最大特征值和二维坐标。
S33：根据最大特征点、最大特征值、二维坐标和预设特征阈值，确定训练样本病灶图像1001中所包含的病灶区域对应的预测中心点A坐标(P x，P y)、预测长度P h和预测宽度P w。
其中，预设特征阈值可以根据所希望得到的病灶区域的预测长度P h和预测宽度P w的范围大小而调整，一般情况下，当预设特征阈值越大，则所得到的病灶区域的预测长度P h和预测宽度P w越大，反之亦然。
示例性的，如图7所示，S33根据最大特征点、最大特征值、二维坐标和预设特征阈值，确定训练样本病灶图像1001中所包含的病灶区域对应的预测中心点A坐标(P x，P y)、预测长度P h和预测宽度P w，包括S331～S335。
请参见图2，S331：将最大特征点的二维坐标确定为预测中心点A坐标(P x，P y)。
S332：计算得到最大特征值和预设特征阈值之间的特征差值绝对值。
S333：根据特征差值绝对值和最大特征点的二维坐标，获取二维特征图中在横轴X方向上的第一特征点A1和第二特征点A2，及在纵轴Y方向上的第三特征点A3和第四特征点A4。
S334：获取第一特征点A1在横轴X方向上的第一坐标值P x-t1，及第二特征点A2在横轴X方向上的第二坐标值P x+t2，其中t1、t2均不小于零。获取第三特征点A3在纵轴Y方向上的第三坐标值P y-t3，及第四特征点A4在纵轴Y方向上的第四坐标值P y+t4，其中t3、t4均不小于零。
S335：基于第一坐标值P x+t1和第二坐标值P x+t2，计算得到预测宽度P w＝t2-t1；基于第三坐标值P y+t3和第四坐标值P y+t4，计算得到预测长度P h＝t4–t3。
在一些实施例中，如图8所示，S4根据至少一个训练样本病灶图像1001对应的分类概率、初始中心点B坐标(G x，G y)、初始长度G h、初始宽度G w、预测中心点A坐标(P x，P y)、预测长度P h和预测宽度P w，获取至少一个训练样本病灶图像1001在初始图像处理模型1中的损失值L，包括S41～S43。
损失值L包括分类损失值和位置损失值。
S41：根据训练样本病灶图像1001对应的分类概率，计算得到分类损失值。
分类损失值是指初始图像处理模型1对训练样本病灶图像1001进行分类，得到分类概率存在的分类损失，分类损失值记为L cls。
在得到训练样本病灶图像1001对应的分类概率之后，可以根据各分类概率，可以得到分类损失值L cls，在本公开中，可以采用softmax、SVM(英文全称为：Support Vector Machine，中文名称为：支持向量机)或sigmoid的方式计算得到分类损失值L cls。
S42：根据训练样本病灶图像1001对应的初始中心点B坐标(G x，G y)、 初始长度G h、初始宽度G w、预测中心点A坐标(P x，P y)、预测长度P h和预测宽度P w，计算得到位置损失值。
位置损失值是指初始图像处理模型1对训练样本病灶图像1001进行处理，确定病灶区域对应的预测中心点A坐标(P x，P y)、预测长度P h和以及预测宽度P w与初始中心点B坐标(G x，G y)、初始长度G h和初始宽度G w存在的位置损失，位置损失值记为L al。
位置损失值L al可以根据初始中心点B坐标(G x，G y)、初始长度G h、初始宽度G w、预测中心点A坐标(P x，P y)、预测长度P h、以及预测宽度P w，进行计算得到。
示例性地，位置损失值L al可以按照下述公式(1)计算得到：
Figure PCTCN2020109611-appb-000001
其中，
Figure PCTCN2020109611-appb-000002
在一些其它的实施例中，位置损失值L al的计算方法还可以采用L1范数或L2范数等。
S43：根据分类损失值和位置损失值，获取训练样本病灶图像1001在初始图像处理模型1中的损失值。
通过上述步骤计算得到分类损失值L cls和位置损失值L al之后，可以基于分类损失值L cls和位置损失值L al计算得到损失值L，分类损失值L cls和位置损失值L al的和值即为损失值L，即损失值L＝L cls+L al。
本公开的一些实施例提供了一种图像处理方法，如图10所示，该方法包括S100～S500。
S100：将待处理病灶图像2001输入目标图像处理模型2(如图9所示)。
如图9所示，在本公开实施例中，目标图像处理模型2是通过上面一些实施例图像处理模型生成方法训练得到，其可以用于对医疗领域的病灶图像进行处理，确定病灶图像对应的病症及确定病灶在病灶图像中所处的区域的模型。
待处理病灶图像2001是指医疗领域的图像，如采用医疗仪器拍摄的图像等。
在目标图像处理模型2中包含有分类层2007和标注层2010，其中，分类层2007可以用于对待处理病灶图像2001进行分类处理，得到待处理病灶图 像2001对应的分类概率，以确定待处理病灶图像2001对应的病症；标注层2010可以用于确定出待处理病灶图像2001中的病灶在病灶图像中所处的区域。对于此过程将在下述步骤中进行详细描述。
在得到待处理病灶图像2001之后，可以将待处理病灶图像2001输入至目标图像处理模型2中，并执行S200。
S200：调用分类层2007，对待处理病灶图像2001进行分类处理，得到待处理病灶图像2001对应的疾病分类概率。
分类层2007是可以用于对病灶图像所对应的疾病种类进行分类的网络层，通过分类层2007可以获取病灶图像所对应的各疾病种类的分类概率，从而获取病灶图像所对应的疾病种类，如肺炎、心脏病等。
S300：调用标注层2010，对待处理病灶图像2001进行处理，得到待处理病灶图像2001中所包含病灶区域的区域中心点坐标、区域长度和区域宽度。
标注层2010是可以用于对病灶在病变组织上的具体位置进行标注的网络层，标注层2010可以在病灶图像上的病灶区域添加标注框，示例性地，标注框可以为矩形框，或者圆形框等。
如图11所示，待处理病灶图像2001输入目标图像处理模型2，经过标注层2010的处理输出病灶框Z，病灶框Z是根据区域中心点B坐标(D x，D y)、区域长度D h和区域宽度D w而得到的。则区域中心点B坐标(D x，D y)即为病灶区域对应的中心点在待处理病灶图像2001对应的图像坐标系中的坐标；区域长度D h和区域宽度D w即分别为病灶区域对应的长和宽。
对上述S200和S300的执行次序并不设限，可以是同时执行S200和S300；也可以是先执行S200，再执行S300，还可以是先执行S300，再执行S200。
由上述S200和S300分别获得待处理病灶图像2001的疾病种类的分类概率和病灶区域的区域数据，可执行S400。
S400：根据疾病分类概率，确定待处理病灶图像2001对应的疾病种类。
疾病分类概率是由S200获得的，如果待处理病灶图像2001中针对特定疾病种类的疾病分类概率处于一定范围内，则说明该待处理病灶图像2001中包含该特定疾病种类。如果待处理病灶图像2001中针对特定疾病种类的疾病分类概率不处于一定范围内，则说明该待处理病灶图像2001中不包含该特定疾病种类。
S500：根据区域中心点D坐标(D x，D y)、区域长度D h和区域宽度D w，确定待处理病灶图像2001中的病灶标注区域。
在S300获得的区域中心点B坐标(D x，D y)即为病灶区域对应的中心点 在待处理病灶图像2001对应的图像坐标系中的坐标，区域长度D h和区域宽度D w即分别为病灶区域对应的长和宽，在待处理病灶图像2001中标注出病灶标注区域，如图11所示的病灶框Z所示。
本公开实施例所提供的图像处理方法利用训练好的图像处理模型，能够自动识别医疗影像中所包含的疾病种类以及对各疾病种类的病灶区域位置进行标注，节省了人力提高效率，且生成的图像处理模型进行疾病种类的分类和标注具有较高的检测精度。
在一些实施例中，如图9所示，目标图像处理模型2还包括：神经网络2000，神经网络2000包括至少一个层，每个层依次包括卷积层2002、激活函数2003和池化层2004。
如图12所示，本公开实施例中图像处理方法，还包括：在S100之后执行S101。
S101：调用神经网络2000，将待处理病灶图像2001依次通过神经网络2000的每个层的卷积层2002、激活函数2003和池化层2004，得到待处理病灶图像2001的多组二维特征图。
其中，卷积层2002用于提取特征；激活函数2003用于对提取到的特征进行激活操作；池化层2004用于对激活后的特征进行下采样；通过多层叠加，可以得到待处理病灶图像2001的多组二维特征图。
在一些实施例中，本公开上述实施例中神经网络2000可以为卷积神经网络、循环神经网络等。
在一些实施例中，本公开上述实施例中神经网络2000可以为长短期记忆网络(Long Short-Term Memory，简称LSTM)、人工神经网络(Artificial Neural Networks，简称ANNs)等。
在一些实施例中，如图9所示，目标图像处理模型2还包括：全局平均池化层。
如图12所示，本公开实施例中图像处理方法，还包括：在S101之后执行S101A。
S101A：调用全局平均池化层2005，对多组二维特征图进行全局平均池化，得到对应多组二维特征图的多组一维特征图。
将待处理病灶图像2001通过神经网络2000得到的多组二维特征图输入全局平均池化层2005，调用全局平均池化层5005，对多组二维特征图进行全局平均池化，得到对应多组二维特征图的多组一维特征图。
在一些实施例中，如图9所示，目标图像处理模型2还包括：全连接层 2006。
如图12所示，本公开实施例中图像处理方法，还包括：在S101A之后执行S102。
S102：调用全连接层2006，对全局平均池化后得到的多组一维特征图进行特征提取，得到待处理病灶图像2001的至少一组一维特征图。
将待处理病灶图像2001通过全局平均池化层2005处理得到的多组一维特征图输入全连接层2006，调用全连接层2006，对经过全局平均池化后得到的多组一维特征图进行特征提取，得到待处理病灶图像2001的至少一组一维特征图，并将至少一组一维特征图输入分类层2007。
在一些实施例中，如图13所示，S200调用分类层2007，对待处理病灶图像2001进行分类处理，得到待处理病灶图像2001对应的疾病分类概率，包括：
S201：将所待处理病灶图像2001的至少一组一维特征图输入分类层2007。
S202：调用分类层2007，对一维特征图进行分类处理，输出待处理病灶图像2001对应的疾病分类概率。
在一些实施例中，如图9所示，目标图像处理模型2还包括：特征加权求和层2007和上采样层2009。
如图12所示，本公开实施例中图像处理方法，还包括：在S101之后执行S101B。
S101B：调用特征加权求和层2008，根据多组一维特征图，对多组二维特征图中的每组二维特征图进行特征加权求和。
将待处理病灶图像2001的多组二维特征图，以及经过全局平均池化后得到的多组一维特征图，输入特征加权求和层2008，调用特征加权求和层2008，根据多组一维特征图，对多组二维特征图中的每组二维特征图进行特征加权求和。
在S101B之后执行S103。
S103：调用上采样层2009，对经过特征加权求和的多组二维特征图进行上采样处理，得到待处理病灶图像2001经过处理的多组二维特征图，并将经过处理的多组二维特征图输入标注层2010。
在一些实施例中，如图14所示，S300调用标注层2010，对待处理病灶图像2001进行处理，得到待处理病灶图像2001中所包含病灶区域的区域中心点D坐标(D x，D y)、区域长度D h和区域宽度D w，包括S301～S303。
S301：将待处理病灶图像2001的经过处理的多组二维特征图输入标注层2010。
S302：调用标注层2010，确定二维特征图中的最大特征点，及最大特征点对应的最大特征值和二维坐标。
S303：根据最大特征点、最大特征值、二维坐标和预设特征阈值，确定待处理病灶图像2001中所包含的病灶区域对应的区域中心点D坐标(D x，D y)、区域长度D h和区域宽度D w。
其中，预设特征阈值可以根据所希望得到的病灶区域的区域长度D h和区域宽度D w的范围大小而调整，一般情况下，当预设特征阈值越大，则所得到的病灶区域的区域长度D h和区域宽度D w越大，反之亦然。
示例性的，如图11和图15所示，S303根据最大特征点、最大特征值、二维坐标和预设特征阈值，确定待处理病灶图像2001中所包含的病灶区域对应的区域中心点D坐标(D x，D y)、区域长度D h和区域宽度D w，包括S3031～S3035。
如图15所示，S3031：将最大特征点的二维坐标作为区域中心点D坐标(D x，D y)。
S3032：计算得到最大特征值和预设特征阈值之间的特征差值绝对值。
S3033：根据特征差值绝对值和最大特征点的二维坐标，获取二维特征图中在横轴X方向上的第一特征点D1和第二特征点D2，及在纵轴Y方向上的第三特征点D3和第四特征点D4。
S3034：获取第一特征点D1在横轴X方向上的第一坐标值D x-t11，及第二特征点D2在横轴X方向上的第二坐标值D x+t22，其中t11、t22均不小于零。获取第三特征点D3在纵轴Y方向上的第三坐标值D y-t33，及第四特征点D4在纵轴Y方向上的第四坐标值D y+t44，其中t33、t44均不小于零。
S3035：基于第一坐标值D x+t11和第二坐标值D x+t22，计算得到区域宽度D w＝t22-t11；基于第三坐标值D y+t33和第四坐标值D y+t44，计算得到区域长度D h＝t4 4–t33。
如图16所示，本公开的一些实施例提供一种图像处理模型生成装置3，该图像处理模型生成装置3包括：
请再次参见图2、图3和图9，样本图像输入部件10，被配置为将已知疾病种类的至少一个训练样本病灶图像1001输入初始图像处理模型1。其中，初始图像处理模型1包括分类层1007和标注层1010，训练样本病灶图像1001包括病灶区域对应的初始中心点B坐标(G x，G y)、初始长度G h和初始宽度 G w。
分类概率获取部件20，与样本图像输入部件10耦接，被配置为调用分类层1007，对训练样本病灶图像1001进行分类处理，得到训练样本病灶图像1001对应各已知疾病种类的分类概率。
预测数据获取部件30，与样本图像输入部件10耦接，被配置为调用标注层1010，对训练样本病灶图像1001进行处理，得到训练样本病灶图像1001中所包含病灶区域的预测中心点A坐标(P x，P y)、预测长度P h和预测宽度P w。
损失值获取部件40，与分类概率获取部件20以及预测数据获取部件30耦接，被配置为根据至少一个训练样本病灶图像1001对应的分类概率、初始中心点B坐标(G x，G y)、初始长度G h、初始宽度G w、预测中心点A坐标、(P x，P y)预测长度P h和预测宽度P w，获取至少一个训练样本病灶图像1001在初始图像处理模型1中的损失值L。
目标模型生成部件50，与损失值获取部件40耦接，被配置为判断损失值L是否处于预设范围内，若损失值L不在预设范围内，则对初始图像处理模型1的参数进行更新，将更新参数后的图像处理模型作为下一次训练的初始图像处理模型1；继续对更新参数后的图像处理模型进行训练，直至损失值L处于预设范围内，将最后一次训练的图像处理模型作为目标图像处理模型2。
在一些实施例中，如图3所示，初始图像处理模型1还包括神经网络1000、全局平均池化层1005、全连接层1006、特征加权求和层1008和上采样层1009；其中，神经网络1000包括至少一个层，每个层依次包括卷积层1002、激活函数1003和池化层1004。
基于此，如图17所示，本公开实施例中图像处理模型生成装置3还包括：
第一二维特征图获取部件11，与样本图像输入部件10耦接，被配置为调用神经网络1000，将训练样本病灶图像1001依次通过神经网络1000的每个层的卷积层1002、激活函数1003和池化层1004，得到训练样本病灶图像1001的多组二维特征图。
第一全局平均池化部件11A，与第一二维特征图获取部件11耦接，被配置为调用全局平均池化层1005，对多组二维特征图进行全局平均池化，得到对应多组二维特征图的多组一维特征图。
第一全连接部件12，与第一全局平均池化部件11A耦接，第一全连接部件12还与分类概率获取部件20耦接，第一全连接部件12被配置为调用全连接层1006，对多组一维特征图进行特征提取，得到训练样本病灶图像1001的 至少一组一维特征图，并将至少一组一维特征图输入分类概率获取部件20。
第一特征加权求和部件11B，与第一二维特征图获取部件11以及第一全局平均池化部件11A耦接，被配置为调用特征加权求和层1008，根据多组一维特征图，对多组二维特征图中的每组二维特征图进行特征加权求和。
第一上采样部件13，与第一特征加权求和部件11B耦接，第一上采样部件13还与预测数据获取部件30耦接，第一上采样部件13被配置为调用上采样层1009，对经过特征加权求和的多组二维特征图进行上采样处理，并将经过上采样处理的多组二维特征图输入预测数据获取部件30。
本公开上述实施例提供的图像处理模型生成装置3中所包括的各部件的功能可参见前述实施例中所述的图像处理模型生成方法中的相应步骤的描述。
本公开的上述实施例中提供的图像处理模型生成装置3可以生成目标图像处理模型2，在应用于医疗影像的识别过程中，能够使得目标图像处理模型2能够对医疗影像的疾病种类和病灶位置自动进行检测，节省了人力提高了疾病诊断的效率，且通过目标图像处理模型2进行医疗影像的疾病种类的判定和病灶位置的标注都更加精确。
如图18所示，本公开的一些实施例还提供一种图像处理装置4，该图像处理装置4包括：
待处理图像输入部件100，被配置为将待处理病灶图像2001输入目标图像处理模型2；其中，目标图像处理模型2为本公开上面一些实施例中图像处理模型生成方法训练得到，目标图像处理模型2包括分类层2007和标注层2010(如图9所示)。
疾病分类概率获取部件200，与待处理图像输入部件100耦接，被配置为调用分类层2007，对待处理病灶图像2001进行分类处理，得到待处理病灶图像2001对应的疾病分类概率。
病灶区域数据获取部件300，与待处理图像输入部件100耦接，被配置为调用标注层2010，对待处理病灶图像2001进行处理，得到待处理病灶图像2001中所包含病灶区域的区域中心点D坐标(D x，D y)、区域长度D h和区域宽度D w(如图11所示)。
疾病种类确定部件400，与疾病分类概率获取部件200耦接，被配置为根据疾病分类概率，确定待处理病灶图像2001对应的疾病种类。
病灶区域确定部件500，与病灶区域数据获取部件300耦接，被配置为根据区域中心点D坐标(D x，D y)、区域长度D h和区域宽度D w，确定待处理 病灶图像2001中的病灶标注区域。
在一些实施例中，如图9所示，目标图像处理模型2还包括神经网络2000、全局平均池化层2005、全连接层2006、特征加权求和层2008和上采样层2009；其中，神经网络2000包括至少一个层，每个层依次包括卷积层2002、激活函数2003和池化层2004。
如图19所示，本公开实施例中图像处理装置4还包括：
第二二维特征图获取部件101，与待处理图像输入部件100耦接，被配置为调用神经网络2000，将待处理病灶图像2001依次通过神经网络2000的每个层的卷积层2002、激活函数2003和池化层2004，得到待处理病灶图像2001的多组二维特征图。
第二全局平均池化部件101A，与第二二维特征图获取部件101耦接，被配置为调用全局平均池化层2005，对多组二维特征图进行全局平均池化，得到对应多组二维特征图的多组一维特征图。
第二全连接部件102，与所第二全局平均池化部件101A和疾病分类概率获取部件200耦接，被配置为调用全连接层2006，对多组一维特征图进行特征提取，得到待处理病灶图像2001的至少一组一维特征图，并将至少一组一维特征图输入疾病分类概率获取部件200。
第二特征加权求和部件101B，与第二二维特征图获取部件101以及第二全局平均池化部件101A耦接，被配置为调用特征加权求和层2008，根据多组一维特征图，对多组二维特征图中的每组二维特征图进行特征加权求和。
第二上采样部件103，与第二特征加权求和部件101B以及病灶区域数据获取部件300耦接，被配置为调用上采样层2009，对经过特征加权求和的多组二维特征图进行上采样处理，并将经过上采样处理的多组二维特征图输入病灶区域数据获取部件300。
本公开的上述实施例中提供的图像处理装置4，将目标图像处理模型2应用于医疗影像的识别，能够使得目标图像处理模型2能够对医疗影像的疾病种类和病灶位置自动进行检测，节省了人力提高了疾病诊断的效率，且通过目标图像处理模型2进行医疗影像的疾病种类的判定和病灶位置的标注都更加精确。
本公开的一些实施例提供了一种计算机可读存储介质(例如，非暂态计算机可读存储介质)，该计算机可读存储介质中存储有计算机程序指令，计算机程序指令在处理器上运行时，使得处理器执行如上述实施例中一些实施例所述的图像处理模型生成方法中的一个或多个步骤，和/或，如上述实施例 中一些实施例所述的图像处理方法中的一个或多个步骤。
示例性的，上述计算机可读存储介质可以包括，但不限于：磁存储器件(例如，硬盘、软盘或磁带等)，光盘(例如，CD(Compact Disk，压缩盘)、DVD(Digital Versatile Disk，数字通用盘)等)，智能卡和闪存器件(例如，EPROM(Erasable Programmable Read-Only Memory，可擦写可编程只读存储器)、卡、棒或钥匙驱动器等)。
本公开描述的各种计算机可读存储介质可代表用于存储信息的一个或多个设备和/或其它机器可读存储介质。术语“机器可读存储介质”可包括但不限于，无线信道和能够存储、包含和/或承载指令和/或数据的各种其它介质。
本公开的一些实施例还提供了一种计算机程序产品。该计算机程序产品包括计算机程序指令，在计算机上执行该计算机程序指令时，该计算机程序指令使计算机执行如上述实施例中一些实施例所述的图像处理模型生成方法中的一个或多个步骤，和/或，如上述实施例中一些实施例所述的图像处理方法中的一个或多个步骤。
本公开的一些实施例还提供了一种计算机程序。当该计算机程序在计算机上执行时，该计算机程序使计算机执行如上述实施例中一些实施例所述的图像处理模型生成方法中的一个或多个步骤，和/或，如上述实施例中一些实施例所述的图像处理方法中的一个或多个步骤。
上述计算机可读存储介质、计算机程序产品及计算机程序的有益效果和上述一些实施例所述的图像处理模型生成方法以及图像处理方法的有益效果相同，此处不再赘述。
本公开一些实施例还提供了一种电子设备，包括：处理器、存储器以及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时实现如上述一些实施例所述的图像处理模型生成方法，和/或，如上述一些实施例所述的图像处理方法。
处理器用于支持上述图像处理模型生成装置3，和/或，图像处理装置4执行如上述实施例中一些实施例所述的图像处理模型生成方法中的一个或多个步骤，和/或，如上述实施例中一些实施例所述的图像处理方法中的一个或多个步骤，和/或用于本文所描述的技术的其它过程。
处理器可以是中央处理单元(Central Processing Unit，简称CPU)，还可以是其他通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现场可编程门阵列(FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。其中，通用处理器可以是微处理器或者该处理器 也可以是任何常规的处理器等。
存储器用于存储本公开实施例提供的上述图像处理模型生成装置3的程序代码和数据，和/或，图像处理装置4的程序代码和数据。处理器可以通过运行或执行存储在存储器内的软件程序，以及调用存储在存储器内的数据，执行图像处理模型生成装置3的各种功能，和/或，图像处理装置4的各种功能。
存储器可以是只读存储器(Read-Only Memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(Random Access Memory，RAM)或者可存储信息和指令的其他类型的动态存储设备，也可以是电可擦可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)、只读光盘(Compact Disc Read-Only Memory，CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质，但不限于此。存储器可以是独立存在，通过通信总线与处理器相连接。存储器也可以和处理器集成在一起。
本说明书中的各个实施例均采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似的部分互相参见即可。
对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本公开并不受所描述的动作顺序的限制，因为依据本公开，某些步骤可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本公开所必须的。
本说明书中的各个实施例均采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似的部分互相参见即可。以上所述，仅为本公开的具体实施方式，但本公开的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，想到变化或替换，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应以所述权利要求的保护范围为准。

Claims (19)

  1. 一种图像处理模型生成方法，包括：
    将已知疾病种类的至少一个训练样本病灶图像输入初始图像处理模型；其中，所述初始图像处理模型包括分类层和标注层，所述训练样本病灶图像包括病灶区域对应的初始中心点坐标、初始长度和初始宽度；
    调用所述分类层，对所述训练样本病灶图像进行分类处理，得到所述训练样本病灶图像对应各已知疾病种类的分类概率；
    调用所述标注层，对所述训练样本病灶图像进行处理，得到所述训练样本病灶图像中所包含的病灶区域的预测中心点坐标、预测长度和预测宽度；
    根据所述至少一个训练样本病灶图像对应的所述分类概率、所述初始中心点坐标、所述初始长度、所述初始宽度、所述预测中心点坐标、所述预测长度和所述预测宽度，获取所述至少一个训练样本病灶图像在所述初始图像处理模型中的损失值；
    判断所述损失值是否处于预设范围内，若所述损失值不在预设范围内，则对所述初始图像处理模型的参数进行更新，将更新参数后的图像处理模型作为下一次训练的初始图像处理模型；
    重复上述步骤，直至所述损失值处于所述预设范围内，将最后一次训练的图像处理模型作为训练后的目标图像处理模型。
  2. 根据权利要求1所述的方法，其中，所述调用所述分类层，对所述训练样本病灶图像进行分类处理，得到所述训练样本病灶图像对应各已知疾病种类的分类概率，包括：
    将所述训练样本病灶图像的至少一组一维特征图输入所述分类层；
    调用所述分类层，对所述一维特征图进行分类处理，输出所述训练样本病灶图像对应各已知疾病种类的分类概率。
  3. 根据权利要求1或2所述的方法，其中，所述调用所述标注层，对所述训练样本病灶图像进行处理，得到所述训练样本病灶图像中所包含的病灶区域的预测中心点坐标、预测长度和预测宽度，包括：
    将所述训练样本病灶图像的经过处理的多组二维特征图输入所述标注层；
    调用所述标注层，确定所述二维特征图中的最大特征点，及所述最大特征点对应的最大特征值和二维坐标；
    根据所述最大特征点、所述最大特征值、所述二维坐标和预设特征阈值，确定所述训练样本病灶图像中所包含的病灶区域对应的预测中心点坐标、预测长度和预测宽度。
  4. 根据权利要求3所述的方法，其中，所述二维坐标包括横轴方向上的第一坐标值和纵轴方向上的第二坐标值；
    所述根据所述最大特征点、所述最大特征值、所述二维坐标和预设特征阈值，确定所述训练样本病灶图像中所包含的病灶区域对应的预测中心点坐标、预测长度和预测宽度，包括：
    将所述二维坐标确定为所述预测中心点坐标；
    计算得到所述最大特征值和所述预设特征阈值之间的特征差值绝对值；
    根据所述特征差值绝对值和所述最大特征点的二维坐标，获取所述二维特征图中在所述横轴方向上的第一特征点和第二特征点，及在所述纵轴方向上的第三特征点和第四特征点；
    获取所述第一特征点在所述横轴方向上的第一坐标值，及所述第二特征点在所述横轴方向上的第二坐标值；
    获取所述第三特征点在所述纵轴方向上的第三坐标值，及所述第四特征点在所述纵轴方向上的第四坐标值；
    基于所述第一坐标值和所述第二坐标值，计算得到所述预测宽度；
    基于所述第三坐标值和所述第四坐标值，计算得到所述预测长度。
  5. 根据权利要求2～4中任一项所述的方法，其中，所述初始图像处理模型还包括：全局平均池化层；
    在所述调用所述分类层和在所述调用所述标注层之前，还包括：
    将所述训练样本病灶图像的多组二维特征图输入所述全局平均池化层；
    调用所述全局平均池化层，对所述多组二维特征图进行全局平均池化，得到对应所述多组二维特征图的多组一维特征图。
  6. 根据权利要求5所述的方法，其中，所述初始图像处理模型还包括：全连接层；
    在所述调用所述全局平均池化层之后，以及在所述调用所述分类层之前，还包括：
    调用所述全连接层，对经过所述全局平均池化后得到的所述多组一维特征图进行特征提取，得到所述训练样本病灶图像的至少一组一维特征图，并将所述至少一组一维特征图输入所述分类层。
  7. 根据权利要求5所述的方法，其中，所述初始图像处理模型还包括：特征加权求和层和上采样层；
    在所述调用所述全局平均池化层之后，以及在所述调用所述标注层之前，还包括：
    将所述训练样本病灶图像的多组二维特征图，以及经过所述全局平均池化后得到的所述多组一维特征图，输入所述特征加权求和层；
    调用所述特征加权求和层，根据所述多组一维特征图，对所述多组二维特征图中的每组二维特征图进行特征加权求和；
    调用所述上采样层，对经过所述特征加权求和的多组二维特征图进行上采样处理，得到所述训练样本病灶图像经过处理的多组二维特征图，并将所述经过处理的多组二维特征图输入所述标注层。
  8. 根据权利要求5～7中任一项所述的方法，其中，所述初始图像处理模型还包括：神经网络；所述神经网络包括至少一个层，每个层依次包括卷积层、激活函数和池化层；
    在所述将所述训练样本病灶图像的多组二维特征图输入所述全局平均池化层之前，还包括：
    将所述训练样本病灶图像输入所述神经网络；
    调用所述神经网络，将所述训练样本病灶图像依次通过所述神经网络的每个层的卷积层、激活函数和池化层，得到所述训练样本病灶图像的多组二维特征图，并将所述多组二维特征图输入所述全局平均池化层；
    在所述初始图像处理模型还包括特征加权求和层的情况下，所述多组二维特征图还被输入至所述特征加权求和层。
  9. 根据权利要求1～8中任一项所述的方法，其中，所述根据所述至少一个训练样本病灶图像对应的所述分类概率、所述初始中心点坐标、所述初始长度、所述初始宽度、所述预测中心点坐标、所述预测长度和所述预测宽度，获取所述至少一个训练样本病灶图像在所述初始图像处理模型中的损失值，包括：
    根据所述训练样本病灶图像对应的所述分类概率，计算得到分类损失值；
    根据所述训练样本病灶图像对应的所述初始中心点坐标、所述初始长度、所述初始宽度、所述预测中心点坐标、所述预测长度和所述预测宽度，计算得到位置损失值；
    根据所述分类损失值和所述位置损失值，获取所述训练样本病灶图像在所述初始图像处理模型中的损失值。
  10. 一种图像处理方法，包括：
    将待处理病灶图像输入目标图像处理模型；所述目标图像处理模型通过如权利要求1～9中任一项所述的方法训练得到，所述目标图像处理模型包括分类层和标注层；
    调用所述分类层，对所述待处理病灶图像进行分类处理，得到所述待处理病灶图像对应的疾病分类概率；
    调用所述标注层，对所述待处理病灶图像进行处理，得到所述待处理病灶图像中所包含病灶区域的区域中心点坐标、区域长度和区域宽度；
    根据所述疾病分类概率，确定所述待处理病灶图像对应的疾病种类；
    根据所述区域中心点坐标、所述区域长度和所述区域宽度，确定所述待处理病灶图像中的病灶标注区域。
  11. 根据权利要求10所述的方法，其中，所述调用所述分类层，对所述待处理病灶图像进行分类处理，得到所述待处理病灶图像对应的疾病分类概率，包括：
    将所述待处理病灶图像的至少一组一维特征图输入所述分类层；
    调用所述分类层，对所述一维特征图进行分类处理，输出所述待处理病灶图像对应的疾病分类概率。
  12. 根据权利要求10或11所述的方法，其中，所述调用所述标注层，对所述待处理病灶图像进行处理，得到所述待处理病灶图像中所包含的病灶区域对应的区域中心点坐标、区域长度和区域宽度，包括：
    将所述待处理病灶图像的经过处理的多组二维特征图输入所述标注层；
    调用所述标注层，确定所述二维特征图中的最大特征点，及所述最大特征点对应的最大特征值和二维坐标；
    根据所述最大特征点、所述最大特征值、所述二维坐标和预设特征阈值，确定所述待处理病灶图像中所包含的病灶区域对应的区域中心点坐标、区域长度和区域宽度。
  13. 根据权利要求12所述的方法，其中，所述二维坐标包括横轴方向上的第一坐标值和纵轴方向上的第二坐标值；
    所述根据所述最大特征点、所述最大特征值、所述二维坐标和预设特征阈值，确定所述待处理病灶图像中所包含的病灶区域的区域中心点坐标、区域长度和区域宽度，包括：
    将所述二维坐标确定为所述区域中心点坐标；
    计算得到所述最大特征值和所述预设特征阈值之间的特征差值绝对值；
    根据所述特征差值绝对值和所述最大特征点的二维坐标，获取所述二维特征图中在所述横轴方向上的第一特征点和第二特征点，及在所述纵轴方向上的第三特征点和第四特征点；
    获取所述第一特征点在所述横轴方向上的第一坐标值，及所述第二特征 点在所述横轴方向上的第二坐标值；
    获取所述第三特征点在所述纵轴方向上的第三坐标值，及所述第四特征点在所述纵轴方向上的第四坐标值；
    基于所述第一坐标值和所述第二坐标值，计算得到所述区域宽度；
    基于所述第三坐标值和所述第四坐标值，计算得到所述区域长度。
  14. 一种图像处理模型生成装置，包括：
    样本图像输入部件，被配置为将已知疾病种类的至少一个训练样本病灶图像输入初始图像处理模型；其中，所述初始图像处理模型包括分类层和标注层，所述训练样本病灶图像包括病灶区域对应的初始中心点坐标、初始长度和初始宽度；
    分类概率获取部件，被配置为调用所述分类层，对所述训练样本病灶图像进行分类处理，得到所述训练样本病灶图像对应各已知疾病种类的分类概率；
    预测数据获取部件，被配置为调用所述标注层，对所述训练样本病灶图像进行处理，得到所述训练样本病灶图像中所包含病灶区域的预测中心点坐标、预测长度和预测宽度；
    损失值获取部件，被配置为根据所述至少一个训练样本病灶图像对应的所述分类概率、所述初始中心点坐标、所述初始长度、所述初始宽度、所述预测中心点坐标、所述预测长度和所述预测宽度，获取所述至少一个训练样本病灶图像在所述初始图像处理模型中的损失值；
    目标模型生成部件，被配置为判断所述损失值是否处于预设范围内，若所述损失值不在预设范围内，则对所述初始图像处理模型的参数进行更新，将更新参数后的图像处理模型作为下一次训练的初始图像处理模型；继续对更新参数后的图像处理模型进行训练，直至所述损失值处于所述预设范围内，将最后一次训练的图像处理模型作为目标图像处理模型。
  15. 根据权利要求14所述的装置，其中，所述初始图像处理模型还包括神经网络、全局平均池化层、全连接层、特征加权求和层和上采样层；其中，所述神经网络包括至少一个层，每个层依次包括卷积层、激活函数和池化层；
    所述图像处理模型生成装置还包括：
    与所述样本图像输入部件耦接的第一二维特征图获取部件，被配置为调用所述神经网络，将所述训练样本病灶图像依次通过所述神经网络的每个层的卷积层、激活函数和池化层，得到所述训练样本病灶图像的多组二维特征图；
    与所述第一二维特征图获取部件耦接的第一全局平均池化部件，被配置为调用所述全局平均池化层，对所述多组二维特征图进行全局平均池化，得到对应所述多组二维特征图的多组一维特征图；
    与所述第一全局平均池化部件耦接的第一全连接部件，所述第一全连接部件还与所述分类概率获取部件耦接，所述第一全连接部件被配置为调用所述全连接层，对所述多组一维特征图进行特征提取，得到所述训练样本病灶图像的至少一组一维特征图，并将所述至少一组一维特征图输入所述分类概率获取部件；
    与所述第一二维特征图获取部件以及所述第一全局平均池化部件耦接的第一特征加权求和部件，被配置为调用所述特征加权求和层，根据所述多组一维特征图，对所述多组二维特征图中的每组二维特征图进行特征加权求和；
    与所述第一特征加权求和部件耦接的第一上采样部件，所述第一上采样部件还与所述预测数据获取部件耦接，所述第一上采样部件被配置为调用所述上采样层，对经过所述特征加权求和的多组二维特征图进行上采样处理，并将经过所述上采样处理的多组二维特征图输入所述预测数据获取部件。
  16. 一种图像处理装置，包括：
    待处理图像输入部件，被配置为将待处理病灶图像输入目标图像处理模型；其中，所述目标图像处理模型通过如权利要求1～9中任一项所述的方法训练得到，所述目标图像处理模型包括分类层和标注层；
    疾病分类概率获取部件，被配置为调用所述分类层，对所述待处理病灶图像进行分类处理，得到所述待处理病灶图像对应的疾病分类概率；
    病灶区域数据获取部件，被配置为调用所述标注层，对所述待处理病灶图像进行处理，得到所述待处理病灶图像中所包含病灶区域的区域中心点坐标、区域长度和区域宽度；
    疾病种类确定部件，被配置为根据所述疾病分类概率，确定所述待处理病灶图像对应的疾病种类；
    病灶区域确定部件，被配置为根据所述区域中心点坐标、所述区域长度和所述区域宽度，确定所述待处理病灶图像中的病灶标注区域。
  17. 根据权利要求16所述的装置，其中，所述目标图像处理模型还包括神经网络、全局平均池化层、全连接层、特征加权求和层和上采样层；其中，所述神经网络包括至少一个层，每个层依次包括卷积层、激活函数和池化层；
    所述图像处理装置还包括：
    与所述待处理图像输入部件耦接的第二二维特征图获取部件，被配置为 调用所述神经网络，将所述待处理病灶图像依次通过所述神经网络的每个层的卷积层、激活函数和池化层，得到所述待处理病灶图像的多组二维特征图；
    与所述第二二维特征图获取部件耦接的第二全局平均池化部件，被配置为调用所述全局平均池化层，对所述多组二维特征图进行全局平均池化，得到对应所述多组二维特征图的多组一维特征图；
    与所述第二全局平均池化部件耦接的第二全连接部件，所述第二全连接部件还与所述疾病分类概率获取部件耦接，所述第二全连接部件被配置为调用所述全连接层，对所述多组一维特征图进行特征提取，得到所述待处理病灶图像的至少一组一维特征图，并将所述至少一组一维特征图输入所述疾病分类概率获取部件；
    与所述第二二维特征图获取部件以及所述第二全局平均池化部件耦接的第二特征加权求和部件，被配置为调用所述特征加权求和层，根据所述多组一维特征图，对所述多组二维特征图中的每组二维特征图进行特征加权求和；
    与所述第二特征加权求和部件耦接的第二上采样部件，所述第二上采样部件还与所述病灶区域数据获取部件耦接，所述第二上采样部件被配置为调用所述上采样层，对经过所述特征加权求和的多组二维特征图进行上采样处理，并将经过所述上采样处理的多组二维特征图输入所述病灶区域数据获取部件。
  18. 一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机程序指令，所述计算机程序指令在处理器上运行时，使得所述处理器执行如权利要求1～9中任一项所述的图像处理模型生成方法，和/或，如权利要求10～13中任一项所述的图像处理方法。
  19. 一种电子设备，包括处理器、存储器、以及存储在所述存储器上并可在所述处理器上运行的计算机程序；
    所述处理器执行所述计算机程序时，实现如权利要求1～9中任一项所述的图像处理模型生成方法，和/或，如权利要求10～13中任一项所述的图像处理方法。
PCT/CN2020/109611 2019-08-22 2020-08-17 图像处理模型生成方法、图像处理方法、装置及电子设备 WO2021032062A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/423,439 US11887303B2 (en) 2019-08-22 2020-08-17 Image processing model generation method, image processing method and device, and electronic device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910778807.0A CN110490262B (zh) 2019-08-22 2019-08-22 图像处理模型生成方法、图像处理方法、装置及电子设备
CN201910778807.0 2019-08-22

Publications (1)

Publication Number Publication Date
WO2021032062A1 true WO2021032062A1 (zh) 2021-02-25

Family

ID=68552908

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/109611 WO2021032062A1 (zh) 2019-08-22 2020-08-17 图像处理模型生成方法、图像处理方法、装置及电子设备

Country Status (3)

Country Link
US (1) US11887303B2 (zh)
CN (1) CN110490262B (zh)
WO (1) WO2021032062A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110490262B (zh) * 2019-08-22 2022-06-03 京东方科技集团股份有限公司 图像处理模型生成方法、图像处理方法、装置及电子设备
CN112926612A (zh) * 2019-12-06 2021-06-08 中移(成都)信息通信科技有限公司 病理图像分类模型训练方法、病理图像分类方法及装置
CN112331314A (zh) * 2020-11-25 2021-02-05 中山大学附属第六医院 一种图像标注方法、装置、存储介质和电子设备

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005339075A (ja) * 2004-05-26 2005-12-08 Nippon Telegr & Teleph Corp <Ntt> 画像探索装置、画像探索方法、この方法のプログラム及びこのプログラムを記録した記録媒体
JP2010067252A (ja) * 2008-08-15 2010-03-25 Fuji Xerox Co Ltd オブジェクト領域抽出装置及びオブジェクト領域抽出プログラム
CN108564567A (zh) * 2018-03-15 2018-09-21 中山大学 一种超高分辨率病理图像癌变区域可视化方法
CN108961296A (zh) * 2018-07-25 2018-12-07 腾讯科技（深圳）有限公司 眼底图像分割方法、装置、存储介质和计算机设备
CN109363698A (zh) * 2018-10-16 2019-02-22 杭州依图医疗技术有限公司 一种乳腺影像征象识别的方法及装置
CN110136809A (zh) * 2019-05-22 2019-08-16 腾讯科技（深圳）有限公司 一种医疗图像处理方法、装置、电子医疗设备和存储介质
CN110490262A (zh) * 2019-08-22 2019-11-22 京东方科技集团股份有限公司 图像处理模型生成方法、图像处理方法、装置及电子设备

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070049915A1 (en) * 2005-08-26 2007-03-01 Dieter Haemmerich Method and Devices for Cardiac Radiofrequency Catheter Ablation
EP2443992A2 (en) * 2010-10-25 2012-04-25 Fujifilm Corporation Diagnosis support apparatus, diagnosis support method, lesioned part detection apparatus, and lesioned part detection method
TWI460679B (zh) * 2010-12-23 2014-11-11 Nat Univ Chung Hsing Section length enhancement unit and method
KR101993716B1 (ko) * 2012-09-28 2019-06-27 삼성전자주식회사 카테고리별 진단 모델을 이용한 병변 진단 장치 및 방법
KR20150002284A (ko) * 2013-06-28 2015-01-07 삼성전자주식회사 병변 검출 장치 및 방법
US10049451B2 (en) * 2015-12-02 2018-08-14 The Cleveland Clinic Foundation Automated lesion segmentation from MRI images
CN106709929A (zh) * 2016-12-30 2017-05-24 上海联影医疗科技有限公司 医学图像感兴趣区域显示方法及装置
US10593051B2 (en) * 2017-12-20 2020-03-17 International Business Machines Corporation Medical image registration guided by target lesion
US10468142B1 (en) * 2018-07-27 2019-11-05 University Of Miami Artificial intelligence-based system and methods for corneal diagnosis
CN109671053A (zh) * 2018-11-15 2019-04-23 首都医科大学附属北京友谊医院 一种胃癌图像识别系统、装置及其应用
CN110110808B (zh) 2019-05-16 2022-04-15 京东方科技集团股份有限公司 一种对图像进行目标标注的方法、装置及计算机记录介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005339075A (ja) * 2004-05-26 2005-12-08 Nippon Telegr & Teleph Corp <Ntt> 画像探索装置、画像探索方法、この方法のプログラム及びこのプログラムを記録した記録媒体
JP2010067252A (ja) * 2008-08-15 2010-03-25 Fuji Xerox Co Ltd オブジェクト領域抽出装置及びオブジェクト領域抽出プログラム
CN108564567A (zh) * 2018-03-15 2018-09-21 中山大学 一种超高分辨率病理图像癌变区域可视化方法
CN108961296A (zh) * 2018-07-25 2018-12-07 腾讯科技（深圳）有限公司 眼底图像分割方法、装置、存储介质和计算机设备
CN109363698A (zh) * 2018-10-16 2019-02-22 杭州依图医疗技术有限公司 一种乳腺影像征象识别的方法及装置
CN110136809A (zh) * 2019-05-22 2019-08-16 腾讯科技（深圳）有限公司 一种医疗图像处理方法、装置、电子医疗设备和存储介质
CN110490262A (zh) * 2019-08-22 2019-11-22 京东方科技集团股份有限公司 图像处理模型生成方法、图像处理方法、装置及电子设备

Also Published As

Publication number Publication date
CN110490262B (zh) 2022-06-03
CN110490262A (zh) 2019-11-22
US11887303B2 (en) 2024-01-30
US20220114724A1 (en) 2022-04-14

Similar Documents

Publication Publication Date Title
WO2021032062A1 (zh) 图像处理模型生成方法、图像处理方法、装置及电子设备
CN114401666A (zh) 基于深度学习的3d点云的对象检测和实例分割
KR20180025093A (ko) 약한 지도 학습 기반의 기계 학습 방법 및 그 장치
Lei et al. Attention-guided multi-branch convolutional neural network for mitosis detection from histopathological images
US11449717B2 (en) System and method for identification and localization of images using triplet loss and predicted regions
CN110838108A (zh) 基于医疗图像的预测模型构建方法、预测方法及装置
WO2022057309A1 (zh) 肺部特征识别方法、装置、计算机设备及存储介质
JP6756406B2 (ja) 画像処理装置、画像処理方法および画像処理プログラム
CN112132815B (zh) 肺结节检测模型训练方法、检测方法及装置
JP6892606B2 (ja) 位置特定装置、位置特定方法及びコンピュータプログラム
CN114445356A (zh) 基于多分辨率的全视野病理切片图像肿瘤快速定位方法
CN115457492A (zh) 目标检测方法、装置、计算机设备及存储介质
Oloko-Oba et al. Pre-trained convolutional neural network for the diagnosis of tuberculosis
CN112488178B (zh) 网络模型的训练方法及装置、图像处理方法及装置、设备
CN111985616B (zh) 一种图像特征提取方法、图像检索方法、装置及设备
CN112967293A (zh) 一种图像语义分割方法、装置及存储介质
CN110929731B (zh) 一种基于探路者智能搜索算法的医疗影像处理方法及装置
CN115527083B (zh) 图像标注方法、装置和电子设备
CN113111879B (zh) 一种细胞检测的方法和系统
CN112750124B (zh) 模型生成、图像分割方法、装置、电子设备及存储介质
CN113192085A (zh) 三维器官图像分割方法、装置及计算机设备
CN114912502B (zh) 一种基于表情与语音的双模态深度半监督情感分类方法
US20230281843A1 (en) Generating depth images for image data
US20230196752A1 (en) Information processing apparatus, information processing method, and non-transitory computer-readable storage medium
CN113706449B (zh) 基于病理图像的细胞分析方法、装置、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20854077

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20854077

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20854077

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 14.02.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 20854077

Country of ref document: EP

Kind code of ref document: A1