KR20210153700A - Image processing method and apparatus, electronic device, storage medium and computer program - Google Patents

Abstract

An embodiment of the present invention provides an image processing method and apparatus, an electronic device, a computer storage medium and a computer program, wherein the image processing method inputs a first image and a second image to a target classification network to obtain a classification result step; and obtaining a prediction result by performing prediction on change data according to the classification result, wherein a target object in the first image is processed based on a region distribution in the second image, Indicates the status of the acquired change.

Description

Image processing method and apparatus, electronic device, storage medium and computer program

Cross-referencing of related applications

The present invention is filed based on a Chinese patent application with an application number of 201910918450.1 and a filing date of September 26, 2019, and claims the priority of the Chinese patent application, all contents of the Chinese patent application are incorporated herein by reference. .

Embodiments of the present invention relate to the field of computer vision technology, and specifically, to an image processing method and apparatus, an electronic device, a computer storage medium, and a computer program, but is not limited thereto.

Deep learning has grown rapidly and has made remarkable achievements in image processing. Current image processing technology based on deep learning requires multiple processing steps to predict lesion progression, so the processing process is cumbersome and the accuracy of the prediction results obtained through this process is not high.

An embodiment of the present invention provides an image processing method and apparatus, an electronic device, a computer storage medium, and a computer program.

An embodiment of the present invention is

inputting the first image and the second image into a target classification network to obtain a classification result; and

according to the classification result, performing prediction on change data to obtain a prediction result,

The change data provides an image processing method representing a change state obtained by processing the target object in the first image based on the area distribution in the second image.

In the technical solution of the embodiment of the present invention, the classification result can be obtained using only one target classification network, thereby simplifying the image processing process, and the classification result is the first image and the second image in the target classification network. Accuracy of the prediction result is improved by performing prediction on a change situation obtained by jointly inputting and acquiring the target object in the first image based on the area distribution in the second image according to the classification result. improve

In some embodiments of the present invention, the image processing method comprises:

performing training on the classification network to obtain a trained classification network; and

The method further includes making the trained classification network the target classification network.

Using the technical solution of the embodiment of the present invention, by using the trained classification network as the target classification network, a classification result can be obtained using the target classification network, and according to the classification result, the target object in the first image is The accuracy of the prediction result is improved by performing prediction on the change situation obtained by performing processing based on the area distribution in the second image.

In some embodiments of the present invention, performing training on the classification network to obtain a trained classification network includes:

obtaining an image to be processed after performing image superposition processing on the first image and the second image, wherein the first image and the second image are different types of image data; and

and obtaining the trained classification network by performing training by inputting the image to be processed as a training sample into the classification network.

Using the technical solution of the embodiment of the present invention, training of the classification network is performed according to the image to be processed obtained after performing image superposition processing on the first image and the second image, and the two image data are combined It is possible to obtain training precision by performing training in the classification network in consideration of .

In some embodiments of the present invention, obtaining the image to be processed comprises:

performing image cropping on the first image and the second image according to the contour of the target object, respectively, to obtain cropped first sub-image data and second sub-image data; and

and using the first sub-image data and the second sub-image data as the image to be processed.

Using the technical solution of the embodiment of the present invention, according to the outline of the target object, the first image and the second image are respectively cut out to obtain the cut out first sub image data and the second sub image data, and the classification network Since it is used for training of , training efficiency can be improved.

In some embodiments of the present invention, the first sub-image data and the second sub-image data are image data of the same size.

In the technical solution of the embodiment of the present invention, by using image data of the same size, image overlap is implemented by directly using the first sub-image data and the second sub-image data without the need to align the pixel positions in the image overlap processing process. Therefore, the processing efficiency of image superposition is improved.

In some embodiments of the present invention, the step of obtaining the trained classification network by inputting the image to be processed into the classification network as a training sample to perform training comprises:

converting the first sub-image data and the second sub-image data into corresponding histograms, respectively, and equalizing the histograms to obtain an equalization result; and

and performing training on the classification network based on the equalization processing result to obtain the trained classification network.

By using the technical solution of the embodiment of the present invention, by converting the image data into a corresponding histogram and performing equalization processing of the histogram, it is possible to change the distribution of the histogram of the image to an almost uniform distribution, thereby improving the contrast of the image. increase and make the image sharper.

In some embodiments of the present invention, the step of obtaining the trained classification network by inputting the image to be processed into the classification network as a training sample to perform training comprises:

obtaining a normalization processing result by performing normalization processing on corresponding pixel points included in the first sub-image data and the second sub-image data; and

and performing training on the classification network based on the normalization processing result to obtain the trained classification network.

By using the technical solution of the embodiment of the present invention, if normalization processing is performed on pixel points, the distribution of pixel points can be summarized, so that pixel points requiring processing are limited within a preset range after normalization processing can In other words, the normalization process is to make a subsequent series of processes more convenient and quick, and is advantageous for accelerating the convergence speed of classification network training.

In some embodiments of the present invention, the classification network comprises at least one classification processing module,

Obtaining the trained classification network by performing training by inputting the image to be processed into the classification network as a training sample,

obtaining a loss function by performing feature extraction, dimension reduction processing, and global average pooling processing on the image to be processed through the at least one classification processing module; and

and training the classification network according to the backpropagation of the loss function to obtain the trained classification network.

By using the technical solution of the embodiment of the present invention, it is possible to obtain a loss function by performing feature extraction, dimensionality reduction processing, and global average pooling processing on the image to be processed through at least one classification processing module, and the loss function By training the classification network according to the backpropagation of , the target classification network is trained and obtained.

In some embodiments of the present invention, each classification processing module includes at least a convolutional layer,

The step of obtaining a loss function by performing feature extraction, dimensionality reduction processing, and global average pooling processing on the image to be processed through the at least one classification processing module includes:

obtaining a first processing result by performing feature extraction on the image to be processed through a corresponding convolutional layer in the at least one classification processing module, and then performing dimensionality reduction processing;

performing global average pooling processing on the first processing result and then inputting it to a fully connected layer to obtain a second processing result for indicating a prediction change state of the extracted feature; and

and obtaining the loss function according to the second processing result and the manual labeling result.

Using the technical solution of the embodiment of the present invention, a second processing result can be obtained according to the first processing result obtained after the dimension reduction processing, and the loss function is obtained according to the second processing result and the manual labeling result Thus, by training the classification network according to the backpropagation of the loss function, the target classification network is trained and obtained.

In some embodiments of the present invention, when the classification processing module is a residual module, each residual module includes a convolutional layer, a normalization layer and an activation layer,

After performing feature extraction on the image to be processed through a corresponding convolutional layer in the at least one classification processing module,

Obtaining a second extraction result after processing through a normalization layer and an activation layer on a first extraction result obtained after performing feature extraction on the image to be processed through a corresponding convolutional layer in at least one residual module step; and

The method further includes obtaining a third extraction result according to the second extraction result and the image to be processed.

Using the technical solution of the embodiment of the present invention, when the classification processing module is a residual module, the structure of the module includes a convolutional layer, a normalization layer and an activation layer, and is characterized through the corresponding convolutional layer of the residual module For the first extraction result obtained after performing extraction, after processing through the normalization layer and the activation layer, the second extraction result is obtained, and the third extraction result is obtained according to the second extraction result and the image to be processed. to obtain a first processing result for calculating a loss function according to the third extraction result, and after obtaining the loss function, by training the classification network according to backpropagation of the loss function, the target classification network acquired by training.

In some embodiments of the present invention, the step of performing the dimension reduction processing to obtain a first processing result includes:

and performing dimensionality reduction processing according to the third extraction result to obtain the first processing result.

Using the technical solution of the embodiment of the present invention, it is possible to obtain a first processing result for calculating a loss function by performing dimensionality reduction processing on the third extraction result, and after obtaining the loss function, the loss By training the classification network according to the backpropagation of a function, the target classification network is trained and obtained.

An embodiment of the present invention is

a classification unit configured to input the first image and the second image into the target classification network to obtain a classification result; and

a prediction unit configured to obtain a prediction result by performing prediction on the change data according to the classification result;

The change data further provides an image processing apparatus indicating a change state obtained by processing the target object in the first image based on the area distribution in the second image.

In some embodiments of the present invention, the image processing device comprises:

performing training on the classification network to obtain a trained classification network;

and a training unit configured to use the trained classification network as the target classification network.

In some embodiments of the present invention, the training unit,

an overlapping sub-unit configured to obtain an image to be processed after performing image overlap processing on the first image and the second image, wherein the first image and the second image are different types of image data; and

and a training sub-unit, configured to obtain the trained classification network by performing training by inputting the image to be processed as a training sample into the classification network.

In some embodiments of the present invention, the training unit,

According to the contour of the target object, image cropping is performed on the first image and the second image, respectively, to obtain cropped first sub-image data and second sub-image data, and the first sub-image data and the and a division sub-unit configured to use the second sub-image data as the image to be processed.

In some embodiments of the present invention, the first sub-image data and the second sub-image data are image data of the same size.

In some embodiments of the present invention, the training unit,

Converting the first sub-image data and the second sub-image data into corresponding histograms, performing equalization processing on the histogram to obtain equalization processing results, and performing training on the classification network based on the equalization processing results and an equalization processing sub-unit, configured to obtain the trained classification network.

In some embodiments of the present invention, the training unit,

Normalization processing is performed on corresponding pixel points included in the first sub-image data and the second sub-image data to obtain a normalization processing result, and training is performed on the classification network based on the normalization processing result. and a normalization processing sub-unit, configured to obtain the trained classification network.

In some embodiments of the present invention, the classification network comprises at least one classification processing module,

The training sub unit,

A loss function is obtained by performing feature extraction, dimensionality reduction processing, and global average pooling processing on the image to be processed through the at least one classification processing module, and training the classification network according to the backpropagation of the loss function. and obtain a trained classification network.

In some embodiments of the present invention, each classification processing module includes at least a convolutional layer,

The training sub unit,

After performing feature extraction on the image to be processed through a corresponding convolutional layer in the at least one classification processing module, dimensionality reduction processing is performed to obtain a first processing result, and a global After performing the average pooling processing, input to the fully connected layer to obtain a second processing result for representing the predicted change state of the extracted feature, and to obtain the loss function according to the second processing result and manual labeling result do.

In some embodiments of the present invention, when the classification processing module is a residual module, each residual module includes a convolutional layer, a normalization layer and an activation layer,

The training sub unit,

A first extraction result obtained after performing feature extraction on the image to be processed through a corresponding convolutional layer in at least one residual module is processed through a normalization layer and an activation layer, and a second extraction result is obtained, , to obtain a third extraction result according to the second extraction result and the image to be processed.

In some embodiments of the present invention, the training sub-unit is configured to perform dimensionality reduction processing according to the third extraction result to obtain the first processing result.

An embodiment of the present invention is

processor; and

a memory configured to store instructions executable by the processor;

The processor further provides an electronic device configured to perform any one of the image processing methods.

An embodiment of the present invention further provides a computer-readable storage medium storing computer program instructions, and when the computer instructions are executed by a processor, any of the image processing methods are implemented.

An embodiment of the present invention further provides a computer program including a computer readable code, and when the computer readable code is executed in an electronic device, the processor of the electronic device implements any one of the image processing methods.

In an embodiment of the present invention, a first image and a second image are input to a target classification network to obtain a classification result, and according to the classification result, a target object in the first image is determined based on a region distribution in the second image. Prediction is performed on the change situation obtained by performing processing to obtain a prediction result. Since the classification result can be obtained by using only one target classification network, the image processing process is simplified by using the technical solution of the embodiment of the present invention, and the classification result is the first image and the second image in the target classification network. It is obtained by jointly inputting into , and according to the classification result, the target object in the first image performs processing based on the area distribution in the second image to perform prediction on the acquired change situation, so the practice of the present invention Improve the accuracy of prediction results by using example technical solutions.

The general description above and the description of the details that follow are illustrative and for purposes of interpretation and not limitation of the present invention.

Other features and aspects of the present invention will become more apparent upon reference to the drawings below and detailed description of exemplary embodiments.

The drawings herein are incorporated in and constitute a part of this specification, and these drawings show embodiments consistent with the present invention, and together with the specification interpret the technical solutions of the embodiments of the present invention.
1 is a flow schematic diagram of an image processing method provided in an embodiment of the present invention.
2 is a schematic diagram of an application scenario of an embodiment of the present invention.
3 is a schematic diagram of a training flow of a classification network provided in an embodiment of the present invention.
4 is a schematic diagram of a classification network architecture implementing the image processing method provided in the embodiment of the present invention.
5 is a structural schematic diagram of an image processing apparatus provided in an embodiment of the present invention.
6 is a structural schematic diagram of an electronic device according to an embodiment of the present invention.
7 is a structural schematic diagram of another electronic device according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Various exemplary embodiments, features and aspects of the present invention will be described in detail below with reference to the drawings. In the drawings, the same reference numbers indicate elements having the same or similar functions. Although the drawings show various aspects of the embodiment, it is not necessary to draw the drawings to scale unless otherwise indicated.

Here, the dedicated word “exemplary” means “used as an example, embodiment, or illustration”. Any embodiment described herein as “exemplary” is not necessarily to be construed as superior or superior to another embodiment.

As used herein, the term “and/or” is a relational relation that describes a subject of association, indicating that three relations may exist. For example, A and/or B may represent three cases in which only A exists, A and B exist simultaneously, and only B exists. In addition, the term "at least one" in the text indicates any one of a plurality or any combination of at least two of the plurality. For example, it may indicate that at least one of A, B, and C includes any one or a plurality of elements selected from the set consisting of A, B, and C.

In addition, in order to better explain the embodiments of the present invention, numerous specific details are provided in the specific embodiments below. It should be understood by those of ordinary skill in the art that embodiments of the present invention may likewise be implemented without some specific details. In some embodiments, the subject matter of embodiments of the present invention is obscured by not describing in detail methods, means, elements and circuits familiar to those skilled in the art.

One application direction of image processing is to obtain more accurate prediction results by optimizing medical image processing and further simplifying the processing process. The prediction result may be a prediction result of the degree of influence on the specific area when the specific area lesion is processed. For example, the lesion may be in the abdomen, lung, kidney, brain, heart, etc., and if the specific area is the lung, the location where the lesion exists in the lung affects the lung after surgery (eg, moderate or non-severe). degree, etc.) should be made predictions.

For example, before performing radiation therapy on a lesion in the lungs, it is necessary to estimate the extent of the effect of radiation therapy on the lungs. Radiation therapy to the lungs can cause radiation pneumonitis, in which the lesion (eg, lung cancer, breast cancer, esophageal cancer, malignant lymphoma, or other thoracic malignancy) has undergone radiation therapy and then normal lung tissue within the radiation field. It is an inflammatory response caused by damage. In mild cases, there are no symptoms and the inflammation disappears spontaneously, but in severe cases, extensive fibrosis of the lungs develops, impairing respiratory function and leading to respiratory failure. The degree of inflammatory response is closely related to the radiation dose and the condition of the lesion before radiation therapy, and in the case of lung cancer radiation therapy, prediction should be made on the severity of radiation pneumonitis after treatment, the process is relatively cumbersome. Image features are extracted from images such as embedded computed tomography (CT) images, and then image features to be processed are selected from the extracted image features, and classification is performed in the classifier, and according to the classification result, Estimate the degree of impact.

In the related art, extraction of image features may be implemented through radiomix. Extracting image features through radiomix is to study the relationship between image features and clinical symptoms (eg, predicting the degree of influence on a specific area) after extracting image features through a radiographic imaging method. After image features are extracted, features are selected, and the degree of influence (eg, severity or non-severity, etc.) on a specific region can be predicted through a classifier such as Support Vector Machines (SVM). As can be seen from this, the entire image processing process includes several steps, which is not only cumbersome, but also the accuracy of the prediction result obtained therefrom is not high. The reasons for not having high accuracy are as follows.

1) There are many manually set hyperparameters at each stage, and whether the artificially set hyperparameter selection is correct has a very large impact on the final prediction result. In other words, if the artificially set hyperparameter selection is not accurate, the final prediction result is also not accurate.

2) The prediction of the radiation dose used for radiation therapy and the prediction of the image processing process are respectively performed, and the prediction process can be implemented by obtaining a dose constant by obtaining an average value of the total internal lung radiation dose. For example, in general, the absorbed dose of radiation is measured in gray (Gy, Gy) units, and the doctor can use the dose constant as the dose constant by stating the percentage of the tissue that the absorbed dose in the lungs exceeds a certain value in the entire lung. have. For example, V20 is the percentage of the total lung volume occupied by tissues with absorbed doses exceeding 20 Gy in the lungs. The treatment method using the dose constant is too general and does not take into account that the radiation dose is different according to the location of the lesion. If the lesion is located in different areas such as abdomen, lung, kidney, brain, and heart, it is natural that the amount of radiation used is different, and the effect after radiation treatment is also different. For example, although the amount of internal radiation in the entire lung is relatively small and the statistical water supply is also relatively small, when radiation is irradiated to major organs, such as major organs, blood vessels, and heart, serious consequences occur. In other words, the constant used in the method of processing the dose constant is only one statistic, and the distribution of radiation in different area spaces is not considered, and the prediction accuracy obtained using the method of processing the dose constant from this is also not high

In summary, in the related art, the task of predicting the severity of pneumonia after radiotherapy is mainly solved by the method of radiomix, which has disadvantages such as low efficiency, low robustness, no consideration of radiation distribution, and low accuracy. Although some radiomix methods provide ideal accuracy, the optimization of feature selection and the selection of dose constants and hyperparameters of SVM in the process lower the robustness of the method and are difficult to apply to other data sets. In addition, it is possible to perform a constantization process on the radiation dose, that is, the radiation dose in the entire lung or cancer area is statistic as a single constant, but this calculation loses the distribution characteristics of the radiation.

Using an embodiment of the present invention, the prediction of the severity of pneumonia after radiation treatment, for example, through a target classification network obtained after deep learning training (eg, a classification neural network, the network may be three-dimensional), By simultaneously inputting a lung image and a radiation distribution image (both of which may be three-dimensional images) into the target classification network, a specific region or a related region where the lesion is located is an image at each position through the target classification network. And radiation distribution can be acquired comprehensively to improve prediction accuracy, and through classification of the target classification network, it is possible to immediately output the severity of pneumonia after radiation treatment at once. The embodiment of the present invention not only considers the distribution of radiation dose in the prediction process of image processing, but can also be used for a similar working data set, for example, to change any parameters and structure for radiation pneumonia data in any hospital. All of the embodiments of the present invention can be directly applied to predict the severity of radiation pneumonitis without need, and the application scenario is not limited to lesion prediction in different regions or related regions such as abdomen, lung, kidney, brain, heart, etc., accurate prediction Results can be obtained quickly.

1 is a flow diagram of an image processing method provided in an embodiment of the present invention, wherein the image processing method is applied to an image processing apparatus, for example, the image processing apparatus is to be executed by a terminal device or a server or other processing device The terminal device may be a user equipment (UE), a mobile device, a cellular phone, a wireless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. can In some embodiments of the present invention, the image processing method may be implemented in a manner of calling a computer readable instruction stored in a memory through a processor. As shown in FIG. 1 , the process includes the following steps.

In step S101, the first image and the second image are input to the target classification network to obtain a classification result.

In some embodiments of the present invention, the first image may be an image with a lesion (which may be a CT image of a region where the lesion is located), for example, a lesion is a different region or related region, such as abdomen, lung, kidney, brain, heart, etc. It is an image located in The second image is a radiation dose distribution map used to perform radiation therapy on a region in which the lesion is located or a related region. In actual implementation, the classification result can be obtained by jointly inputting the image of the lesion and the radiation dose distribution into the target classification network.

In step S102, according to the classification result, prediction is performed on the change data to obtain a prediction result.

Here, the change data represents a change state obtained by processing the target object in the first image based on the area distribution in the second image.

In some embodiments of the present invention, the target object may be an organ in which a lesion is located, such as abdomen, lung, kidney, brain, heart, and the like. The area distribution may be a distribution of radiation doses used for different lesions in different areas. The change situation may be a severity (eg, probability of severity or probability of non-severity) that may cause inflammation in an organ (eg, lung) in which the lesion is located after radiotherapy is performed on the lesion. For example, according to the classification result, it is possible to obtain a prediction result by predicting the severity of inflammation obtained based on the radiation distribution in the radiation dose distribution diagram for the lung among the images in which the lesion exists.

Using the technical solution of the embodiment of the present invention, prediction can be implemented simply by classifying according to the target classification network, so that the prediction result can be obtained end-to-end in one step without the need for cumbersome manipulations in multiple steps. Comprehensive consideration of the mutual influence of the image and the radiation dose distribution in which the lesion exists, and the radiation dose distribution map is used as a common input rather than a process that separates both, and due to the mutual influence between the images, the radiation dose at different locations The prediction accuracy is improved by sufficiently considering the different influences due to the difference of . In addition, the target classification network can implement self-adaptive adjustment of prediction in the entire image processing process using the target classification network obtained after deep learning training without the need to adjust with artificial hyperparameters, which helps to improve prediction accuracy.

The image processing method of the embodiment of the present application may be applied to a scenario such as a radiation treatment plan before lung cancer surgery and a prediction of radiation pneumonia. 2 is a schematic diagram of an application scenario of an embodiment of the present invention, and as shown in FIG. 2 , a CT image 201 of a lung cancer patient is the first image, and a radiation dose distribution diagram 202 is the second image. , doctors should plan radiation therapy surgery after acquiring CT images of lung cancer patients. At this time, the radiation dose distribution map and the CT image of the lung cancer patient can be input to the image processing apparatus 200, and when the image processing apparatus processes the image processing method according to the above-described embodiment, radiation pneumonia occurs after radiation treatment Because it is possible to obtain a predictive result of the highest severity and predict the severity of the occurrence of radiation pneumonia after radiation treatment, it can help doctors to predict the risk after surgery in advance to take preventive measures or to revise the radiation treatment plan. can

In some embodiments of the present invention, training may be performed on the classification network to obtain a trained classification network, and the trained classification network may be used as the target classification network.

3 is a schematic diagram of a training flow of a classification network provided in an embodiment of the present invention, and the training process of the classification network may be implemented by an image processing apparatus, for example, the image processing apparatus is a terminal device or a server or other may be executed by a processing device, wherein the terminal device may be a UE, a mobile device, a cellular phone, a cordless phone, a PDA, a hand-held device, a computing device, an in-vehicle device, a wearable device, and the like. In some embodiments of the present invention, the image processing method may be implemented in a manner of calling a computer readable instruction stored in a memory through a processor. As shown in FIG. 3 , the training process of the classification network may include the following steps.

In step S301, an image to be processed is obtained after image superimposition processing is performed on the first image and the second image.

In an embodiment of the present invention, the first image and the second image may be different types of image data.

In some embodiments of the present invention, after image superposition processing is performed on the first image and the second image, image cropping is performed on the first image and the second image respectively according to the outline of the target object, , the cropped first sub-image data and the second sub-image data may be obtained.

It should be pointed out that the contour of the target object is not the contour of the lesion, but the contour of the region where the lesion is located or a related region, for example, the entire lung contour of the lung where the lesion is located, or the contour of the heart, kidney, etc., where the lesion is located, in different regions Different radiation doses are used.

In some embodiments of the present invention, the first sub-image data and the second sub-image data may be image data of the same size. For example, the three-dimensional first image and the second image are divided according to the lung contour, but both have the same size. Next, the first sub-image data and the second sub-image data may be used as the image to be processed.

In some embodiments of the present invention, the first sub-image data and the second sub-image data may be image data of different sizes. In this case, the first sub-image data and the second sub-image data include A pixel position alignment process is performed on a corresponding pixel point to obtain aligned first sub-image data and aligned second sub-image data, and the aligned first sub-image data and aligned second sub-image data include After implementing image overlapping processing by overlapping corresponding pixel points, the image to be processed may be obtained.

In step S302, the trained classification network is obtained by inputting the image to be processed as a training sample to the classification network to perform training.

An embodiment of the present invention may obtain a trained classification network by performing training on the classification network through steps S301 to S302. Let the trained classification network be the target classification network.

In some embodiments of the present invention, the step of obtaining the trained classification network by inputting the image to be processed as a training sample into the classification network and performing training includes: first sub-image data and second sub-image data; converting each of the histograms into corresponding histograms and performing equalization processing on the histograms to obtain equalization processing results; and performing training on the classification network based on the equalization processing result to obtain the trained classification network.

In some embodiments of the present invention, the step of obtaining the trained classification network by inputting the image to be processed as a training sample to the classification network and performing training includes: first sub-image data and second sub-image data; performing normalization processing on the included corresponding pixel points to obtain a normalization processing result; and performing training on the classification network based on the normalization processing result to obtain the trained classification network.

In some embodiments of the present invention, first, the first sub-image data and the second sub-image data are converted into corresponding histograms, and the histogram is equalized to obtain an equalization processing result. Next, normalization processing may be performed on corresponding pixel points included in the corresponding first sub-image data and the corresponding second sub-image data among the results of the equalization processing. For example, after histogram equalization is performed on these two sub-images, both are normalized and combined into image data represented by a 2-channel 4-dimensional matrix. Input the image data into the classification network, first perform feature extraction for each layer through the convolutional layer of the classification network, perform dimensionality reduction processing, and ultimately perform radioactive treatment after radiation treatment through fully connected layer processing Acquires a probability of severity that inflammation occurs.

In some embodiments of the present invention, the classification network may include at least one classification processing module.

Correspondingly, the step of performing training by inputting the image to be processed as a training sample into the classification network to obtain the target classification network includes: extracting features for the image to be processed through the at least one classification processing module ( For example, performing feature extraction through a convolutional layer), dimensionality reduction processing (eg, pooling processing), and global average pooling processing to obtain a loss function; and training the classification network (eg, an error calculated according to the loss function may be backpropagated) according to the backpropagation of the loss function to obtain the trained classification network. .

In some embodiments of the present invention, each classification processing module includes at least a convolutional layer, and performing feature extraction, dimensionality reduction processing, and global average pooling processing on the image to be processed through the at least one classification processing module to obtain a loss function by performing layer-by-layer feature extraction on the image to be processed through a corresponding convolutional layer in at least one classification processing module, and then performing dimensionality reduction processing for each layer to obtain the first processing result obtaining; After performing global average pooling processing on the first processing result, input to the fully connected layer to obtain a second processing result - The second processing result is a prediction result output by the classification network, and prediction change of the extracted feature It is intended to indicate a situation - ; and obtaining the loss function according to the second processing result and the manual labeling result (eg, the actual change situation already labeled by the doctor).

In other words, the loss function can be obtained according to the prediction result output by the classification network and the actual change situation already labeled by the doctor. If the error between the predicted change situation and the actual situation reflected by the loss function is within the preset range (for example, the error is 0), explain that the difference between the generated predicted change situation and the actual situation has reached the convergence condition; , to obtain a trained target classification network by terminating training on the classification network.

In some embodiments of the present invention, when the classification processing module is a residual module, each residual module may include a convolutional layer, a normalization layer and an activation layer.

Correspondingly, after performing layer-by-layer feature extraction on the image to be processed through the corresponding convolutional layer in the at least one classification processing module, the feature is applied to the image to be processed through the corresponding convolutional layer in the at least one residual module. obtaining a second extraction result after processing the first extraction result obtained after performing extraction through a normalization layer and an activation layer; and obtaining a third extraction result for dimensionality reduction processing according to the second extraction result and the image to be processed. In other words, the input of the residual module is an "image to be processed", and the final extraction result obtained by adding the input of the residual module and the output of the last activation layer among the residual modules is the third extraction result. After performing feature extraction through the residual module, the first processing result may be obtained by performing dimensionality reduction processing for each layer. For example, the first processing result is obtained by performing dimension reduction processing for each layer according to the third extraction result.

4 is a schematic diagram of a classification network architecture implementing the image processing method provided in an embodiment of the present invention, and as shown in FIG. 4 , the classification network (eg, a classification neural network) includes at least one classification processing module 11 ) may be included. The classification processing module 11 may be a residual module 12 , and may further include a fully connected layer 13 . Each residual module 12 may include at least one convolutional layer 121 , at least one regularization layer 122 , and at least one activation layer 123 . By extracting useful features and performing prediction by applying these features rather than selecting features again after feature extraction, prediction accuracy is improved compared to related technologies.

The prediction of the severity of radiation pneumonitis after radiation therapy will be described as an example. In the process of training the classification network, we first divide the lung image and the radiation distribution image (both are three-dimensional images) into two sub-images of the same size according to the lung contour. Histogram equalization is performed on these two sub-images, and both are normalized and combined into a 2-channel 4-dimensional matrix. The four-dimensional matrix is input to the classification network, and the convolutional layer of the classification processing module 11 (specifically, the convolutional layer 121, the normalization layer 122 and the activation layer 123 of each residual module 12) ), feature extraction is performed for each layer and dimensionality reduction processing is performed, and finally, the probability of severity of similar pneumonia after radiation treatment is obtained through the fully connected layer 12. .

In one example, the process of training the classification network may include the following contents.

1. Referring to Fig. 4, the three-dimensional lung image and the radiation dose distribution image are cut into images of the same size (200x240x240) according to the lung contour, and then an image of the size (100x120x120) is acquired through downsampling and stored in the video memory. It adapts, and combines the down-sampled image corresponding to the lung image and the down-sampled image corresponding to the radiation distribution image into a four-dimensional matrix (2x100x120x120).

2. Using a three-dimensional convolutional neural network such as Res-Net (e.g., ResNeXt50 shown in Fig. 4), Dense-Net, etc., to perform convolution, generalization and activation operations on the four-dimensional matrix after combining, After enhancing the feature channels from 2 to 2048, global average pooling is performed on the features to obtain a one-dimensional vector, and the one-dimensional vector is input to a fully connected layer to obtain two values (probability of severe or non-serious). output, and finally, the final prediction result (prediction probability) can be obtained through the softmax function.

In an embodiment of the present invention, the classification network may use a serialized, modularized neural network.

Serialization refers to the ability to sequentially process data input to the neural network according to the serialized module among the neural networks (for example, a four-dimensional matrix obtained by combining two sub-images cut to the same size), and modularization means that the module of the neural network It indicates that it can be freely replaced with other modules that can implement embodiments of the present invention, and the present invention can be implemented even after module replacement, and all fall within the protection scope of the present invention.

It should be pointed out that an image obtained after combining the two sub-images cut to the same size corresponds to one 4D matrix (which may be a 4D matrix of 2 channels). Performing feature extraction through the convolutional layer includes performing convolution processing on an input 3D matrix using at least one convolutional kernel, outputting a 4D matrix in which the number of channels is the number of convolutional kernels, and convolution As the number of solution kernels increases, the number of channels in the matrix also increases to 2048. A generalization process may be performed through a normalization layer, and the generalization process may be performed on the four-dimensional matrix using Equation (1).

X=(X-u)/v (1)

Here, X is a four-dimensional matrix, u is the matrix mean, and v is the matrix variance.

In the case of an implementation in which an activation operation is performed through an activation layer, the expressive ability of a neural network can be improved by adding a non-linear element using an activation function.

Global average pooling is to obtain a one-dimensional vector whose length is the number of channels by calculating the average value of the three-dimensional matrix of each channel. After obtaining a one-dimensional vector by performing global average pooling on the features, the one-dimensional vector can be calculated through a neural network through a fully connected layer, and finally two values (probability of serious or non-serious) are obtained Finally, through softmax, the output values of various classifications (severe or non-serious probabilities) can be converted into relative probabilities and used as the final prediction result.

3. Calculate the error between the predicted change situation and the actual situation generated through the cross entropy loss function by weighting the predicted probability and the actual probability, obtain the gradient of each parameter in the classification network after inverse derivation, and use the deep learning optimizer ( For example, by calculating the difference value updated through the Adam optimizer) and adding it to the original parameter to implement the update of the classification network parameter, this process is performed until the error is within a preset range (for example, the error is 0) By continuous iteration, the classification network converges. In this way, a trained target classification network can be obtained.

A person of ordinary skill in the art can understand that in the image processing method of the specific embodiment, the writing order of each step does not imply a strict execution order and does not constitute any limitation on the implementation process, The specific execution order of each step should be determined by its function and possible internal logic.

Each of the method embodiments mentioned in the present invention may be combined with each other to form a combined embodiment as long as the principles and logic are not violated, and the embodiments of the present invention will not be described further due to content limitations.

In addition, the embodiment of the present invention further provides an image processing device, an electronic device, a computer readable storage medium, and a program, all of which can be used to implement any one image processing method provided in the embodiment of the present invention, , the corresponding technical solutions and explanations refer to the corresponding description in the method section and are not further described.

5 is a structural schematic diagram of an image processing apparatus provided in an embodiment of the present invention, and as shown in FIG. 5 , the image processing apparatus according to an embodiment of the present invention inputs a first image and a second image to a target classification network a classification unit 31 configured to obtain a classification result; and a prediction unit 32 configured to obtain a prediction result by performing prediction on the change data according to the classification result, wherein the change data is a region in the second image where the target object in the first image is It represents the change situation obtained by performing processing based on the distribution.

In some embodiments of the present invention, the image processing device comprises:

performing training on the classification network to obtain a trained classification network;

and a training unit configured to use the trained classification network as the target classification network.

In some embodiments of the present invention, the training unit,

an overlapping sub-unit configured to obtain an image to be processed after performing image overlap processing on the first image and the second image, wherein the first image and the second image are different types of image data; and

and a training sub-unit, configured to obtain the trained classification network by performing training by inputting the image to be processed as a training sample into the classification network.

In some embodiments of the present invention, the training unit,

According to the contour of the target object, image cropping is performed on the first image and the second image, respectively, to obtain cropped first sub-image data and second sub-image data, and the first sub-image data and the and a division sub-unit configured to use the second sub-image data as the image to be processed.

In some embodiments of the present invention, the first sub-image data and the second sub-image data are image data of the same size.

In some embodiments of the present invention, the training unit,

Converting the first sub-image data and the second sub-image data into corresponding histograms, performing equalization processing on the histogram to obtain equalization processing results, and performing training on the classification network based on the equalization processing results and an equalization processing sub-unit, configured to obtain the trained classification network.

In some embodiments of the present invention, the training unit,

Normalization processing is performed on corresponding pixel points included in the first sub-image data and the second sub-image data to obtain a normalization processing result, and training is performed on the classification network based on the normalization processing result. and a normalization processing sub-unit, configured to obtain the trained classification network.

In some embodiments of the present invention, the classification network comprises at least one classification processing module,

The training sub unit obtains a loss function by performing feature extraction, dimensionality reduction processing, and global average pooling processing on the image to be processed through the at least one classification processing module, and the classification according to the backpropagation of the loss function and train the network to obtain the trained classification network.

In some embodiments of the present invention, each said classification processing module comprises at least a convolutional layer,

The training sub unit performs feature extraction on the image to be processed through a corresponding convolutional layer in the at least one classification processing module, and then performs dimensionality reduction processing to obtain a first processing result, and After performing global average pooling processing on the processing result, the second processing result is obtained by inputting it into the fully connected layer to indicate the predicted change status of the extracted feature, and the loss according to the second processing result and manual labeling result is configured to obtain a function.

In some embodiments of the present invention, when the classification processing module is a residual module, each residual module includes a convolutional layer, a normalization layer and an activation layer,

The training sub unit performs feature extraction on the image to be processed through a corresponding convolutional layer in at least one residual module, and then processes the first extraction result obtained through a normalization layer and an activation layer, and then a second and obtain an extraction result, and obtain a third extraction result according to the second extraction result and the image to be processed.

In some embodiments of the present invention, the training sub-unit is configured to perform dimensionality reduction processing according to the third extraction result to obtain the first processing result.

In some embodiments, a function or a module included in the apparatus provided in the embodiment of the present invention may perform the method described in the above-described method embodiment, the specific implementation of which is the description of the above-described method embodiment. , which is not further described here for the sake of brevity.

An embodiment of the present invention further provides a computer-readable storage medium storing computer program instructions, and when the computer program is executed by a processor, any one of the image processing methods is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

An embodiment of the present invention further provides an electronic device comprising a processor and a memory configured to store instructions executable by the processor, wherein the processor is configured to perform any one of the image processing methods.

The electronic device may be a terminal, server, or other type of device.

An embodiment of the present invention further provides a computer program including a computer readable code, and when the computer readable code is executed in an electronic device, the processor of the electronic device performs any one of the image processing methods.

6 is a structural schematic diagram of an electronic device according to an embodiment of the present invention, and as shown in FIG. 6 , the electronic device 800 includes a mobile phone, a computer, a digital broadcasting terminal, a message transmitting/receiving device, a game console, a tablet device, a medical device, It may be a terminal such as a fitness device or a personal information terminal.

Referring to FIG. 6 , the electronic device 800 includes a first processing component 802 , a first memory 804 , a first power component 806 , a multimedia component 808 , an audio component 810 , and a first input may include one or more components of an Input Output (I/O) interface 812 , a sensor component 814 , and a communication component 816 .

The first processing component 802 generally controls the overall operation of the electronic device 800 , such as operations related to displays, phone calls, data communications, camera operations, and recording operations. The first processing component 802 includes one or more processors 820 to execute instructions to complete all or some steps of the image processing method. In addition, the first processing component 802 may include one or a plurality of modules to facilitate the interaction between the first processing component 802 and other components. For example, the first processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the first processing component 802 .

The first memory 804 stores various types of data to support operations in the electronic device 800 . Examples of such data include commands, contact data, phonebook data, messages, photos, videos, and the like of any application or method operating in the electronic device 800 . The first memory 804 includes Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EEPROM), and Electrical Erasable Programmable Read Only Memory (EEPROM). Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, any It may be implemented by a tangible volatile or non-volatile storage device or a combination thereof.

The first power component 806 provides power to various components of the electronic device 800 . The first power component 806 may include a power management system, one or more power sources, and other components related to the creation, management, and power allocation of the electronic device 800 .

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Pad (TP). If the screen includes a touch pad, the screen may be implemented as a touch screen to receive an input signal from a user. The touch pad includes one or a plurality of touch sensors for sensing touch, slide, and gestures on the touch pad. The touch sensor not only senses a boundary of a touch or slide action, but also senses a duration and pressure associated with the touch or slide action. In some embodiments, multimedia component 808 includes one front camera and/or one rear camera. When the electronic device 800 is in an operation mode such as a photographing mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras may be one fixed optical lens system or may have focal length and optical zoom functions.

The audio component 810 is configured to output and/or input an audio signal. For example, the audio component 810 includes one microphone (MIC), and when the electronic device 800 is in an operation mode such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. is composed The received audio signal may be stored in the first memory 804 or transmitted via the communication component 816 . In some embodiments, the audio component 810 further includes one speaker for outputting an audio signal.

The first input/output interface 812 provides an interface between the first processing component 802 and a peripheral device interface module, which may be a keyboard, a click wheel, a button, or the like. Such buttons may include, but are not limited to, a home button, a volume button, a start button, and a lock button.

The sensor component 814 includes one or a plurality of sensors for providing the electronic device 800 with status assessments of various aspects. For example, the sensor component 814 may sense the on/off state of the electronic device 800 , the relative position of the component, for example, the component is the display and keypad of the electronic device 800 , and a sensor The component 814 is the electronic device 800 or a change in the position of a component of the electronic device 800 , whether the user and the electronic device 800 are in contact, the orientation or acceleration/deceleration of the electronic device 800 , and the electronic device 800 . temperature change can be detected. The sensor component 814 may include a proximity sensor that senses the presence of a surrounding object without physical contact. The sensor component 814 may include a proximity sensor configured to sense the presence of a nearby object when no physical contact is present. The sensor component 814 may further include an optical sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor for use in imaging applications. In some embodiments, the sensor component 814 may further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system through a broadcast channel. In one exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module that facilitates near field communication. For example, the NFC module includes a radio frequency identification (RFID) technology, an infrared communication standard (Infrared Data Association, IrDA) technology, an Ultra Wide Band (UWB) technology, a Bluetooth (BT) technology, and It can be implemented based on other technologies.

In an exemplary embodiment, the electronic device 800 includes one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processor devices (DSPDs). ), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, microcontroller, microprocessor or other electronic device capable of executing the image processing method. have.

In an exemplary embodiment, a non-volatile computer readout comprising a first memory 804 comprising computer program instructions executable by the processor 820 of the electronic device 800 to perform any of the above image processing methods. A possible storage medium is further provided.

7 is a structural schematic diagram of another electronic device according to an embodiment of the present invention, and as shown in FIG. 7 , the electronic device 900 may be provided as one server. Referring to FIG. 7 , the electronic device 900 includes a second processing component 922 , which also stores instructions executable by one or more processors and processing components 922 , for example an application program. and a memory resource representative of the second memory 932 for The application program stored in the second memory 932 may include one or more modules corresponding to each command. In addition, the second processing component 922 executes instructions to perform the image processing method.

The electronic device 900 includes one second power component 926 configured to perform power management of the electronic device 900 , one wired or wireless network interface 950 configured to connect the electronic device 900 to a network; and a second input/output (I/O) interface 958 . The electronic device 900 may operate an operating system stored in the second memory 932 , for example, Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

In an exemplary embodiment, a non-volatile computer read comprising a second memory 932 comprising computer program instructions executable by a second processing component 922 of an electronic device 900 to perform the image processing method A possible storage medium is further provided.

An embodiment of the invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having loaded thereon computer-readable program instructions for causing a processor to implement each aspect of the present invention.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include, but are not limited to, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory). , static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital video disc (DVD), memory sticks, floppy disks, machine encoding devices, punch cards with instructions stored therein, or projections in grooves structures and any suitable combination thereof. As used herein, a computer-readable storage medium includes wireless electrical waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or electrical signals transmitted over electrical wires and It is not interpreted as the same temporary signal itself.

The computer readable program instructions described herein may be downloaded to each computing/processing device from a computer readable storage medium, or through a network such as the Internet, a local area network, a wide area network and/or a wireless Internet, an external computer or external storage device. can be downloaded to Networks may include copper transport cables, fiber optic transport, wireless transport, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter or network interface of each computing/processing device receives computer readable program instructions from the network to be stored in a computer readable storage medium of each computing/processing device, and transmits the computer readable program instructions.

The computer program instructions for executing the operations of the embodiments of the present invention may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or any of one or more programming languages. It may be source code or object code written in combination, and the programming language includes an object-oriented programming language such as Smalltalk, C++, or the like, and an existing procedural programming language such as a "C" language or similar programming language. The computer readable program instructions run entirely on the user's computer, partially on the user's computer, as a standalone software package, in part on the user's computer and partly on the remote computer, or entirely on the remote computer Or it can run on a server. In the case of a remote computer, the remote computer may be connected to your computer or to an external computer through any kind of network, including a local area network (LAN) or a wide area network (WAN) (e.g. For example, using an Internet service provider to connect via the Internet). In some embodiments, state information in computer readable program instructions is used to personalize an electronic circuit, such as a programmable logic circuit, an FPGA, or a programmable logic array (PLA), the electronic circuit being computer readable. Each aspect of the embodiments of the present invention may be implemented by executing program instructions.

Herein, each aspect of an embodiment of the present invention will be described with reference to a flowchart and/or a block diagram of a method, an apparatus (system) and a computer program product of the embodiment of the present invention. It should be understood that each block in the flowcharts and/or block diagrams and combinations of respective blocks in the flowcharts and/or block diagrams may all be implemented by computer readable program instructions.

These computer readable program instructions are provided to the processor of a general purpose computer, special purpose computer, or other programmable data processing device to create a machine, such that when these instructions are executed by the processor of the computer or other programmable data processing device, the flow chart and/or an apparatus that implements functions/acts specified in one or a plurality of blocks in the block diagram. Further, such computer readable program instructions may be stored in a computer readable storage medium, which instructions cause a computer, programmable data processing apparatus, and/or other device to work in a specified manner, thereby causing the computer readable medium having the instructions stored thereon. includes an article of manufacture comprising instructions for implementing each aspect of a function/action specified in one or a plurality of blocks of the flowchart and/or block diagrams.

Also, a process implemented by a computer by loading computer readable program instructions into a computer, other programmable data processing device, or other device causes a sequence of steps to be performed on the computer, other programmable data processing device, or other device. By causing them to be generated, instructions executed on a computer, other programmable data processing device, or other device may cause functions/actions specified in one or a plurality of blocks in the flowcharts and/or block diagrams to be implemented.

The flowchart and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products in accordance with various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, program segment, or portion of an instruction comprising one or more executable instructions for implementing specified logical functions. It should be noted that, in some alternative implementations, the functions indicated in the blocks may be implemented in a different order than shown in the figures. For example, two blocks expressed in succession may actually be executed in parallel, or may sometimes be executed in the reverse order depending on the function involved. In addition, each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, may be implemented by dedicated hardware-based systems that perform specified functions or operations, or combinations of dedicated hardware and computer instructions. It should be noted that it may be implemented by

In the above content, each embodiment of the present invention has already been described, and the above description is illustrative only and not exhaustive, and is not limited to each disclosed embodiment. Many modifications and changes without departing from the scope and spirit of each described embodiment will be apparent to those of ordinary skill in the art. The choice of terminology used herein best describes the principle of each embodiment, practical application, or technological improvement over market technology, or one of ordinary skill in the art will describe each embodiment disclosed herein by one of ordinary skill in the art. to make it understandable.

Industrial Applicability

An embodiment of the present invention provides an image processing method and apparatus, an electronic device, a computer storage medium and a computer program, wherein the image processing method inputs a first image and a second image to a target classification network to obtain a classification result step; and obtaining a prediction result by performing prediction on change data according to the classification result, wherein the change data is processed based on the distribution of regions in the second image of the target object in the first image It indicates the status of the change obtained.

Claims (25)

An image processing method comprising:
inputting the first image and the second image into a target classification network to obtain a classification result; and
and obtaining a prediction result by performing prediction on change data according to the classification result, wherein a target object in the first image is processed based on an area distribution in the second image, An image processing method, characterized in that it represents the acquired change situation. According to claim 1,
The image processing method is
performing training on the classification network to obtain a trained classification network; and
The image processing method according to claim 1, further comprising the step of using the trained classification network as the target classification network. 3. The method of claim 2,
Obtaining a trained classification network by performing training on the classification network,
obtaining an image to be processed after performing image superposition processing on the first image and the second image, wherein the first image and the second image are different types of image data; and
and acquiring the trained classification network by performing training by inputting the image to be processed as a training sample into the classification network. 4. The method of claim 3,
The step of obtaining the image to be processed includes:
performing image cropping on the first image and the second image according to the contour of the target object, respectively, to obtain cropped first sub-image data and second sub-image data; and
and using the first sub-image data and the second sub-image data as the image to be processed. 5. The method of claim 4,
The image processing method according to claim 1, wherein the first sub-image data and the second sub-image data are image data of the same size. 6. The method according to claim 4 or 5,
Obtaining the trained classification network by performing training by inputting the image to be processed into the classification network as a training sample,
converting the first sub-image data and the second sub-image data into corresponding histograms, respectively, and equalizing the histograms to obtain an equalization result; and
and acquiring the trained classification network by performing training on the classification network based on the equalization processing result. 6. The method according to claim 4 or 5,
Obtaining the trained classification network by performing training by inputting the image to be processed into the classification network as a training sample,
obtaining a normalization processing result by performing normalization processing on corresponding pixel points included in the first sub-image data and the second sub-image data; and
and acquiring the trained classification network by performing training on the classification network based on the normalization processing result. 8. The method according to any one of claims 3 to 7,
the classification network comprises at least one classification processing module;
Obtaining the trained classification network by performing training by inputting the image to be processed into the classification network as a training sample,
obtaining a loss function by performing feature extraction, dimension reduction processing, and global average pooling processing on the image to be processed through the at least one classification processing module; and
and training the classification network according to backpropagation of the loss function to obtain the trained classification network. 9. The method of claim 8,
each said classification processing module comprises at least a convolutional layer,
The step of obtaining a loss function by performing feature extraction, dimension reduction processing, and global average pooling processing on the image to be processed through the at least one classification processing module,
obtaining a first processing result by performing feature extraction on the image to be processed through a corresponding convolutional layer in the at least one classification processing module, and then performing dimensionality reduction processing;
performing global average pooling processing on the first processing result and then inputting it to a fully connected layer to obtain a second processing result for indicating a prediction change state of the extracted feature; and
and obtaining the loss function according to the second processing result and the manual labeling result. 10. The method of claim 9,
When the classification processing module is a residual module, each of the residual module includes a convolutional layer, a normalization layer and an activation layer,
After performing feature extraction on the image to be processed through a corresponding convolutional layer in the at least one classification processing module,
Obtaining a second extraction result after processing through a normalization layer and an activation layer on a first extraction result obtained after performing feature extraction on the image to be processed through a corresponding convolutional layer in at least one residual module step; and
The image processing method further comprising the step of obtaining a third extraction result according to the second extraction result and the image to be processed. 11. The method of claim 10,
The step of performing the dimension reduction processing to obtain a first processing result,
and obtaining the first processing result by performing dimensionality reduction processing according to the third extraction result. An image processing device comprising:
a classification unit configured to input the first image and the second image into the target classification network to obtain a classification result;
a prediction unit configured to obtain a prediction result by performing prediction on the change data according to the classification result;
The image processing apparatus according to claim 1, wherein the change data represents a change state obtained by processing a target object in the first image based on a region distribution in the second image. 13. The method of claim 12,
The image processing device,
performing training on the classification network to obtain a trained classification network;
The image processing apparatus according to claim 1, further comprising a training unit configured to use the trained classification network as the target classification network. 14. The method of claim 13,
The training unit,
an overlapping sub-unit configured to obtain an image to be processed after performing image overlap processing on the first image and the second image, wherein the first image and the second image are different types of image data; and
and a training sub-unit configured to obtain the trained classification network by performing training by inputting the image to be processed as a training sample into the classification network. 15. The method of claim 14,
The training unit,
performing image cropping on the first image and the second image according to the contour of the target object, respectively, to obtain cropped first sub-image data and second sub-image data; and a division sub-unit configured to use the first sub-image data and the second sub-image data as the image to be processed. 16. The method of claim 15,
The image processing apparatus according to claim 1, wherein the first sub-image data and the second sub-image data are image data of the same size. 17. The method of claim 15 or 16,
The training unit,
converting the first sub-image data and the second sub-image data into corresponding histograms, and equalizing the histogram to obtain an equalization result; and an equalization processing sub-unit configured to obtain the trained classification network by performing training on the classification network based on the equalization processing result. 17. The method of claim 15 or 16,
The training unit,
performing normalization processing on corresponding pixel points included in the first sub-image data and the second sub-image data to obtain a normalization processing result; and a normalization processing sub-unit configured to perform training on the classification network based on the normalization processing result to obtain the trained classification network. 19. The method according to any one of claims 14 to 18,
the classification network comprises at least one classification processing module;
The training sub unit,
performing feature extraction, dimension reduction processing, and global average pooling processing on the image to be processed through the at least one classification processing module to obtain a loss function; and train the classification network according to backpropagation of the loss function to obtain the trained classification network. 20. The method of claim 19,
each said classification processing module comprises at least a convolutional layer,
The training sub unit,
performing feature extraction on the image to be processed through a corresponding convolutional layer in the at least one classification processing module, and then performing dimensionality reduction processing to obtain a first processing result; after performing global average pooling processing on the first processing result, input to a fully connected layer to obtain a second processing result for indicating a prediction change state of the extracted feature; and obtain the loss function according to the second processing result and the manual labeling result. 21. The method of claim 20,
When the classification processing module is a residual module, each residual module includes a convolutional layer, a normalization layer and an activation layer,
The training sub unit,
A first extraction result obtained after performing feature extraction on the image to be processed through a corresponding convolutional layer in at least one residual module is processed through a normalization layer and an activation layer, and a second extraction result is obtained, ; and obtain a third extraction result according to the second extraction result and the image to be processed. 22. The method of claim 21,
The training sub unit,
and perform dimensionality reduction processing according to the third extraction result to obtain the first processing result. As an electronic device,
processor; and
a memory configured to store instructions executable by the processor;
12. Electronic device, characterized in that the processor is configured to perform the method according to any one of claims 1 to 11. A computer readable storage medium having computer program instructions stored thereon, comprising:
12. A computer-readable storage medium, characterized in that the computer program instructions, when executed by a processor, implement the method according to any one of claims 1 to 11. A computer program comprising computer readable code, comprising:
12. A computer program, characterized in that when the computer readable code is executed in an electronic device, the processor of the electronic device implements the method according to any one of claims 1 to 11.

Images