TW202221568A - Image recognition method, electronic device and computer readable storage medium - Google Patents

Abstract

The application discloses an image recognition method, an electronic device and a computer readable storage medium, wherein the image recognition method includes: acquiring a plurality of medical images to be recognized; extracting the style feature representation of each medical image to be recognized; classifying the style feature representation of a plurality of medical images to be recognized to obtain the scanned image category of each medical image to be recognized.

Description

Image recognition method, electronic device, computer-readable storage medium

The present invention relates to the technical field of artificial intelligence, and relates to, but is not limited to, an image recognition method, an electronic device, and a computer-readable storage medium.

Medical images such as Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) are of great clinical significance. In order to apply a medical image to the clinic, it is generally necessary to scan to obtain a medical image of at least one type of scanned image. Taking liver-related clinics as an example, scan image categories often include time-series-related pre-contrast scan, early arterial phase, late arterial phase, portal phase, delayed phase, etc. Parameter-related T1-weighted in-phase imaging, T1-weighted in-phase imaging, T2-weighted imaging, diffusion-weighted imaging, surface diffusion coefficient imaging, etc.

During the scanning process, a radiologist is usually required to identify the scanned image category of the scanned medical image to ensure that the required medical image is obtained; The image is recognized, the scanned image type of each medical image is judged, and then the image is read.

Embodiments of the present invention provide an image recognition method, an electronic device, and a computer-readable storage medium.

An embodiment of the present invention provides an image recognition method, including: acquiring a plurality of medical images to be recognized; extracting a style feature representation of each medical image to be recognized; The classification process is performed to obtain the scanned image category of each medical image to be identified.

Therefore, by acquiring a plurality of medical images to be recognized, and extracting the style feature representation of each medical image to be recognized, the style feature representation of the plurality of medical images to be recognized can be classified and processed. Considering the differences in the respective style features of multiple medical images to be identified, the accuracy of the recognized scanned image category can be improved, and because the style feature representation of multiple medical images to be identified can be classified and processed, and The scanned image category of each medical image to be recognized is obtained, so multiple scanned image categories of the medical image to be recognized can be obtained at one time, thereby improving the efficiency of image recognition. Therefore, the above solution can improve the efficiency of image recognition. Efficiency and accuracy.

In some embodiments of the present invention, classifying the style feature representations of multiple medical images to be identified, and obtaining the scanned image category of each medical image to be identified includes: classifying the style features of the multiple medical images to be identified. The first fusion processing is performed to obtain the final style feature representation; the final style feature representation is classified and processed to obtain the scanned image category of each medical image to be recognized.

Therefore, when classifying the style feature representations of multiple medical images to be recognized, the first fusion processing is performed on the style feature representations of the multiple unidentified medical images to obtain the final style feature representation, so the final style feature representation can be It represents the difference between the style feature representation of each to-be-recognized medical image and the style-feature representations of other to-be-recognized medical images, so the classification process using the final style feature representation can improve the accuracy of the recognized scanned image category.

In some embodiments of the present invention, the image recognition method further includes at least one of the following after classifying and processing the style feature representations of a plurality of medical images to be recognized to obtain the scanned image category of each medical image to be recognized: Sort the multiple medical images to be recognized according to their scanned image categories; display at least one medical image to be recognized after being sorted according to the scanned image categories; if the scanned image category of the medical image to be recognized If there is duplication, output the first warning information to remind the scanning personnel; if there is no preset scanning image category in the scanned image categories of the multiple medical images to be identified, output the second warning information to remind the scanning personnel; If the classification confidence of the scanned image category of the medical image to be recognized is smaller than the preset reliability threshold, output third warning information to prompt the scanning personnel.

Therefore, after determining the scanned image category to which each medical image to be recognized belongs, sorting at least one medical image to be recognized according to its scanned image category can improve the convenience for doctors to read images; After the image categories are sorted, at least one to-be-recognized medical image is displayed on the same screen, which can eliminate the need for doctors to check back and forth when viewing the to-be-recognized medical image, thereby improving the efficiency of the doctor's reading; when scanning the to-be-recognized medical image When the image categories are duplicated, output the first warning information to remind the scanning personnel, and when there is no preset scanning image category in the scanned image category of at least one medical image to be identified, output the second warning information to remind the scanning image category. When the classification confidence of the scanned image category of the medical image to be recognized is less than the preset reliability threshold, the scanner will output the third warning information to remind the scanner, and the image quality control can be realized during the scanning process, so as to ensure the quality of the image in the scanning process. When it is contrary to the actual situation, it can correct the error in time and avoid the second registration of the patient.

In some embodiments of the present invention, before extracting the style feature representation of each to-be-recognized medical image, the above method further includes: preprocessing each to-be-recognized medical image, wherein the preprocessing includes at least one of the following: The image size of the medical image to be recognized is adjusted to a preset size, and the image intensity of the medical image to be recognized is normalized to a preset range.

Therefore, before extracting the style feature representation, the image data of each destination area is preprocessed, and the preprocessing includes at least one of the following: adjusting the image size of the destination area to a preset size, The image intensity is normalized to a preset range, which can help to improve the accuracy of subsequent image recognition.

In some embodiments of the present invention, the image recognition method further includes: extracting the content feature representation of each to-be-recognized medical image; Lesion areas in medical images.

Therefore, by extracting the content feature representation of each to-be-recognized medical image, and performing lesion identification on the content-feature representations of a plurality of to-be-recognized medical images, the lesion area in each to-be-recognized medical image can be obtained. When the scanned image category of the medical image is to be identified, the lesion area in the medical image is determined, which can help to improve the overall reading performance, and at the same time, it can help to eliminate the interference caused by the lesion to the identification of the scanned image category, so as to improve the Image recognition accuracy.

In some embodiments of the present invention, performing lesion identification on the content feature representations of a plurality of medical images to be identified, and obtaining a lesion area in each medical image to be identified includes: representing the content feature representations of the multiple medical images to be identified. The second fusion process is performed to obtain the final content feature representation; the lesion identification is performed on the final content feature representation to obtain the lesion area in each to-be-recognized medical image.

Therefore, the second fusion processing is performed on the content feature representations of multiple medical images to be recognized to obtain the final content feature representation, which can help the final content feature representation to compensate for inconspicuous or moving lesions that may exist in a single medical image to be recognized. Problems such as artifacts generated by interference, so that the accuracy of lesion identification can be improved when using the final content feature representation for lesion identification.

In some embodiments of the present invention, the image recognition method further includes: prompting the currently displayed lesion area of the medical image to be recognized.

Therefore, by prompting the lesion area of the currently displayed medical image to be identified, the doctor's reading experience can be improved.

In some embodiments of the present invention, performing a second fusion process on the content feature representations of the plurality of medical images to be recognized includes any one of the following: performing a splicing process on the content feature representations of the plurality of medical images to be recognized; The content feature representations of the multiple medical images to be recognized are added; wherein, the final content feature representation has the same dimension as the content feature representations of the multiple medical images to be recognized.

Therefore, by performing a splicing process on the content feature representations of multiple medical images to be recognized, or performing an addition process on the content feature representations of multiple medical images to be recognized, the final content feature representation is obtained, and finally The content feature representation has the same dimension as the content feature representation of multiple medical images to be recognized, and the final content feature representation can be obtained in various ways, thereby improving the robustness of image recognition.

In some embodiments of the present invention, extracting the style feature representation of each to-be-recognized medical image separately includes: extracting the style feature representation of each to-be-recognized medical image by using the style coding sub-network of the recognition network; The style feature representation of each medical image to be recognized is classified and processed to obtain the scanned image category of each medical image to be recognized, including: using the classification processing sub-network of the recognition network to classify the styles of the medical images to be recognized. The feature representation is classified and processed to obtain the scanned image category of each medical image to be recognized; the content feature representation of each medical image to be recognized is extracted separately, including: using the content coding sub-network of the recognition network to extract each The content feature representation of the medical image to be recognized; the lesion identification is performed on the content feature representation of a plurality of medical images to be recognized, and the lesion area in each to-be-recognized medical image is obtained, including: using the region of the recognition network to segment the subnet The road performs lesion identification on the content feature representations of multiple medical images to be identified, and obtains the lesion area in each medical image to be identified.

Therefore, use the style coding sub-network of the recognition network to extract the style feature representation of each medical image to be recognized, and use the classification processing sub-network of the recognition network to classify the style feature representations of multiple medical images to be recognized Process to obtain the scanned image category of each medical image to be recognized, use the content coding sub-network of the recognition network to extract the content feature representation of each medical image to be recognized, and use the area of the recognition network to divide the sub-network Perform lesion identification on the content feature representations of multiple medical images to be identified, and obtain the lesion area in each to-be-recognized medical image, and can use the recognition network to perform extraction of style feature representations, classification processing, extraction of content feature representations, and extraction of content feature representations. It can help to improve the efficiency of image recognition.

In some embodiments of the present invention, before extracting the style feature representation of each to-be-recognized medical image, the image recognition method further includes: acquiring a plurality of sample medical images, wherein the plurality of sample medical images are marked with their real Scan the image category and the real lesion area; use the style coding sub-network to extract the sample style feature representation of each sample medical image, and use the content coding sub-network to extract the sample content feature representation of each sample medical image respectively; The classification processing sub-network is used to classify the sample style feature representation of multiple sample medical images, and the predicted scanning image category of each sample medical image is obtained, and the region segmentation sub-network is used to classify the multiple sample medical images. The feature representation of the sample content is used to identify the lesions, and obtain the predicted lesion area in each sample medical image; use the difference between the real scanned image category and the predicted scanned image category to adjust the style coding sub-network and the classification processing sub-network. network parameters, as well as network parameters for the content coding sub-network and the region segmentation sub-network using the difference between the actual lesion area and the predicted lesion area.

Therefore, while training the style coding sub-network and the classification processing sub-network, the training of the content-coding sub-network and the region segmentation sub-network can be added, so as to improve the lesion identification ability of the region segmentation sub-network. At the same time, improving the acquisition of content features related to lesions by the content coding sub-network can help to make the style coding sub-network not respond to features related to lesions, so that subsequent classification is not affected by features related to lesions, so it can be Improve the robustness of image recognition.

In some embodiments of the present invention, the image recognition method further includes: acquiring the distribution of sample data represented by the sample style features of each sample medical image; network parameters.

Therefore, in the training process, the distribution of the sample data is obtained at the same time, and the difference between the distribution of the sample data is used to adjust the network parameters of the style coding sub-network, so it can be beneficial to make the subsequent extraction of the style feature representations. independent, which can help to improve the accuracy of the recognized scanned image categories.

In some embodiments of the present invention, the image recognition method further includes: using a sample style feature representation and a content feature representation to construct a reconstructed image corresponding to the sample style feature representation; using the reconstructed image and the corresponding sample style feature To represent the differences between the belonging sample medical images, adjust the network parameters of the style coding sub-network and the content-coding sub-network.

Therefore, in the training process, a sample style feature representation and a content feature representation are used at the same time to construct a reconstructed image corresponding to the sample style feature representation, and the reconstructed image and the corresponding sample style feature are used to represent the sample medical map to which they belong. Adjust the network parameters of the style coding subnet and the content coding subnet, so that the style coding subnet can extract as complete and accurate style features as possible, and the content coding subnet can extract as much as possible the complete and accurate style features. It can extract complete and accurate style features, which can help to improve the accuracy of subsequent scanning image categories and lesion identification.

In some embodiments of the present invention, the style encoding sub-network includes: a sequentially connected downsampling layer and a global pooling layer; and/or, the content encoding sub-network includes any one of the following: a sequentially connected downsampling layer and a residual Difference blocks, sequentially connected convolutional layers and pooling layers.

Therefore, by setting the style encoding sub-network to include sequentially connected downsampling layers and global pooling layers, it can facilitate network training while simplifying the network structure; by setting the content encoding sub-network to include the following Either: sequentially connected downsampling layers and residual blocks, sequentially connected convolutional layers and pooling layers, can facilitate network training while simplifying the network structure.

The embodiment of the present invention also provides an image recognition device, including an image acquisition module, a style extraction module and a classification processing module, the image acquisition module is configured to acquire a plurality of medical images to be recognized; the style extraction module It is configured to separately extract the style feature representation of each medical image to be recognized; the classification processing module is configured to classify and process the style feature representation of a plurality of medical images to be recognized, and obtain a scanned image of each medical image to be recognized. category.

An embodiment of the present invention further provides an electronic device, including a memory and a processor coupled to each other, the processor is configured to execute program instructions stored in the memory, so as to implement any one of the above image recognition methods.

Embodiments of the present invention further provide a computer-readable storage medium, which stores program instructions, and when the program instructions are executed by a processor, any one of the above image recognition methods is implemented.

An embodiment of the present invention further provides a computer program, including computer-readable code, when the computer-readable code is executed in an electronic device, a processor in the electronic device executes any one of the image recognition methods described above. .

In the embodiment of the present invention, by acquiring a plurality of medical images to be recognized, and extracting the style feature representation of each medical image to be recognized, the style feature representation of the plurality of medical images to be recognized is classified and processed, so it is possible to During the classification process, the differences in the respective style features of the multiple medical images to be identified are considered, so as to improve the accuracy of the recognized scanned image categories, and because the style feature representation of the multiple medical images to be identified can be performed. classification processing, and obtain the scanned image category of each medical image to be recognized, so multiple scanned image categories of the medical image to be recognized can be obtained at one time, so that the efficiency of image recognition can be improved. Therefore, the embodiment of the present invention It can improve the efficiency and accuracy of image recognition.

The solutions of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

In the following description, for purposes of illustration and not limitation, specific details such as specific system structures, interfaces, techniques, etc. are set forth in order to provide a thorough understanding of the present invention.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" in this article is only an association relationship to describe associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the character "/" in this text generally indicates that the related objects are an "or" relationship. Also, "multiple" herein means two or more than two.

Please refer to FIG. 1 , which is a schematic flowchart of an embodiment of an image recognition method of the present invention. Specifically, the following steps may be included.

Step S11: Acquire a plurality of medical images to be recognized.

The medical images to be identified may include CT images and MR images, which are not limited herein. In an implementation scenario, the medical image to be recognized may be obtained by scanning regions such as the abdomen, chest, etc., which may be specifically set according to the actual application, which is not limited herein. For example, when the liver, spleen, and kidney are the organs that need diagnosis and treatment, the abdomen can be scanned to obtain the medical image to be identified; or, when the heart and lungs are the organs that need diagnosis and treatment, the chest can be scanned to obtain For medical images to be recognized, other situations can be deduced by analogy, and examples are not given here. In another implementation scenario, the scanning mode may be a flat scanning, an enhanced scanning, or the like, which is not limited herein. In yet another implementation scenario, the medical image to be recognized may be a three-dimensional image, which is not limited herein. In yet another implementation scenario, multiple medical images to be recognized may be obtained by scanning the same object.

Step S12: Extract the style feature representation of each medical image to be recognized, respectively.

The style feature represents the style used to describe the medical image to be recognized, eg, the degree of enhancement of a contrast agent injected into a blood vessel represented in the medical image to be recognized. Taking the liver as an example, the degree of enhancement of angiographic contrast agents such as veins and portal veins in the liver in the medical images to be identified in different scan image categories is different. Among them, the scan image category is the portal vein in the medical images to be identified in the early arterial stage. Not yet enhanced, the portal vein has been enhanced in the medical image to be identified whose scan image category is the late arterial stage, and the portal vein has been fully enhanced in the unidentified medical image whose scan image category is the portal venous phase, and the liver blood vessels have been Flow enhancement, the liver parenchyma has reached the peak value under the marker, the portal vein and arteries are in an enhanced state and weaker than the portal phase in the scan image category to be identified in the delayed phase, and the liver parenchyma is enhanced The state is weaker than that of the portal venous phase, and other scan image types are not listed one by one here.

In an implementation scenario, the style feature representation may be represented by a vector, and the size of the vector may be set according to the actual situation. For example, the size of the vector may be set to 8 bits, which is not limited herein.

In an implementation scenario, in order to improve the convenience of extracting the style feature representation, a recognition network can be pre-trained, and the recognition network includes a style encoding sub-network, and the style encoding sub-network of the recognition network is used to extract the A representation of style features for each medical image to be identified. In one implementation scenario, in order to simplify the network structure, the style encoding sub-network may include a downsampling layer and a global pooling layer that are connected in sequence, so that after the medical image to be recognized is subjected to downsampling processing, the global pooling layer is used to perform Pooling to get the style feature representation.

In an implementation scenario, in order to improve the accuracy of subsequent image recognition, before extracting the style feature representation of the medical image to be recognized, each medical image to be recognized may also be preprocessed, for example, preprocessing It may include resizing the image of the medical image to be recognized to a preset size (eg, 32*256*256). Alternatively, the preprocessing may further include normalizing the image intensity of the medical image to be recognized to a preset range (for example, a range of 0 to 1). In one implementation scenario, a preset grayscale cumulative distribution function may be used. The gray value corresponding to the ratio (for example, 99.9%) is used as the normalized clamp value, so that the contrast of the medical image to be recognized can be enhanced, which is beneficial to improve the accuracy of subsequent image recognition.

Step S13: Perform classification processing on the style feature representations of a plurality of medical images to be recognized, to obtain a scanned image category of each medical image to be recognized.

The scanned image category can be set according to the actual situation. For example, still taking the medical image to be identified obtained by scanning the liver as an example, the scan image category may include time-series-related pre-contrast scan, early arterial, late arterial, portal venous phase, and delayed phase; or, scan images Image categories may also include T1-weighted in-phase imaging, T1-weighted in-phase imaging, T2-weighted imaging, diffusion-weighted imaging, and surface diffusion coefficient imaging related to scanning parameters. Illustratively, the early arterial phase may indicate that the portal vein has not been enhanced, the late arterial phase may indicate that the portal vein has been enhanced, and the portal venous phase may indicate that the portal vein is sufficiently enhanced and the liver vessels have been enhanced by forward blood flow, and liver parenchyma tissue is under the marker. The peak value has been reached, and the delay period can indicate that the portal vein and arteries are in an enhanced state and weaker than the portal venous phase, and the liver parenchyma tissue is in an enhanced state and weaker than the portal venous phase. Other scan image categories are not listed here. When the medical image to be identified is a medical image obtained by scanning other organs, it can be deduced by analogy, and no examples will be given here.

Since there are differences in the respective style feature representations of the medical images to be recognized belonging to different scanned image categories, the respective style feature representations of the respective style feature representations are classified to obtain the scanned image category of each medical image to be recognized. In an implementation scenario, in order to improve the convenience of classification processing, the above-mentioned identification network further includes a classification processing sub-network, so that the classification processing sub-network can be used to classify and process the style feature representations of multiple medical images to be identified. , obtain the scanned image category of each medical image to be recognized, and then only need to classify the style feature representation by the classification processing sub-network to obtain the scanned image category to which the medical image to be recognized belongs, so the classification can be improved. Ease of handling. In an implementation scenario, in order to simplify the network structure, the classification processing sub-network may include sequentially connected fully connected layers and softmax layers, which are not limited herein.

In an implementation scenario, in order to improve the accuracy of image recognition, the style feature representations of multiple medical images to be recognized can also be subjected to a first fusion process to obtain a final style feature representation, and the final style feature representation is classified. Obtain the scanned image category of each medical image to be recognized. Exemplarily, the operation of the first fusion process may be to perform end-to-end splicing of the style feature representations of a plurality of medical images to be recognized, so as to obtain a final style feature representation; The style feature representations of the identified medical images are stacked to obtain the final style feature representation, which is not limited here. The final style feature representation obtained by performing the first fusion process on the style feature representations of multiple medical images to be recognized can represent the style feature representation of each to-be-recognized medical image and the style feature representations of other to-be-recognized medical images Therefore, using the final style feature representation for classification processing can improve the accuracy of the recognized scanned image categories.

In an implementation scenario, in order to improve the reading efficiency and the accuracy of scanning image category recognition, the content feature representation of each to-be-recognized medical image can also be extracted, and the content feature representation of multiple to-be-recognized medical images can be represented. Perform lesion identification to obtain the lesion area in each medical image to be identified, so the lesion area can be determined while obtaining the scanned image category of each medical image to be identified, so it can be beneficial to improve the overall reading. At the same time, it can help to eliminate the interference caused by the lesions to the classification recognition of the scanned image, so that the accuracy of the image recognition can be improved. The content feature represents the content in the medical image to be recognized, for example, the anatomical features of an organ in the medical image to be recognized. Still taking the liver as an example, the content feature representation can describe the physiological positional relationship between the liver and its adjacent organs (such as spleen, kidney), the shape characteristics of the liver, the texture (such as soft, hard, etc.) characteristics, the composition (such as spleen, kidney, etc.) characteristics. , water-containing, fat-containing, etc.) characteristics, etc., are not limited here. In addition, the lesions may include tumors, thrombus, nodules, etc., which may be set according to actual conditions, and are not limited herein.

In an implementation scenario, in order to improve the convenience of extracting the content feature representation, the identification network may further include a content encoding sub-network, so that each medical image to be identified can be extracted separately by using the content encoding sub-network of the identification network. Image content feature representation. Exemplarily, in order to simplify the network structure, the content coding sub-network can use sequentially connected downsampling layers and residual blocks (resblocks), and the number of residual blocks can be set according to the actual situation. network depth, which can improve the depth of the extracted content feature representation; alternatively, the content coding sub-network can also use sequentially connected convolutional layers and pooling layers, and the number of groups of convolutional layers and pooling layers can be determined according to the actual situation. The setting, for example, can adopt a set of sequentially connected convolution layers and pooling layers, two sets of sequentially connected convolution layers and pooling layers, three sets of sequentially connected convolution layers and pooling layers, etc., which are not limited here.

In another implementation scenario, in order to improve the convenience of lesion identification, the identification network may further include a region segmentation sub-network, so that the content of multiple medical images to be identified is analyzed by using the region segmentation sub-network of the identification network. The feature representation is used for lesion identification, and the lesion area in each medical image to be identified is obtained. Exemplarily, the area dividing subnet may adopt Unet, Vnet, etc., which is not limited herein. In addition, the lesion area may include an area surrounding the lesion, for example, the outline of the lesion, etc., which is not limited herein.

In yet another implementation scenario, the content feature representation may be represented by a tensor, for example, the content feature representation may be represented by a low-resolution tensor, which is not limited herein.

In yet another implementation scenario, in order to improve the accuracy of lesion identification, a second fusion process may be performed on the content feature representations of multiple medical images to be identified to obtain a final content feature representation, and the final content feature representation is used for lesion identification. , to obtain the lesion area in each medical image to be identified, so that the final content feature representation can help to compensate for the problems of inconspicuous lesions or artifacts caused by motion interference that may exist in a single medical image to be identified. The final content feature representation can improve the accuracy of lesion identification when performing lesion identification. Exemplarily, the second fusion process may specifically be performing a concatenation process on the content feature representations of a plurality of medical images to be recognized to implement a fusion process on the content feature representations. In some embodiments, after the concatenation process is performed, , the final content feature representation can be obtained through several simple convolutional layers without pooling; here, the splicing of content features can be regarded as a concatenation operation of tensors; The content feature representation is added (add) to realize the fusion processing of the content feature representation, which can be regarded as the summation operation of tensors; or, the content feature representation of multiple medical images to be recognized can also be stacked, Then use a convolution kernel such as 1*1 to perform a convolution operation on the stacked content feature representations to realize the fusion processing of the content feature representations; or, the content feature representations of multiple medical images to be recognized can also be weighted. , which is not limited here. In addition, the final content feature representation and the content feature representation of the plurality of medical images to be recognized have the same dimension.

In yet another implementation scenario, in order to improve the doctor's experience, after obtaining the lesion area in the medical image to be recognized, the currently displayed lesion area of the medical image to be recognized may be prompted. For example, preset lines (eg, bold lines, dot-dash lines, double solid lines, etc.) and/or preset colors (eg, yellow, red, green, etc.) can be used to represent the lesion area; or, preset lines can also be used. Symbols (for example, arrows pointing to the lesion area, etc.) represent the lesion area, which can be set according to the actual situation, which is not limited here.

In yet another implementation scenario, the above-mentioned trained identification network can be set up in image post-processing workstations, imaging workstations, computer-assisted image reading systems, remote medical diagnosis scenarios, cloud platform-assisted smart diagnosis scenarios, etc., so as to realize Automatic recognition of medical images to be recognized to improve recognition efficiency.

In an implementation scenario, at least one medical image to be recognized is obtained by scanning the same object, so in order to facilitate the doctor to read the image, after obtaining the scanned image category to which each medical image to be recognized belongs, at least one medical image can also be scanned. The medical images to be identified are sorted according to their scan image categories, for example, T1-weighted inverse imaging, T1-weighted in-phase imaging, pre-contrast scan, early arterial, late arterial, portal venous phase, delayed phase, and T2-weighted imaging , Diffusion Weighted Imaging, and Surface Diffusion Coefficient Imaging in the preset order. In addition, the preset order can also be set according to the doctor's habit, which is not limited here, so as to improve the convenience of the doctor's reading.

In another implementation scenario, in order to further improve the convenience of image reading, at least one medical image to be recognized sorted according to the scan image category can also be displayed on the same screen. For example, the number of medical images to be recognized is 5, the medical images to be recognized can be displayed in the 5 display windows respectively. Therefore, it is possible to reduce the time for doctors to review multiple medical images to be recognized and compare them back and forth, and improve the efficiency of reading images.

In yet another implementation scenario, at least one medical image to be recognized is obtained by scanning the same object, so in order to perform quality control during the scanning process, after obtaining the scanned image category to which each medical image to be recognized belongs, the It can be judged whether the scanned image categories of the medical images to be identified are duplicated, and when duplicates exist, first warning information is output to prompt the scanning personnel. For example, if there are two medical images to be identified with both scanned image categories as "delayed period", it can be considered that the scanning quality is not compliant during the scanning process. Therefore, in order to remind the scanning personnel, the first warning information can be output , exemplarily, the cause of the warning (eg, there are medical images to be identified with duplicate types of scanned images, etc.) can be output. Alternatively, after obtaining the scanned image category to which each to-be-recognized medical image belongs, it can also be determined that there is no preset scanned image category in the scanned image category of at least one to-be-identified medical image, and if there is no preset scanned image category. When scanning the image category, output the second warning information to remind the scanning personnel. For example, the preset scanned image category is "portal phase". If there is no image with the scanned image category of "portal phase" in at least one of the medical images to be identified, it can be considered that there is an inconsistency in scan quality during the scanning process. In order to prompt the scanning personnel, the second warning information may be output, for example, the warning reason (for example, there is no portal phase image in the to-be-recognized medical image, etc.) may be output. Alternatively, after obtaining the scanned image category to which each medical image to be recognized belongs, it is also possible to determine whether the classification confidence of the scanned image category of the medical image to be recognized is less than a preset reliability threshold, and the When the classification confidence of the scanned image category of the image is less than the preset confidence threshold, the third warning information is output to remind the scanning personnel. Exemplarily, the classification confidence may be predicted by the classification processing sub-network during the classification processing. For example, when the classification processing sub-network performs the classification processing, it predicts the scanned image category and the corresponding classification confidence of each medical image to be recognized. The classification confidence (for example, 20%) of the image is lower than the preset confidence threshold (for example, 90%), it can be considered that the image quality of the medical image to be recognized whose scan image category is "portal phase" is not suitable. Therefore, in order to prompt the scanning personnel, the third warning information can be output, for example, the warning reason can be output (for example, the medical image to be recognized may have a problem of poor scanning quality). Therefore, the image quality control can be realized during the scanning process, so that when it is contrary to the actual situation, the error can be corrected in time and the second registration of the patient can be avoided.

In the above scheme, by acquiring a plurality of medical images to be recognized, and extracting the style feature representation of each medical image to be recognized, the style feature representation of the plurality of medical images to be recognized is classified and processed, so it can be used in the classification process. , considering the differences in the respective style features of the multiple medical images to be identified, thereby improving the accuracy of the recognized scanned image category, and because the style feature representation of the multiple medical images to be identified can be classified and processed, and obtain the scanned image category of each medical image to be recognized, so multiple scanned image categories of the medical image to be recognized can be obtained at one time, thereby improving the efficiency of image recognition. Therefore, the above solution can improve image recognition. efficiency and accuracy.

Please refer to FIG. 2 , which is a schematic flowchart of an embodiment of training a recognition network. Specifically, the identification network includes the content encoding sub-network, the style encoding sub-network, the classification processing sub-network and the area segmentation sub-network in the foregoing embodiments, and the specific process is as follows.

Step S21 : acquiring multiple sample medical images, wherein the multiple sample medical images are marked with their real scanned image categories and real lesion areas.

The multiple sample medical images used in a training process can be obtained by scanning the same object. For example, the sample medical image used in a certain training may be obtained by scanning object A, and the sample medical image used in another training may be obtained by scanning object B.

The sample medical images may also include CT images and MR images, which are not limited herein. For details, reference may be made to the medical images to be recognized in the foregoing embodiments, and details are not described herein again.

The real scanned image category and real lesion area marked by the sample medical image can be marked by clinicians, radiologists and other personnel with medical imaging knowledge. The scan image category can be set according to the actual situation. For example, if the sample medical image is obtained by scanning the liver, the scan image category can specifically include the timing-related pre-contrast scan, early arterial, late arterial, and portal scans. Pulse phase, delay phase; or, the scan image category may also include T1-weighted inverse imaging, T1-weighted in-phase imaging, T2-weighted imaging, diffusion-weighted imaging, and surface diffusion coefficient imaging related to scanning parameters. For details, please refer to the foregoing embodiments. The relevant steps in are not repeated here. When the sample medical image is obtained by scanning other organs, it can be deduced by analogy, and examples will not be given here. The real lesion area may be marked with polygons, for example, the contour of the lesion may be marked with polygons, etc., which is not limited herein.

Please refer to FIG. 3 . FIG. 3 is a schematic state diagram of an embodiment of a training identification network. As shown in Figure 3, in a training process, sample medical image 1, sample medical image 2, sample medical image 3, ..., sample medical image n can be obtained, where the value of n can be based on the actual situation For example, when the scan image category includes time-related pre-contrast scan, early arterial, late arterial, portal venous phase, and delayed phase, the value of n can be set to an integer less than or equal to 5, for example, 5 , 4, 3, etc.; or, when the scanned image category includes T1-weighted inverse imaging, T1-weighted in-phase imaging, T2-weighted imaging, diffusion-weighted imaging, and surface diffusion coefficient imaging related to scanning parameters, the value of n can be set is an integer less than or equal to 5, for example, 5, 4, 3, etc.; or, when the scan image category includes both T1-weighted inverse imaging, T1-weighted in-phase imaging, T2-weighted imaging, and diffusion-weighted imaging related to scanning parameters , surface diffusion coefficient imaging, including timing-related pre-contrast scan, early arterial, late arterial, portal venous phase, and delayed phase, the value of n can be set to an integer less than or equal to 10, for example, 10, 9, 8, etc., which can be set according to the actual situation, which is not limited here.

Step S22 : using the style coding sub-network to extract the sample style feature representation of each sample medical image, and using the content coding sub-network to separately extract the sample content feature representation of each sample medical image.

For the network structures of the style encoding sub-network and the content encoding sub-network, reference may be made to the relevant steps in the foregoing embodiments, which will not be repeated here. Please refer to Figure 3 in conjunction with the sample medical image 1, the sample medical image 2, the sample medical image 3, ..., and the sample medical image n after the style feature extraction is performed by the style coding sub-network, the sample styles can be obtained respectively. Feature representation 1, sample style feature representation 2, sample style feature representation 3, ..., sample style feature representation n; after content feature extraction by the content coding sub-network, sample content feature representation 1, sample content feature representation can be obtained respectively 2. Sample content feature representation 3, ..., sample content feature representation n. For the sample style feature representation and the sample content feature representation, reference may be made to the style feature representation and the content feature representation in the foregoing embodiments, which will not be repeated here.

Step S23: Use the classification processing sub-network to classify and process the sample style feature representations of the multiple sample medical images to obtain the predicted scanning image category of each sample medical image, and use the region segmentation sub-network to classify the multiple samples. The feature representation of the sample content of the medical image is used for lesion identification, and the predicted lesion area in each sample medical image is obtained.

In an implementation scenario, the sample style feature representations of multiple sample medical images can be spliced to obtain the final sample style feature representation, and the final sample style feature representation can be classified by using the classification processing sub-network to obtain each sample style feature representation. The predicted scan image category of this medical image, so that the final sample style feature representation can represent the difference between the sample style feature representation of each sample medical image and the sample style feature representation of other sample medical images, so the classification process is used. When the sub-network performs classification processing on the final sample style feature representation, the accuracy of the classification processing can be improved.

As shown in FIG. 3 , when the sample medical image 1, sample medical image 2, sample medical image 3, . The sample style features of the above-mentioned sample medical images can be spliced (or other processing methods such as stacking processing, for details, please refer to the aforementioned disclosed embodiments, which will not be repeated here) to obtain the final sample style feature representation, so as to use the classification processor. The network performs classification processing on the final sample style feature representation, and obtains the predicted scanning image categories of sample medical image 1, sample medical image 2, sample medical image 3, ..., and sample medical image n.

In one implementation scenario, the sample content feature representations of multiple sample medical images can be fused to obtain the final sample content feature representation, and the region segmentation sub-network can be used to identify lesions on the final sample content feature representation to obtain each sample content feature representation. The predicted lesion area in this medical image can help to make the final sample content feature representation to compensate for problems such as inconspicuous lesions or artifacts caused by motion interference that may exist in a single sample medical image, so as to use the area segmentation subnet. When the path is used to identify the final sample content feature representation, the accuracy of the lesion identification can be improved. Exemplarily, the operation of the fusion processing may include any one of the following: performing a splicing process on the sample content feature representations of multiple sample medical images, and performing a splicing process on the sample content feature representations of the multiple sample medical images by using at least one convolution layer. Feature extraction, and the final sample content feature representation has the same dimension as the content feature representation of multiple sample medical images. For details, reference may be made to the relevant steps in the foregoing embodiments, which will not be repeated here.

As shown in FIG. 3 , when using the region segmentation sub-network to perform lesion identification on the sample content feature representation of sample medical image 1, sample medical image 2, sample medical image 3, . . . , sample medical image n, The sample content feature representation of the above-mentioned sample medical images can be subjected to fusion processing (eg, splicing processing, addition processing, etc., for details, please refer to the aforementioned disclosed embodiments, which will not be repeated here) to obtain the final sample content feature representation. The region segmentation sub-network performs lesion identification on the final sample content feature representation, and obtains the predicted lesion regions in the sample medical image 1, the sample medical image 2, the sample medical image 3, ... and the sample medical image n.

Step S24 : Adjust the network parameters of the style coding sub-network and the classification processing sub-network by using the difference between the real scanned image category and the predicted scanned image category, and adjust the content coding by using the difference between the real lesion area and the predicted lesion area Network parameters for subnetworks and zone-split subnetworks.

In an implementation scenario, the real scanned image category and the predicted scanned image category can be used to calculate the first loss value of the style encoding sub-network and the classification processing sub-network, and use the first loss value to adjust the style encoding sub-network Roads and classes handle network parameters for subnetworks. In an implementation scenario, a cross entropy loss (cross entropy loss) or a softmax logistic loss (logistic softmax loss) or the like may be used to calculate the first loss value, which is not limited herein.

（1） 上式中，

表示以dice係數損失計算得到的第二損失值，

表示真實病灶區域，

表示預測病灶區域，

表示真實病灶區域與預測病灶區域的交集。 In an implementation scenario, the actual lesion area and the predicted lesion area can be used to calculate the second loss value of the content coding sub-network and the region segmentation sub-network, and the second loss value can be used to adjust the content coding sub-network and the region segmentation Network parameters for the subnet. In an implementation scenario, a binary cross-entropy loss (binary cross-entropy loss) or a dice coefficient loss may be used to calculate the second loss value, which is not limited herein. Exemplarily, the dice coefficient loss is an ensemble similarity measure function, which is usually used to calculate the similarity between two samples (ranging from 0 to 1). Specifically, formula (1) can be used to calculate:

(1) In the above formula,

represents the second loss value calculated by the dice coefficient loss,

represents the real lesion area,

represents the predicted lesion area,

Represents the intersection of the real lesion area and the predicted lesion area.

（2） 上式中，

和

分別表示兩個樣本風格特徵表示的樣本資料分佈情況，

表示其中一個樣本風格特徵表示的概率分佈期望，

和

分別表示兩個樣本風格特徵表示中的元素 x在各自樣本資料分佈情況中的分佈概率。在一個實施場景中，可以採用高斯分佈函數獲取樣本風格特徵表示的樣本資料分佈情況，從而通過上述訓練，能夠使風格編碼子網路後續提取得到的風格特徵表示為中心相同且各向異性的高斯分佈，進而故能夠有利於提高識別到的掃描圖像類別的準確性。 In an implementation scenario, in order to improve the accuracy of scanning image category recognition, the distribution of sample data represented by the sample style features of each sample medical image can also be obtained during the training process, and use the difference between the distribution of sample data. Adjust the network parameters of the style coding sub-network, which can help to make the subsequent extracted style feature representations independent of each other, so it can help to improve the accuracy of the identified scanned image categories. Exemplarily, in the training process, the KL divergence can be used to measure the difference between the distributions of the sample data, and use it as the third loss value, so as to constrain the distribution of the style feature representation. KL divergence (Kullback–Leibler divergence), also known as relative entropy (relative entropy), is a method to describe the difference between two probability distributions P and Q, which can be calculated by formula (2):

(2) In the above formula,

and

respectively represent the distribution of the sample data represented by the two sample style features,

represents the probability distribution expectation represented by one of the sample style features,

and

Respectively represent the distribution probability of the element x in the two sample style feature representations in the distribution of the respective sample data. In an implementation scenario, a Gaussian distribution function can be used to obtain the distribution of sample data represented by the sample style features, so that through the above training, the style features obtained by the style coding sub-network subsequently extracted can be represented as Gaussians with the same center and anisotropy distribution, and thus can help to improve the accuracy of the identified scanned image categories.

In an implementation scenario, in order to enable the style encoding sub-network to extract as complete and accurate style features as possible, and the content encoding sub-network to extract as complete and accurate style features as possible, a sample style feature representation can also be used. With the same sample content feature representation, a reconstructed image corresponding to the sample style feature representation is constructed, and the difference between the reconstructed image and the corresponding sample style feature representation belongs to the sample medical image, and the style coding sub-network and the corresponding sample medical image are adjusted. The network parameters of the content coding subnet can make the reconstructed image and the corresponding sample medical image as similar as possible during the training process, so that the style coding subnet can extract the complete and accurate style features as much as possible. , and the content coding sub-network can extract complete and accurate style features as much as possible, so it can help improve the accuracy of subsequent scanned image categories and lesion identification. Exemplarily, the reconstructed image can be compared with the corresponding sample style features The difference between the belonging sample medical images is represented as the fourth loss value. In an implementation scenario, the sample style feature representation and the sample content feature representation of each sample medical image can be used to obtain an intra-domain reconstructed image of the corresponding sample medical image, and each sample medical image and its in-domain reconstructed image can be obtained. Reconstruct the difference of the image, adjust the network parameters of the style coding sub-network and the content coding sub-network, so as to ensure that the decomposed sample content feature representation and sample style feature representation can reconstruct themselves stably and prevent midway variation. In another implementation scenario, the sample style feature representation of each sample medical image and the sample content feature representation (or final sample content feature representation) of any other sample medical image can also be used to obtain the corresponding sample medical image Cross-domain reconstructed images, and use the differences between each sample medical image and its cross-domain reconstructed images to adjust the network parameters of the style coding sub-network and the content coding sub-network, so as to ensure that the extracted content feature representation is correct. The underlying feature set that truly coincides with each other between medical images. Exemplarily, in the training process, the generator can be used to achieve reconstruction, and the discriminator can be used to identify whether the reconstructed image is a real sample medical image or a reconstructed image, so Generative Adversarial Networks loss (GAN) can be used. loss) to measure the loss value of the above-mentioned cross-domain reconstruction, and L1 norm loss to measure the loss value of the above-mentioned intra-domain reconstruction, which will not be repeated here.

As shown in Fig. 3, when performing image reconstruction, the sample content features corresponding to sample medical image 1, sample medical image 2, sample medical image 3, ... and sample medical image n can be represented as 1, The sample content feature representation 2, the sample content feature representation 3, ..., the sample content feature representation n are fused to obtain the final sample content feature representation, and the sample medical image 1, the sample medical image 2, and the sample medical image are respectively 3, ..., the sample style feature representation corresponding to the sample medical image n 1, the sample style feature representation 2, the sample style feature representation 3, ..., the sample style feature representation n and the final sample content feature representation are reconstructed to obtain a reconstructed image Like 1, reconstructed image 2, reconstructed image 3, ..., reconstructed image n, so as to achieve intra-domain reconstruction, and use the L1 norm loss to measure the loss value of the above-mentioned intra-domain reconstruction. In addition, the sample style feature representation 1 corresponding to the sample medical image 1 and the sample content feature representation corresponding to other sample medical images (ie, sample medical images 2~n) can also be used for cross-domain reconstruction, and other sample medical images are represented by And so on, and adopt the generative adversarial loss (GAN loss) to measure the loss value of the above cross-domain reconstruction.

In an implementation scenario, the above-mentioned first loss value, second loss value, third loss value and fourth loss value may also be calculated at the same time, and the network parameters of the identification network may be adjusted according to these loss values, so as to improve the The degree of acquisition of content features related to lesions by the content coding sub-network makes the style coding sub-network not respond to features related to lesions, improves the robustness of image recognition, and makes the subsequent extracted style feature representations interact with each other. It is independent and enables the style encoding sub-network to extract complete and accurate style feature representation, and the content encoding sub-network can extract complete and accurate content feature representation, thereby improving the accuracy of image recognition.

Different from the foregoing embodiments, while training the style coding sub-network and the classification processing sub-network, the training of the content coding sub-network and the region segmentation sub-network is added, so that the performance of the region segmentation sub-network can be improved. At the same time, it can improve the acquisition of content features related to lesions by the content coding subnet, which can help make the style coding subnet not respond to the features related to lesions, so that the subsequent classification will not be affected by the features related to lesions. Therefore, the robustness of image recognition can be improved.

Please refer to FIG. 4 , which is a block diagram of an embodiment of an image recognition apparatus 40 of the present invention. The image recognition device 40 includes an image acquisition module 41, a style extraction module 42 and a classification processing module 43. The image acquisition module 41 is configured to acquire a plurality of medical images to be recognized; the style extraction module 42 is configured to separately Extract the style feature representation of each to-be-recognized medical image; the classification processing module 43 is configured to classify and process the style-feature representations of a plurality of to-be-recognized medical images to obtain the scanned image category of each to-be-recognized medical image.

In the above scheme, by acquiring a plurality of medical images to be recognized, and extracting the style feature representation of each medical image to be recognized, the style feature representation of the plurality of medical images to be recognized is classified and processed, so it can be used in the classification process. , considering the differences in the respective style features of the multiple medical images to be identified, thereby improving the accuracy of the recognized scanned image category, and because the style feature representation of the multiple medical images to be identified can be classified and processed, and obtain the scanned image category of each medical image to be recognized, so multiple scanned image categories of the medical image to be recognized can be obtained at one time, thereby improving the efficiency of image recognition. Therefore, the above solution can improve image recognition. efficiency and accuracy.

In some embodiments, the classification processing module 43 includes a first fusion processing sub-module, configured to perform a first fusion processing on the style feature representations of a plurality of to-be-recognized medical images to obtain a final style feature representation; the classification processing module 43 includes a classification processing sub-module configured to perform classification processing on the final style feature representation to obtain the scanned image category of each medical image to be identified.

Different from the foregoing embodiments, when the style feature representations of multiple medical images to be recognized are classified and processed, the style feature representations of the multiple unidentified medical images are subjected to a first fusion process to obtain the final style feature representation, so the final style feature representation is obtained. The style feature representation can represent the difference between the style feature representation of each medical image to be recognized and the style feature representation of other to-be-recognized medical images, so using the final style feature representation for classification processing can improve the classification of scanned images obtained by recognition. accuracy.

In some embodiments, the image recognition device 40 further includes at least one of an image exclusion module, an image display module, a first early warning module, a second early warning module and a third early warning module.

The image exclusion module is configured to sort a plurality of medical images to be identified according to their scanned image categories; the image display module is configured to sort at least one to-be-identified image according to the scanned image categories. Identifying medical images and displaying them on the same screen; the first warning module is configured to output first warning information to prompt the scanning personnel when there is a repetition of the scanned image categories of the medical images to be identified; the second warning The module is configured to output the second early warning information to prompt the scanning personnel when the preset scanning image category does not exist in the scanned image categories of the medical images to be identified; the third early warning module is configured as When the classification confidence of the scanned image category of the to-be-recognized medical image is less than the preset confidence threshold, output third warning information to prompt the scanning personnel.

Different from the foregoing embodiments, after the scanning image category to which each medical image to be recognized belongs is determined, the at least one medical image to be recognized is sorted according to its scanning image category, which can improve the convenience of doctor reading. ; Display at least one to-be-recognized medical image sorted according to the scanned image category on the same screen, which can save the doctor from checking back and forth when reading the to-be-recognized medical image, thereby improving the efficiency of the doctor's image reading; When the scanned image categories of the images are repeated, output the first warning information to prompt the scanning personnel, and output the second warning when there is no preset scanned image category in the scanned image categories of at least one of the medical images to be identified. Information to remind the scanning personnel, to achieve image quality control during the scanning process, to correct errors in time when it is inconsistent with the actual situation, and avoid the second registration of the patient.

In some embodiments, the image recognition apparatus 40 further includes a preprocessing module configured to preprocess each medical image to be recognized, wherein the preprocessing includes at least one of the following: The size is adjusted to a preset size, and the image intensity of the medical image to be recognized is normalized to a preset range.

Different from the foregoing embodiments, before extracting the style feature representation, preprocessing is performed on the image data of each destination area, and the preprocessing includes at least one of the following: adjusting the image size of the destination area to a preset size, The image intensity of the destination area is normalized to a preset range, which can help improve the accuracy of subsequent image recognition.

In some embodiments, the image recognition apparatus 40 further includes a content extraction module, configured to extract the content feature representation of each medical image to be recognized, respectively; the image recognition apparatus 40 further includes a lesion identification module, configured to multiple The content features of each medical image to be identified represent the lesion identification, and the lesion area in each medical image to be identified is obtained.

Different from the foregoing embodiments, by extracting the content feature representation of each to-be-recognized medical image, and performing lesion identification on the content-feature representations of a plurality of to-be-recognized medical images, the lesion area in each to-be-recognized medical image is obtained, It is possible to determine the lesion area while obtaining the scanned image category of each medical image to be identified, so it can help improve the overall reading performance, and at the same time, it can help eliminate the interference caused by lesions to the scanning image category recognition. , which can improve the accuracy of image recognition.

In some embodiments, the lesion identification module includes a second fusion processing sub-module configured to perform a second fusion process on the content feature representations of multiple medical images to be identified to obtain a final content feature representation; the lesion identification module includes The lesion identification sub-module is configured to perform lesion identification on the final content feature representation to obtain the lesion area in each medical image to be identified.

Different from the foregoing embodiments, the second fusion processing is performed on the content feature representations of a plurality of medical images to be recognized to obtain a final content feature representation, which can help the final content feature representation to compensate for possible lesions in a single to-be-recognized medical image. Therefore, the accuracy of lesion identification can be improved when using the final content feature representation for lesion identification.

In some embodiments, the lesion identification module further includes a lesion prompting sub-module configured to prompt the lesion area of the currently displayed medical image to be identified.

Different from the foregoing embodiments, by prompting the lesion area of the currently displayed medical image to be identified, the doctor's reading experience can be improved.

In some embodiments, the second fusion processing sub-module is specifically configured to perform any one of the following: perform splicing processing on the content feature representations of a plurality of to-be-recognized medical images; Addition processing is performed; wherein, the final content feature representation and the content feature representations of multiple medical images to be identified have the same dimension.

Different from the foregoing embodiments, the final content feature is obtained by splicing the content feature representations of a plurality of medical images to be recognized, or performing an addition process on the content feature representations of a plurality of medical images to be recognized. and the final content feature representation has the same dimension as the content feature representation of multiple medical images to be recognized, the final content feature representation can be obtained in various ways, thereby improving the robustness of image recognition.

In some embodiments, the style extraction module 42 is specifically configured to use the style coding sub-network of the recognition network to extract the style feature representation of each medical image to be recognized; the classification processing module 43 is specifically configured to use the recognition network The classification processing sub-network performs classification processing on the style feature representation of a plurality of medical images to be recognized, and obtains the scanned image category of each medical image to be recognized; the content extraction module is specifically configured to use the content coding of the recognition network. The sub-network separately extracts the content feature representation of each medical image to be identified; the lesion identification module is specifically configured to use the region segmentation sub-network of the identification network to perform lesion identification on the content feature representation of a plurality of medical images to be identified, Obtain the lesion area in each medical image to be identified.

Different from the foregoing embodiments, the style feature representation of each medical image to be recognized is extracted by the style coding sub-network of the recognition network, and the style of the medical images to be recognized is analyzed by the classification processing sub-network of the recognition network. The feature representation is classified to obtain the scanned image category of each medical image to be recognized, and the content feature representation of each medical image to be recognized is extracted by using the content coding sub-network of the recognition network. The segmentation sub-network performs lesion identification on the content feature representation of multiple medical images to be identified, and obtains the lesion area in each medical image to be identified. The identification network can be used to perform extraction of style feature representation, classification processing, and content feature representation. It can help to improve the efficiency of image recognition because of tasks such as representation extraction and lesion recognition.

In some embodiments, the image recognition device 40 further includes a sample acquisition module configured to acquire multiple sample medical images, wherein the multiple sample medical images are marked with their real scanned image categories and real lesion areas; Fig. The image recognition device 40 also includes a feature extraction module configured to extract the sample style feature representation of each sample medical image by using the style coding sub-network, and extract the sample of each sample medical image by using the content coding sub-network. Content feature representation; the image recognition device 40 further includes a recognition processing module, configured to use a classification processing sub-network to classify and process the sample style feature representations of a plurality of sample medical images to obtain a predictive scan of each sample medical image image category, and use the region segmentation sub-network to perform lesion identification on the sample content feature representation of multiple sample medical images to obtain the predicted lesion area in each sample medical image; the image recognition device 40 also includes a first adjustment A module configured to adjust the network parameters of the style coding subnet and the classification processing subnet using the difference between the actual scan category and the predicted scan category, and to adjust the content coding subnet using the difference between the actual lesion area and the predicted lesion area Network parameters for road and area split subnets.

Different from the foregoing embodiments, while training the style coding sub-network and the classification processing sub-network, the training of the content coding sub-network and the region segmentation sub-network is added, so that the performance of the region segmentation sub-network can be improved. At the same time, it can improve the acquisition of content features related to lesions by the content coding subnet, which can help make the style coding subnet not respond to the features related to lesions, so that the subsequent classification will not be affected by the features related to lesions. Therefore, the robustness of image recognition can be improved.

In some embodiments, the image recognition apparatus 40 further includes a distribution acquisition module configured to acquire the distribution of sample data represented by the sample style features of each sample medical image; the image recognition apparatus 40 further includes a second adjustment module , which is configured to adjust the network parameters of the style coding sub-network by taking advantage of the differences between the distributions of the sample data.

Different from the previous embodiment, in the training process, the distribution of sample data is obtained at the same time, and the difference between the distribution of sample data is used to adjust the network parameters of the style coding sub-network, so it can be beneficial to the subsequent extraction of style features. The representations are independent of each other, which can help improve the accuracy of the scanned image categories identified.

In some embodiments, the image recognition apparatus 40 further includes an image reconstruction module configured to use a sample style feature representation and a content feature representation to construct a reconstructed image corresponding to the sample style feature representation; the image recognition apparatus 40 also includes a third adjustment module configured to adjust the network parameters of the style coding sub-network and the content coding sub-network by using the difference between the reconstructed image and the corresponding sample medical image to which the sample style feature representation belongs.

Different from the foregoing embodiments, in the training process, a sample style feature representation and a content feature representation are used to construct a reconstructed image corresponding to the sample style feature representation, and the reconstructed image and the corresponding sample style feature are used to represent the The difference between the sample medical images, adjust the network parameters of the style coding subnet and the content coding subnet, so that the style coding subnet can extract as complete and accurate style features as possible, while the content coding subnet can extract the complete and accurate style features as much as possible. The network extracts complete and accurate style features as much as possible, which can help to improve the classification of subsequent scanned images and the accuracy of lesion identification.

In some embodiments, the style encoding sub-network includes: sequentially connected downsampling layers and global pooling layers; and/or, the content encoding sub-network includes any of the following: sequentially connected downsampling layers and residual blocks , sequentially connected convolutional and pooling layers.

Different from the foregoing embodiments, by setting the style encoding sub-network to include a sequentially connected downsampling layer and a global pooling layer, it can facilitate network training while simplifying the network structure; Setting to include any of the following: sequentially connected downsampling layers and residual blocks, sequentially connected convolutional layers and pooling layers, can facilitate network training while simplifying the network structure.

Please refer to FIG. 5 , which is a block diagram illustrating an embodiment of an electronic device 50 of the present invention. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other, and the processor 52 is configured to execute program instructions stored in the memory 51 to implement the steps of any of the above image recognition method embodiments. In an implementation scenario, the electronic device 50 may include, but is not limited to, a microcomputer and a server. In addition, the electronic device 50 may also include mobile devices such as a notebook computer and a tablet computer, which are not limited herein.

Specifically, the processor 52 is configured to control itself and the memory 51 to implement the steps of any of the image recognition method embodiments described above. The processor 52 may also be referred to as a central processing unit (Central Processing Unit, CPU). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA) Or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. Additionally, the processor 52 may be commonly implemented by an integrated circuit die.

The above solution can improve the efficiency and accuracy of image recognition.

Please refer to FIG. 6 , which is a block diagram illustrating an embodiment of a computer-readable storage medium 60 according to the present invention. The computer-readable storage medium 60 stores program instructions 601 that can be executed by the processor, and the program instructions 601 are used to implement any of the above-mentioned image recognition methods.

The above solution can improve the efficiency and accuracy of image recognition.

Correspondingly, an embodiment of the present invention also provides a computer program, including computer-readable code, when the computer-readable code is executed in an electronic device, a processor in the electronic device executes any one of the above diagrams. like identification methods.

In the several embodiments provided by the present invention, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the device implementations described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other divisions. For example, units or elements may be combined or integrated. to another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

Units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed over network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this implementation manner.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, Several instructions are included to cause a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or CD, etc. medium.

Industrial Applicability The embodiment of the present invention discloses an image recognition method, an electronic device, and a computer-readable storage medium. The image recognition method includes: acquiring a plurality of medical images to be recognized; extracting a style feature of each medical image to be recognized separately Representation; classifying and processing the style feature representations of the plurality of medical images to be recognized, to obtain a scanned image category of each of the medical images to be recognized. The above solution can improve the efficiency and accuracy of image recognition.

40: Image recognition device 41: Image acquisition module 42: Style Extraction Module 43: Classification processing module 50: Electronics 51: Memory 52: Processor 60: Computer-readable storage medium 601: Program command S11~S13, S21~S24: Steps

1 is a schematic flowchart of an embodiment of an image recognition method of the present invention; 2 is a schematic flow chart of an embodiment of a training identification network; 3 is a schematic state diagram of an embodiment of a training recognition network; 4 is a schematic block diagram of an embodiment of an image recognition apparatus of the present invention; 5 is a schematic block diagram of an embodiment of an electronic device of the present invention; FIG. 6 is a schematic block diagram of an embodiment of a computer-readable storage medium of the present invention.

S11~S13: Steps

Claims (14)

An image recognition method, comprising: Obtain multiple medical images to be recognized; Extracting the style feature representation of each of the medical images to be identified, respectively; The style feature representation of the plurality of medical images to be recognized is classified and processed to obtain a scanned image category of each of the medical images to be recognized. The image recognition method according to claim 1, wherein, by performing classification processing on the style feature representations of the plurality of medical images to be recognized, obtaining the scanned image category of each of the medical images to be recognized includes the following steps: : Perform a first fusion process on the style feature representations of the plurality of medical images to be recognized to obtain a final style feature representation; The final style feature representation is classified to obtain a scanned image category of each of the medical images to be identified. The image recognition method according to claim 1 or 2, wherein the style feature representation of the plurality of medical images to be recognized is classified to obtain a scanned image of each of the medical images to be recognized After the classification, the image recognition method further includes at least one of the following: sorting the plurality of medical images to be identified according to their scanned image categories; Displaying at least one of the medical images to be identified, sorted according to the scanned image categories, on the same screen; If the scanned image categories of the to-be-recognized medical images are duplicated, output first warning information to prompt the scanning personnel; If there is no preset scanned image category in the scanned image categories of the plurality of medical images to be identified, outputting second warning information to prompt the scanning personnel; If the classification confidence of the scanned image category of the to-be-recognized medical image is less than the preset confidence threshold, output third warning information to prompt the scanning personnel. The image recognition method according to claim 1 or 2, wherein, before extracting the style feature representation of each of the medical images to be recognized, the method further comprises: Preprocessing is performed on each of the medical images to be recognized, wherein the preprocessing includes at least one of the following: adjusting the image size of the medical images to be recognized to a preset size; The image intensity of the image is normalized to a preset range. The image recognition method according to claim 1 or 2, wherein the method further comprises: Respectively extract the content feature representation of each of the medical images to be identified; Perform lesion identification on the content feature representations of the plurality of medical images to be identified, to obtain a lesion area in each of the medical images to be identified. The image recognition method according to claim 5, wherein the performing focus recognition on the content feature representations of the plurality of medical images to be recognized, and obtaining the focus area in each of the medical images to be recognized includes: performing a second fusion process on the content feature representations of the plurality of to-be-identified medical images to obtain a final content feature representation; Perform lesion identification on the final content feature representation to obtain a lesion area in each of the to-be-identified medical images; and/or, The method also includes: Prompt the currently displayed lesion area of the medical image to be identified. The image recognition method according to claim 6, wherein performing a second fusion process on the content feature representations of the plurality of medical images to be recognized includes any one of the following: Perform splicing processing on the content feature representations of the plurality of medical images to be identified; performing addition processing on the content feature representations of the plurality of medical images to be identified; The dimensions of the final content feature representation and the content feature representations of the multiple medical images to be recognized are the same. The image recognition method according to claim 5, wherein the separately extracting the style feature representation of each of the medical images to be recognized comprises: Extract the style feature representation of each medical image to be identified by using the style coding sub-network of the identification network; The classification processing is performed on the style feature representations of the plurality of medical images to be recognized, and the scanned image category of each medical image to be recognized is obtained, including: Use the classification processing sub-network of the recognition network to classify and process the style feature representations of the plurality of medical images to be recognized, to obtain a scanned image category of each of the medical images to be recognized; The separately extracting the content feature representation of each of the medical images to be identified includes: Extract the content feature representation of each of the medical images to be identified by using the content coding sub-network of the identification network; Performing focus identification on the content feature representations of the plurality of medical images to be identified, to obtain a focus area in each of the medical images to be identified, including: Use the region segmentation sub-network of the identification network to perform lesion identification on the content feature representation of the plurality of medical images to be identified, to obtain a lesion area in each of the medical images to be identified. The image recognition method according to claim 8, wherein, before extracting the style feature representation of each of the medical images to be recognized, the image recognition method further comprises: acquiring a plurality of sample medical images, wherein the plurality of sample medical images are marked with their real scanned image categories and real lesion areas; Using the style coding sub-network to extract the sample style feature representation of each of the sample medical images respectively, and using the content coding sub-network to separately extract the sample content feature representation of each of the sample medical images; Use the classification processing sub-network to classify and process the sample style feature representations of the multiple sample medical images to obtain the predicted scanned image category of each of the sample medical images, and use the region segmentation sub-network performing lesion identification on the sample content feature representation of the plurality of sample medical images to obtain a predicted lesion area in each of the sample medical images; Using the difference between the real scanned image category and the predicted scanned image category, adjust the network parameters of the style encoding sub-network and the classification processing sub-network, and use the real lesion area and the The difference of the lesion area is predicted, and the network parameters of the content coding sub-network and the area segmentation sub-network are adjusted. The image recognition method according to claim 9, wherein the image recognition method further comprises: Obtaining the distribution of sample data represented by the sample style feature of each of the sample medical images; Using the difference between the distributions of the sample data, the network parameters of the style encoding sub-network are adjusted. The image recognition method according to claim 9, wherein the image recognition method further comprises: Using one of the sample style feature representations and one of the content feature representations, constructing a reconstructed image corresponding to the sample style feature representations; The network parameters of the style encoding sub-network and the content encoding sub-network are adjusted by using the difference between the reconstructed image and the sample medical image to which the corresponding sample style feature representation belongs. The image recognition method according to claim 8, wherein the style encoding sub-network comprises: a sequentially connected downsampling layer and a global pooling layer; and/or, The content encoding sub-network includes any of the following: sequentially connected downsampling layers and residual blocks, sequentially connected convolutional layers and pooling layers. An electronic device includes a mutually coupled memory and a processor, wherein the processor is configured to execute program instructions stored in the memory, so as to implement the image recognition method according to any one of claim 1 to 12. A computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, implement the image recognition method described in any one of claim 1 to 12.

Images