CN116128819A

CN116128819A - Image processing method, model training method and device and electronic equipment

Info

Publication number: CN116128819A
Application number: CN202211658622.4A
Authority: CN
Inventors: 陈修远; 王俊; 管添; 洪楠; 孙超; 于朋鑫; 王大为; 陈宽; 王少康
Original assignee: Peking University Peoples Hospital; Infervision Medical Technology Co Ltd
Current assignee: Peking University Peoples Hospital; Infervision Medical Technology Co Ltd
Priority date: 2022-12-22
Filing date: 2022-12-22
Publication date: 2023-05-16

Abstract

The application discloses an image processing method, a model training device and electronic equipment. The method comprises the following steps: partitioning a mediastinum region of the lung medical image to obtain at least two mediastinum partition images, wherein different mediastinum partition images comprise different anatomical structures of the mediastinum region; and according to the at least two mediastinal subarea images, performing lesion detection on different anatomical structures through at least two branch networks in the neural network model to obtain a lesion detection result of a mediastinal region of the lung medical image, wherein one branch network is at least used for detecting the mediastinal lesions in the anatomical structures of one mediastinal subarea image. One branch network is at least used for detecting lesions in the anatomical structure of one mediastinal subarea image, that is, different branch networks are adopted for detecting lesions for different mediastinal subarea images so as to realize the optimal lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

Description

Image processing method, model training method and device and electronic equipment

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to an image processing method, a model training device, and an electronic device.

Background

In recent years, deep learning technology is applied to the medical field to improve disease diagnosis and adjuvant therapy, and detection, identification, quantitative evaluation, diagnosis and analysis of chest and lung diseases in medical images by using the deep learning technology have been studied in a large amount. However, the current research mainly aims at the deep learning detection algorithm developed for lung field lesions, but omits the lesions of the mediastinum region, and meanwhile, as the lung field and the mediastinum have obviously different intensity distribution, for example, the lung field is mainly composed of air, the mediastinum is completely composed of body tissues, and the anatomical structure in the lung field is different from that in the mediastinum, the deep learning detection algorithm developed for the lung field lesions cannot well detect the mediastinum lesions, so that the detection accuracy of the mediastinum lesions is greatly reduced.

Disclosure of Invention

In view of the foregoing, embodiments of the present application are directed to providing an image processing method, a model training method and apparatus, and an electronic device capable of accurately detecting a mediastinal lesion.

According to a first aspect of an embodiment of the present application, there is provided an image processing method, including: partitioning a mediastinum region of the lung medical image to obtain at least two mediastinum partition images, wherein different mediastinum partition images comprise different anatomical structures of the mediastinum region; and according to the at least two mediastinal subarea images, performing lesion detection on different anatomical structures through at least two branch networks in the neural network model to obtain a lesion detection result of a mediastinal region of the lung medical image, wherein one branch network is at least used for detecting the mediastinal lesions in the anatomical structures of one mediastinal subarea image.

In one embodiment, each branch network includes a self-attention network, and lesion detection is performed on different anatomical structures through at least two branch networks in a neural network model according to at least two mediastinal subarea images to obtain a lesion detection result of a mediastinal region of a lung medical image, including: according to at least two sub-subarea images corresponding to each mediastinum subarea image, obtaining a characteristic diagram corresponding to each mediastinum subarea image through a self-attention network; acquiring a total feature map according to the feature map corresponding to each mediastinum subarea image; and performing function operation or convolution operation on the total feature map to obtain a lesion detection result of the mediastinal region of the lung medical image.

In one embodiment, before obtaining, through the self-attention network, a feature map corresponding to each mediastinal subarea image from at least two subarea images corresponding to each mediastinal subarea image, the method further comprises: expanding one mediastinal subarea image in two adjacent mediastinal subarea images to the direction of the other mediastinal subarea image to obtain an expanded mediastinal subarea image, wherein the expanded mediastinal subarea image and the other mediastinal subarea image have overlapping parts; partitioning the expanded mediastinum partition image and the other mediastinum partition image to obtain at least two sub-partition images corresponding to the expanded mediastinum partition image and at least two sub-partition images corresponding to the other mediastinum partition image.

In one embodiment, according to at least two sub-subarea images corresponding to each mediastinal subarea image, a feature map corresponding to each mediastinal subarea image is obtained through a self-attention network, and the feature map comprises: obtaining a first feature map through a self-attention network according to at least two sub-subarea images corresponding to one mediastinal subarea image in two adjacent mediastinal subarea images; and obtaining a second feature map through a self-attention network according to the first feature map and at least two sub-subarea images corresponding to the other mediastinal subarea image in the two adjacent mediastinal subarea images.

In one embodiment, when two adjacent mediastinal subarea images have overlapping portions, the overlapping portion of the first feature map corresponds to the first overlapping feature map, the overlapping portion of the second feature map corresponds to the second overlapping feature map, wherein, according to the feature map corresponding to each mediastinal subarea image, a total feature map is obtained, including: carrying out weighted summation on the first overlapped characteristic diagram and the second overlapped characteristic diagram to obtain an overlapped characteristic diagram; and combining the total feature map, the feature map except the first overlapped feature map in the first feature map and the feature map except the second overlapped feature map in the second feature map to obtain the total feature map.

In one embodiment, zoning the mediastinal region of the medical image of the lung results in at least two mediastinal zonal images comprising: windowing an image of a region of interest corresponding to the lung medical image by using a mediastinum window to obtain a mediastinum region; windowing is carried out on an image of a region of interest corresponding to the lung medical image by utilizing a bone window, so as to obtain a bone region, wherein the bone region comprises a thoracic vertebra and a vertebra; dividing the mediastinum area by taking a straight line between the thoracic vertebra and the vertebra as a dividing reference to obtain at least two mediastinum dividing images.

In one embodiment, partitioning the mediastinum region with a line between the thoracic vertebra and the vertebra as a partition reference, to obtain at least two sub-partitioned images, including: taking an area image corresponding to a straight line of a first preset length, which is close to one side of the thoracic vertebrae, as an upper mediastinum partition image; taking an area image corresponding to a straight line of a second preset length, which is close to one side of the vertebra, as a lower mediastinum partition image; the region image between the upper mediastinum subarea image and the lower mediastinum subarea image is a middle mediastinum subarea image, wherein the upper mediastinum subarea image and the lower mediastinum subarea image adopt the same branch network for lesion detection, and the middle mediastinum subarea image adopts another branch network for lesion detection.

In one embodiment, the method further comprises: thresholding is carried out on the lung medical image to obtain a lung field mask; expanding the lung field mask outwards to contain the thoracic vertebrae to obtain a mask of the region of interest; and cutting the lung medical image by using the region-of-interest mask to obtain a region-of-interest image.

According to a second aspect of embodiments of the present application, there is provided a training method of a neural network model, including: partitioning a mediastinum area of the lung medical sample image to obtain at least two mediastinum partitioned sample images, wherein the lung medical sample image is marked with a lesion label; obtaining a lesion prediction result of a mediastinum region of the lung medical sample image through at least two branch networks in a neural network model according to at least two mediastinum partition sample images corresponding to each mediastinum partition sample image, wherein one branch network is at least used for predicting a mediastinum lesion in an anatomical structure of one mediastinum partition sample image; calculating a loss function value according to the lesion prediction result and the lesion label; and updating parameters of the neural network model according to the loss function value.

According to a third aspect of the embodiments of the present application, there is provided an image processing apparatus including: the first acquisition module is configured to partition a mediastinum region of the lung medical image to obtain at least two mediastinum partition images, wherein different mediastinum partition images comprise different anatomical structures of the mediastinum region; and the second acquisition module is configured to perform lesion detection on different anatomical structures through at least two branch networks in the neural network model according to at least two mediastinal subarea images to obtain a lesion detection result of a mediastinal region of the lung medical image, wherein one branch network is at least used for detecting mediastinal lesions in the anatomical structure of one mediastinal subarea image.

According to a third aspect of embodiments of the present application, there is provided a training apparatus for a neural network model, including: the fourth acquisition module is configured to partition a mediastinum area of the lung medical sample image to obtain at least two mediastinum partition sample images, wherein the lung medical sample image is marked with a lesion label; the training module is configured to obtain a lesion prediction result of a mediastinum region of the lung medical sample image through at least two branch networks in the neural network model according to at least two mediastinum partition sample images corresponding to each mediastinum partition sample image, wherein one branch network is at least used for predicting a mediastinum lesion in an anatomical structure of one mediastinum partition sample image; calculating a loss function value according to the lesion prediction result and the lesion label; and updating parameters of the neural network model according to the loss function value.

According to a fifth aspect of embodiments of the present application, there is provided an electronic device including: a processor; and a memory having stored therein computer program instructions which, when executed by the processor, cause the processor to perform the method of the first or second aspect mentioned above.

According to a sixth aspect of embodiments of the present application, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the method mentioned in the first or second aspect above.

According to the image processing method provided by the embodiment of the application, at least two mediastinal subarea images can be obtained by dividing the mediastinal area of the lung medical image, and then lesion detection is carried out on different anatomical structures of the mediastinal area through at least two branch networks in the neural network model according to the at least two mediastinal subarea images, so that a lesion detection result of the mediastinal area of the lung medical image is obtained. One branch network is at least used for detecting lesions in the anatomical structure of one mediastinal subarea image, that is, different branch networks are adopted for detecting lesions for different mediastinal subarea images so as to realize the optimal lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

Drawings

The foregoing and other objects, features and advantages of the present application will become more apparent from the following more particular description of embodiments of the present application, as illustrated in the accompanying drawings. The accompanying drawings are included to provide a further understanding of embodiments of the application and are incorporated in and constitute a part of this specification, illustrate the application and not constitute a limitation to the application. In the drawings, like reference numerals generally refer to like parts or steps.

Fig. 1 is a schematic system architecture diagram of an application scenario of an image processing method according to an embodiment of the present application.

Fig. 2 is a flow chart of an image processing method according to an embodiment of the present application.

Fig. 3 is a flowchart of an image processing method according to another embodiment of the present application.

FIG. 4 is a schematic illustration of a mediastinum partition image provided in one embodiment of the present application.

Fig. 5 is a flowchart of a data processing procedure of the image processing method provided in one embodiment of the present application.

Fig. 6 is a flowchart of a process for acquiring an image of a region of interest corresponding to a medical image of a lung according to an embodiment of the present application.

Fig. 7 is a flowchart of a training method of a neural network model according to an embodiment of the present application.

Fig. 8 is a block diagram of an image processing apparatus provided in one embodiment of the present application.

Fig. 9 is a block diagram of a training apparatus for neural network models provided in one embodiment of the present application.

Fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

The mediastinum area is a common site of various lesions, mainly including hyperplasia, cyst, tumor, lymph node metastasis from lung, etc., and detection of the mediastinum lesions plays an important role in early screening and diagnosis of related diseases. The deep learning detection algorithm developed for lung field lesions at present can not well detect mediastinal lesions due to the following two aspects. On the one hand, the lung fields have significantly different intensity distribution from the mediastinum, the lung fields are mainly air, and the mediastinum is completely composed of body tissues; on the other hand, the anatomical structure in the lung fields is different from that in the mediastinum, and the lung fields mainly comprise bronchi and branch arteries and veins, and are distributed in a sparse tree shape; the mediastinum can be divided into three areas, and the upper mediastinum area is positioned between the sternum body and the pericardium and is very narrow; the lower mediastinum region is located between the pericardium and spine for accommodating the tracheal bifurcation and the left and right main bronchi, esophagus, blood vessels, lymph nodes, etc.; the mediastinum region is located between the superior and inferior mediastinum regions for accommodating the heart, large blood vessels into and out of the heart, phrenic nerves, lymph nodes, and the like. Because the anatomical structures contained in different mediastinum areas are different, the types and the appearances of lesions in different mediastinum areas are different, and if the lesions in different mediastinum areas are detected by adopting the same neural network, the accuracy of the mediastinum lesion detection can be greatly reduced.

In order to solve the above-mentioned problems, an embodiment of the present application provides an image processing method, which includes dividing a mediastinum region of a lung medical image to obtain at least two mediastinum divided region images, and performing lesion detection on different anatomical structures of the mediastinum region through at least two branch networks in a neural network model according to the at least two mediastinum divided region images to obtain a lesion detection result of the mediastinum region of the lung medical image. One branch network is at least used for detecting lesions in the anatomical structure of one mediastinal subarea image, that is, different branch networks are adopted for detecting lesions for different mediastinal subarea images so as to realize the optimal lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

Since the embodiments of the present application relate to applications of neural networks, for ease of understanding, related terms and related concepts of neural networks that may be related to the embodiments of the present application will be briefly described below.

A neural network is an operational model consisting of a large number of nodes (or neurons) interconnected, each node corresponding to a policy function, and the connections between each two nodes representing a weighting value, called weight, for signals passing through the connection. The neural network generally includes a plurality of neural network layers, the upper and lower network layers are cascaded with each other, an output of the ith neural network layer is connected to an input of the (i+1) th neural network layer, an output of the (i+1) th neural network layer is connected to an input of the (i+2) th neural network layer, and so on. After training samples are input into cascaded neural network layers, an output result is output through each neural network layer and is used as the input of the next neural network layer, therefore, the output is obtained through calculation of a plurality of neural network layers, the output predicted result of the output layer is compared with a real target value, then the weight matrix and the strategy function of each layer are adjusted according to the difference condition between the predicted result and the target value, the neural network continuously passes through the adjustment process by using the training samples, parameters such as the weight of the neural network are adjusted until the predicted result output by the neural network accords with the real target result, and the process is called as the training process of the neural network. After the neural network is trained, a neural network model can be obtained.

In training the neural network, because the output of the neural network is expected to be as close to the truly desired value as possible, the weight vector of each layer of the neural network can be updated by comparing the predicted value of the current network with the truly desired target value and then according to the difference between the two (of course, there is usually an initialization process before the first update, that is, the parameters are preconfigured for each layer in the neural network), for example, if the predicted value of the network is higher, the weight vector is adjusted to be lower than predicted, and the adjustment is continued until the neural network can predict the truly desired target value or a value very close to the truly desired target value. Thus, the difference between the predicted value and the target value can be compared using a loss function (loss function) or an objective function (objective function), which are important equations for measuring the difference between the predicted value and the target value. Taking the loss function as an example, the higher the output value (loss) of the loss function is, the larger the difference is, and the training of the neural network becomes the process of reducing the loss as much as possible.

The following describes in detail, with reference to fig. 1, a system architecture of an application scenario of an image processing method according to an embodiment of the present application. As shown in fig. 1, the application scenario provided in the embodiment of the present application relates to a CT scanner 110, a server 120, and a computer device 130.

The CT scanner 110 is used to perform X-ray scanning on human tissue to obtain a CT image of the human tissue. In one embodiment, a medical image of the lung may be obtained by scanning the lung with CT scanner 110.

The computer device 120 may be a general-purpose computer or a computer apparatus composed of an application specific integrated circuit, etc., which is not limited in this embodiment. For example, the computer device 120 may be a mobile terminal device such as a tablet computer or may also be a personal computer (Personal Computer, PC), such as a laptop portable computer and a desktop computer, etc. The computer device 120 is connected to the CT scanner 110 via a communication network. Optionally, the communication network is a wired network or a wireless network. In some alternative embodiments, the computer device 120 receives the lung medical image sent by the CT scanner 110, the computer device 120 segments the mediastinum region of the lung medical image to obtain at least two mediastinum segmented images, and the computer device 120 performs lesion detection on different anatomical structures of the mediastinum region through at least two branch networks in the neural network model deployed thereon according to the at least two mediastinum segmented images, thereby obtaining a lesion detection result of the mediastinum region of the lung medical image.

Server 130 is a server, or is composed of several servers, or is a virtualization platform, or is a cloud computing service center. The server 130 is connected to the computer device 120 via a communication network. Optionally, the communication network is a wired network or a wireless network. In some alternative embodiments, server 130 receives the lung medical sample images acquired by computer device 110 and trains at least two branch networks with the lung medical sample images to obtain a neural network model for detecting lesions in the mediastinal region of the lung medical images. The computer device 120 may send the lung medical image obtained from the CT scanner 110 to the server 130, the server 130 segments the mediastinum region of the lung medical image to obtain at least two mediastinum segmented images, and the server 130 performs lesion detection on different anatomical structures of the mediastinum region according to the at least two mediastinum segmented images and through at least two branch networks in a neural network model deployed thereon, thereby obtaining a lesion detection result of the mediastinum region of the lung medical image, and sends the lesion detection result to the computer device 120 for viewing by medical staff.

The embodiment of the application provides an image processing method, which comprises the steps of firstly partitioning a mediastinum region of a lung medical image to obtain at least two mediastinum partition images, and then performing lesion detection on different anatomical structures of the mediastinum region through at least two branch networks in a neural network model according to the at least two mediastinum partition images to obtain a lesion detection result of the mediastinum region of the lung medical image. One branch network is at least used for detecting lesions in the anatomical structure of one mediastinal subarea image, that is, different branch networks are adopted for detecting lesions for different mediastinal subarea images so as to realize the optimal lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

The image processing method mentioned in the embodiment of the present application is described in detail below with reference to fig. 2 to 7.

Fig. 2 is a flow chart of an image processing method according to an embodiment of the present application. The method described in fig. 2 is performed by the server 130 mentioned in fig. 1 or other type of electronic device with data processing functionality. As shown in fig. 2, the method includes the following steps.

Step S210, partitioning the mediastinum area of the lung medical image to obtain at least two mediastinum partition images, wherein different mediastinum partition images comprise different anatomical structures of the mediastinum area.

The medical image of the lung may be a medical image such as an electronic computed tomography (Computed Tomography, CT), a magnetic resonance imaging (Magnetic Resonance Imaging, MRI), a computed radiography (Computed Radiography, CR) or a digital radiography (Digital radiography, DR), which is not particularly limited in this embodiment of the present application. The mediastinal region refers to the region near the left and right mediastinal pleura, between which the heart and large blood vessels, esophagus, trachea, thymus, nerves, lymphoid tissues, etc. come in and go out.

According to anatomical prior knowledge of the mediastinal region, different anatomical structures of the mediastinal region of the medical image of the lung are segmented to divide the mediastinal region into at least two mediastinal segmented images. In particular, different anatomical structures of the mediastinal region of the medical image of the lung are segmented according to the complexity of the anatomical structures of the mediastinal region to divide the mediastinal region into at least two mediastinal segmented images.

For example, the mediastinal region of the medical image of the lung is divided into a left mediastinal subzone image and a right mediastinal subzone image in a left-right manner; for another example, according to the mode of the upper side and the lower side, dividing the mediastinum area of the lung medical image into an upper mediastinum subarea image, a middle mediastinum subarea image and a lower mediastinum subarea image; for another example, the mediastinal region of the medical image of the lung is divided into a anterior mediastinal subzone image and a posterior mediastinal subzone image in a anteroposterior manner. The zoning mode of the mediastinal region is not specifically limited in the embodiment of the present application, and the number of at least two mediastinal zoning images is not specifically limited in the embodiment of the present application.

In one example, at least two mediastinal subzone images are acquired by the following subzone approach: windowing an image of a region of interest corresponding to the lung medical image by using a mediastinum window to obtain a mediastinum region; windowing is carried out on an image of a region of interest corresponding to the lung medical image by utilizing a bone window, so as to obtain a bone region, wherein the bone region comprises a thoracic vertebra and a vertebra; dividing the mediastinum area by taking a straight line between the thoracic vertebra and the vertebra as a dividing reference to obtain at least two mediastinum dividing images.

The window level and width of the mediastinal window and the bone window are different, but the embodiments of the present application do not specifically limit the specific values of the window level and width of both. And windowing is carried out on the region-of-interest image corresponding to the lung medical image by utilizing the window level and the window width specified by the mediastinum window, so that the mediastinum region corresponding to the lung medical image can be obtained, and the region-of-interest image corresponding to the lung medical image can be windowed by utilizing the window level and the window width specified by the bone window, so that the bone region can be obtained.

After the windowing operation is performed on the region-of-interest image, thresholding operation may also be performed on the windowed region-of-interest image to determine that the foreground region is a mediastinum region or a bone region.

Determining a straight line between the thoracic vertebra and the vertebra, and partitioning the mediastinum area by taking the straight line as a partitioning reference to obtain at least two mediastinum partition images. For example, the mediastinal region on the left side of the line is divided into left mediastinal subzone images, and the mediastinal region on the right side of the line is divided into right mediastinal subzone images; for another example, the straight line is divided into at least two sections of straight lines, and the mediastinal area corresponding to each section of straight line is a mediastinal subarea image.

The mediastinal subzone image may comprise only the anatomy of the mediastinal region, i.e. the mediastinal subzone image is only the subzone result of the mediastinal region; the mediastinal subzone image may also include not only the anatomy of the mediastinal region but also the anatomy of the lung region, i.e. the mediastinal subzone image extends to the lung region on the basis of the outcome of the segmentation of the mediastinal region.

Preferably, the mediastinal subzone image comprises not only the anatomy of the mediastinal region but also the anatomy of the lung region, since lesions occurring in the anatomy of the mediastinal region may extend into the lung region, which thus enables determination of the mediastinal lesions from the overall appearance, facilitating detection of the mediastinal lesions.

Step S220, according to at least two mediastinal subarea images, performing lesion detection on different anatomical structures through at least two branch networks in the neural network model to obtain a lesion detection result of a mediastinal area of the lung medical image.

Since the complexity of the anatomy in each mediastinal segmented image is different, the specific network structure of the branched network of lesion detection may be determined for the complexity of the anatomy. For mediastinal segmented images of anatomical structures of relatively high complexity, the network structure of the branched network may be more complex, e.g. comprising a decoder-encoder or the like; for mediastinal subzone images of relatively low complexity anatomical structures, the network structure of the branched network may be relatively simple, e.g. the network structure may consist of only a few basic layers such as convolutional layers. The specific network structure of the branch network is not limited in the embodiment of the application, and a person skilled in the art can select different networks according to actual requirements.

The number of at least two branch networks may also be determined according to the complexity of the anatomy in each mediastinal subzone image, for example, for two mediastinal subzone images where the anatomy with lower complexity is located, one branch network may be used together to detect the mediastinal lesion, that is, one branch network is at least used to detect the mediastinal lesion in the anatomy of one mediastinal subzone image, and for the mediastinal subzone image where the anatomy with higher complexity is located, one branch network is used separately to detect the mediastinal lesion, which can reduce the complexity of the neural network model, thereby improving the efficiency of the mediastinal lesion detection. The number of the at least two branch networks is not particularly limited, and a person skilled in the art can select different branch networks according to actual requirements.

By using a branch network together for the two mediastinal subarea images of the anatomical structure with lower complexity, the weights of the two mediastinal subarea images in the branch network can be shared, so that the training of the neural network model is facilitated, and the training efficiency of the neural network model is improved.

The lesion detection result can be a probability value of the mediastinum lesion and the true mediastinum lesion marked on the lung medical image through a detection frame or a point, or can also be a thermodynamic diagram which is an image representation form of visual network attention and is used for showing the area where the mediastinum lesion is located on the lung medical image in a special highlight form and showing the lesion weight level through different color highlights, so that the diagnosis of the mediastinum lesion by medical staff is facilitated. The form of the lesion detection result is not particularly limited in the embodiment of the present application, and a person skilled in the art may perform different selections according to actual needs.

The thermodynamic diagram can be understood as a gaussian thermodynamic diagram built with the center of the mediastinal lesion as the core, and by controlling the threshold values such that points at different pixel positions of the thermodynamic diagram are represented by different colors Gao Lianglai, the area of the thermodynamic diagram most likely to belong to the mediastinal lesion can be determined.

In one embodiment of the present application, each branch network includes at least a self-attention network that calculates the mediastinum subzone images based on a self-attention mechanism, such that each sub-zone image in the mediastinum subzone image can be associated with other sub-zone images, thereby enhancing the feature representation. As shown in fig. 3, step S220 includes the following.

Step S310, obtaining a feature map corresponding to each mediastinal subarea image through a self-attention network according to at least two subarea images corresponding to each mediastinal subarea image.

The self-attention network may be referred to as a transformer network, or as Transformer Network, which links each sub-subarea image of the mediastinum subarea image with other sub-subarea images by means of a self-attention mechanism, i.e. the feature map corresponding to each mediastinum subarea image is a feature enhanced feature map.

The mediastinum lesion may exist at the intersection of two adjacent mediastinum subareas images, that is, one mediastinum subarea image of a part of the mediastinum lesion in two adjacent mediastinum subareas images and the other mediastinum subarea image of the other mediastinum subarea image, if the mediastinum lesion detection is directly performed on each mediastinum subarea image, the mediastinum lesion at the intersection is undetectable or the mediastinum detection result is inaccurate, therefore, at least two sub-subarea images corresponding to each mediastinum subarea image may be obtained by the following methods: expanding one mediastinal subarea image in two adjacent mediastinal subarea images to the direction of the other mediastinal subarea image to obtain an expanded mediastinal subarea image; partitioning the expanded mediastinum partition image and the other mediastinum partition image to obtain at least two sub-partition images corresponding to the expanded mediastinum partition image and at least two sub-partition images corresponding to the other mediastinum partition image. The expanded mediastinal subzone image has an overlapping portion with another mediastinal subzone image so the expanded mediastinal subzone image may include all of the mediastinal lesions.

The subdividing of the mediastinal subzone image is performed according to the position, for example, the mediastinal subzone image is truncated with a window of a preset size, so that the mediastinal subzone image is truncated into four sub-subzone images, which are respectively located at the upper left position, the upper right position, the lower left position and the lower right position of the mediastinal subzone image.

In an example, a first feature map is obtained from at least two sub-subarea images corresponding to one of two adjacent mediastinal subarea images through a self-attention network corresponding thereto; and obtaining a second characteristic diagram according to at least two sub-subarea images corresponding to the other mediastinal subarea image in the two adjacent mediastinal subarea images through a self-attention network corresponding to the at least two subarea images.

In another example, a first feature map is obtained from at least two sub-subarea images corresponding to the expanded mediastinal subarea image of the adjacent two mediastinal subarea images through a self-attention network corresponding thereto; and obtaining a second characteristic diagram according to at least two sub-subarea images corresponding to the other mediastinal subarea image in the two adjacent mediastinal subarea images through a self-attention network corresponding to the at least two subarea images.

It should be noted that, the two adjacent mediastinal subareas images may use the same self-attention network, or may use different self-attention networks, which is not specifically limited in the embodiment of the present application.

Through the self-attention mechanism of the self-attention network, not only can each sub-subarea image in the mediastinum subarea image be connected with other sub-subarea images, but also the sub-subarea images in the two adjacent mediastinum subarea images can be connected, so that a second characteristic image is obtained through the self-attention network corresponding to at least two sub-subarea images corresponding to the other mediastinum subarea image in the two adjacent mediastinum subarea images, and the method comprises the following steps: and obtaining a second characteristic diagram according to the first characteristic diagram and at least two sub-subarea images corresponding to the other mediastinal subarea image in the two adjacent mediastinal subarea images through a self-attention network corresponding to the first characteristic diagram. That is, by the self-attention mechanism of the self-attention network, the first feature map can establish a relationship between at least two sub-subarea images corresponding to the other mediastinal subarea image of the adjacent two mediastinal subarea images.

Step S320, obtaining a total feature map according to the feature map corresponding to each mediastinal subarea image.

In an example, for two neighboring mediastinal subarea images that are expanded, the two neighboring mediastinal subarea images have overlapping portions, the overlapping portion of the first feature map corresponds to a first overlapping feature map, the overlapping portion of the second feature map corresponds to a second overlapping feature map, and the first overlapping feature map and the second overlapping feature map are weighted and summed to obtain a total overlapping feature map; and combining the total feature map, the feature map except the first overlapped feature map in the first feature map and the feature map except the second overlapped feature map in the second feature map to obtain the total feature map.

In another example, for two adjacent mediastinal subareas images which are not expanded, the feature map corresponding to one mediastinal subarea image in the two adjacent mediastinal subareas images is directly combined with the feature map corresponding to the other mediastinal subarea image in the two adjacent mediastinal subareas images, so that the total feature map is obtained.

Step S330, performing function operation or convolution operation on the total feature map to obtain a lesion detection result of the mediastinal region of the lung medical image.

In an example, when the lesion detection structure marks a mediastinal lesion and a probability value of the mediastinal lesion, which is a true mediastinal lesion, on the lung medical image in the form of a detection frame or a point, the total feature map is subjected to function operation by using an activation function, so as to obtain a lesion detection result of a mediastinal region of the lung medical image.

In another example, when the lesion detection structure is a thermodynamic diagram, a convolution operation is performed on the total feature map by using at least one convolution layer, so as to obtain a lesion detection result of a mediastinal region of the lung medical image.

In another embodiment of the present application, the segmentation of tissue structures in the mediastinal region of the heart or aorta may not be accurate enough to account for changes in tissue structures due to mediastinal lesions, and therefore, the lung physician is based on a priori knowledge of the anatomy of the mediastinal regionThe different anatomical structures of the mediastinal region of the image are segmented. As shown in fig. 4, which shows a schematic view of zoning the mediastinum region with the line between the thoracic vertebra and the spine as a zonal reference, the line between the thoracic vertebra and the spine is denoted as L, and at least two mediastinum zonal images include an upper mediastinum zonal image M ₁ Mediastinum partition image M ₂ Inferior mediastinum subzone image M ₃ . At a first preset length L ₁ The region image corresponding to the straight line L near the thoracic vertebrae side is an upper mediastinum partition image M ₁ The method comprises the steps of carrying out a first treatment on the surface of the At a second preset length L ₃ The region image corresponding to the straight line L near one side of the vertebra is a lower mediastinum partition image M ₃ The method comprises the steps of carrying out a first treatment on the surface of the The above mediastinum partition image M ₁ And lower mediastinum partition image M ₃ The area image between is the mediastinum partition image M ₂ 。

The value of the first preset length is not specifically limited, for example, may be 1/5 of the length of the straight line L, and the value of the second preset length is not specifically limited, for example, may be the length from the bottom end to the top end of the spine.

It should be noted that the mediastinal subarea image shown in fig. 4 includes not only the anatomy of the mediastinal area but also the anatomy of the lung area, i.e. the mediastinal subarea image extends to the lung area on the basis of the subarea result of the mediastinal area.

Of the three mediastinum partition images, the mediastinum partition image M ₂ Is the largest and contains the most complex anatomy; upper mediastinum partition image M ₁ And lower mediastinum partition image M ₃ And contains fewer anatomical structures, so that different branch networks are adopted for different mediastinal subarea images in order to enhance detection performance, improve calculation efficiency and reduce calculation burden. For example, the upper mediastinum subzone image and the lower mediastinum subzone image are used for lesion detection using the same branch network, which may be a self-attentive network only, and the calculation using the self-attentive network does not cause excessive calculation load due to the smaller area of the upper mediastinum subzone image and the lower mediastinum subzone image, while the middle mediastinum subzone image uses another branch network The branch network can be composed of a self-attention network and an encoder-decoder, wherein the encoder-decoder is used for mixing the self-attention network to extract the characteristics, because the area of the mediastinum partition image is relatively large and the included anatomical structure is more complex. In addition, the features of two adjacent sub-partition images in any mediastinum partition image are subjected to association calculation, and the features of two adjacent mediastinum partition images are subjected to association calculation.

That is, when the at least two mediastinal subarea images include an upper mediastinal subarea image, a middle mediastinal subarea image, and a lower mediastinal subarea image, and the branch network corresponding to the upper mediastinal subarea image and the lower mediastinal subarea image is a self-attention network, the branch network corresponding to the middle mediastinal subarea image is a self-attention network and an encoder-decoder, step S220 shown in fig. 2 includes: respectively expanding the upper mediastinum subarea image and the lower mediastinum subarea image to the direction of the middle mediastinum subarea image to obtain an expanded upper mediastinum subarea image and an expanded lower mediastinum subarea image; dividing the expanded upper mediastinum subarea image and the expanded lower mediastinum subarea image respectively to obtain at least two subarea images corresponding to the expanded upper mediastinum subarea image and at least two subarea images corresponding to the expanded lower mediastinum subarea image; inputting at least two sub-subarea images corresponding to the expanded upper mediastinum subarea image and at least two sub-subarea images corresponding to the expanded lower mediastinum subarea image into a self-attention network corresponding to the at least two sub-subarea images for respectively extracting the characteristics to obtain a first characteristic image corresponding to the upper mediastinum subarea image and a first characteristic image corresponding to the lower mediastinum subarea image; inputting the middle mediastinum characteristic image into an encoder for encoding to obtain a decoding characteristic image; partitioning the decoding feature image to obtain at least two sub-partitioned images corresponding to the middle mediastinum feature image; inputting a first feature map corresponding to the upper mediastinum subarea image, a first feature map corresponding to the lower mediastinum subarea image and at least two subarea images corresponding to the middle mediastinum subarea feature image into a self-attention network corresponding to the first feature map, the first feature map and the at least two subarea images to obtain a feature enhanced feature map; inputting the feature-enhanced feature map into a decoder for decoding to obtain a second feature map; combining the first characteristic image corresponding to the upper mediastinum subarea image, the first characteristic image corresponding to the lower mediastinum subarea image and the second characteristic image to obtain a total characteristic image; and performing function operation or convolution operation on the total feature map to obtain a lesion detection result of the mediastinal region of the lung medical image. As shown in fig. 5, which shows the data processing procedure of the image processing method as described above.

When the first feature map corresponding to the upper mediastinum subarea image, the first feature map corresponding to the lower mediastinum subarea image and the second feature map are combined, the first overlapping feature map corresponding to the upper mediastinum subarea image and the second overlapping feature map corresponding to the second feature map can be subjected to weighted summation, and the second overlapping feature map corresponding to the first overlapping feature map corresponding to the lower mediastinum subarea image and the second overlapping feature map corresponding to the second feature map are subjected to weighted summation, so that the total overlapping feature map is obtained; and merging the total feature map, the feature maps except the first overlapped feature map of the first feature map corresponding to the upper mediastinum partition image, the feature maps except the first overlapped feature map of the first feature map corresponding to the lower mediastinum partition image and the feature maps except the second overlapped feature map in the second feature map to obtain the total feature map.

For example, the encoder may be constructed of a convolutional neural network, which contains two downsampling, so that the size of the encoded signature on each axis is 1/4 of the size of the medical image of the lung. The decoder is also composed of a convolutional neural network, which contains two upsamples and a skip connection from the encoder to the decoder, which can combine the position information into features to avoid positional deviations for subsequent lesion localization. The embodiments of the present application are not particularly limited to the specific composition of the encoder and decoder.

As shown in fig. 6, a process for acquiring an image of a region of interest corresponding to a medical image of the lung is shown. Specifically, thresholding operation is carried out on the lung medical image to obtain a lung field mask; expanding the lung field mask outwards to contain the thoracic vertebrae to obtain a mask of the region of interest; and cutting the lung medical image by using the region-of-interest mask to obtain a region-of-interest image.

When the lung field mask is expanded outwards to contain the thoracic vertebrae, the minimum circumscribed rectangle of the lung field mask can be taken, and the minimum circumscribed rectangle is expanded outwards to contain the thoracic vertebrae, so that the region-of-interest mask is obtained.

Considering that the lung field region mainly consists of air, the intensity of the lung field region is low, so that the lung medical image can be thresholded to obtain a lung field mask. Second, to enable a straight line between the thoracic vertebrae for zoning to be obtained, the lung field mask is expanded outwardly to encompass the thoracic vertebrae. Finally, the lung medical image is cut by using the region-of-interest mask to obtain a region-of-interest image, wherein the region-of-interest image comprises a mediastinum region and a majority of irrelevant regions are removed.

In summary, according to the image processing method provided by the embodiment of the application, based on anatomical priori knowledge, the mediastinum area of the lung medical image is divided into three parts, features are extracted from each part respectively, and feature interaction enhancement between neighborhoods is performed in each part, so that the computing efficiency is improved and the computing load is reduced while the accurate detection performance is maintained. In addition, the defect that the current computer-aided algorithm lacks in detecting the mediastinal lesions in the medical image is overcome, and the application of the AI in medical image analysis is further perfected.

The following describes in detail the training method of the neural network model mentioned in the embodiment of the present application with reference to fig. 7. The method described in fig. 7 is performed by the server 130 mentioned in fig. 1 or other type of electronic device with data processing functionality. As shown in fig. 7, the training method of the neural network model provided in the embodiment of the application includes the following steps.

It should be noted that, some of the content mentioned in the embodiments related to the training method of the neural network model is the same as that mentioned in the embodiments related to the image processing method, and differences between the two are emphasized below, which are not repeated herein, and please refer to the embodiments related to the image processing method specifically.

Step S710, partitioning the mediastinum area of the lung medical sample image to obtain at least two mediastinum partition sample images.

The lung medical sample image is marked with a lesion label, but the embodiment of the application does not particularly limit the specific form of the lesion label, and the lesion label can be a mediastinum lesion marked on the lung medical sample image in a detection frame or point form, or can be a Gaussian thermodynamic diagram constructed by taking the center of each mediastinum lesion on the lung medical sample image as a core, wherein the value of a pixel point close to the center is larger than that of a pixel point far from the center.

Step S720, obtaining a lesion prediction result of the mediastinal region of the lung medical sample image through at least two branch networks in the neural network model according to at least two mediastinal regional sample images corresponding to each mediastinal regional sample image.

The specific form of the lesion prediction result is also determined according to a lesion label, and when the lesion label is a mediastinum lesion marked on a lung medical sample image in the form of a detection frame or a point, the lesion prediction result refers to a probability value that the mediastinum lesion and the true mediastinum lesion are marked on the lung medical sample image in the form of the detection frame or the point; when the lesion label is a Gaussian thermodynamic diagram constructed by taking the center of each mediastinal lesion on a lung medical sample image as a core, the lesion prediction result refers to the thermodynamic diagram and the value of each pixel point in the thermodynamic diagram.

Step S730, calculating a loss function value according to the lesion prediction result and the lesion label.

And determining the difference between the lesion prediction result and the lesion label, inputting the difference into a loss function, and calculating a loss function value. In one example, when the lesion prediction result is a thermodynamic diagram, the regression loss is calculated pixel by pixel, resulting in the loss function value.

Step S740, updating parameters of the neural network model according to the loss function value.

The loss function value is gradient back-transferred to update parameters in the neural network model, such as weights, bias values, etc., which are not specifically limited in the embodiments of the present application.

According to the training method of the neural network model, firstly, mediastinum areas of lung medical sample images are segmented to obtain at least two mediastinum segmented sample images, then according to the at least two mediastinum segmented sample images corresponding to each mediastinum segmented sample image, lesion prediction results of the mediastinum areas of the lung medical sample images are obtained through at least two branch networks in the neural network model, finally, loss function values are calculated according to the lesion prediction results and lesion labels, and parameters of the neural network model are updated according to the loss function values. One branch network in the neural network model obtained by training is at least used for detecting the lesion in the anatomical structure of one mediastinum subarea image, that is, different branch networks are adopted for detecting the lesion for different mediastinum subarea images so as to realize the optimal lesion detection performance, thereby improving the accuracy of mediastinum lesion detection.

Method embodiments of the present application are described above in detail in connection with fig. 2-7, and apparatus embodiments of the present application are described below in detail in connection with fig. 8 and 9. It is to be understood that the description of the method embodiments corresponds to the description of the device embodiments, and that parts not described in detail can therefore be seen in the preceding method embodiments.

Fig. 8 is a schematic structural diagram of an image processing apparatus 800 provided in an embodiment of the present application. As shown in fig. 8, the apparatus 800 of fig. 8 may include: a first acquisition module 810 and a second acquisition module 820. These modules are described in detail below.

The first acquisition module 810 is configured to segment a mediastinal region of the medical image of the lung resulting in at least two mediastinal segmented images, wherein different mediastinal segmented images comprise different anatomical structures of the mediastinal region.

The second acquisition module 820 is configured to perform lesion detection on different anatomical structures according to at least two mediastinal subarea images through at least two branch networks in the neural network model, to obtain a lesion detection result of a mediastinal region of the lung medical image, wherein one branch network is used for detecting a lesion in the anatomical structure of one mediastinal subarea image.

According to the image processing device provided by the embodiment of the application, at least two mediastinal subarea images can be obtained by dividing the mediastinal area of the lung medical image, and then lesion detection is carried out on different anatomical structures of the mediastinal area through at least two branch networks in the neural network model according to the at least two mediastinal subarea images, so that a lesion detection result of the mediastinal area of the lung medical image is obtained. One branch network is at least used for detecting lesions in the anatomical structure of one mediastinal subarea image, that is, different branch networks are adopted for detecting lesions for different mediastinal subarea images so as to realize the optimal lesion detection performance, thereby improving the accuracy of mediastinal lesion detection.

In an embodiment of the present application, each branch network includes a self-attention network, and the second obtaining module 820 is further configured to obtain, through the self-attention network, a feature map corresponding to each mediastinal subarea image according to at least two subarea images corresponding to each mediastinal subarea image; acquiring a total feature map according to the feature map corresponding to each mediastinum subarea image; and performing function operation or convolution operation on the total feature map to obtain a lesion detection result of the mediastinal region of the lung medical image.

In an embodiment of the present application, the apparatus 800 further includes: the expansion module 830 is configured to expand one mediastinal subarea image of two adjacent mediastinal subarea images to the direction of the other mediastinal subarea image to obtain an expanded mediastinal subarea image, wherein the expanded mediastinal subarea image and the other mediastinal subarea image have an overlapping part; the partition module 840 is configured to partition the expanded mediastinal partition image and the other mediastinal partition image to obtain at least two sub-partition images corresponding to the expanded mediastinal partition image and at least two sub-partition images corresponding to the other mediastinal partition image.

In an embodiment of the present application, when the second obtaining module 820 obtains, through the self-attention network, the feature map corresponding to each mediastinum subarea image according to at least two subarea images corresponding to each mediastinum subarea image, the second obtaining module is further configured to obtain, through the self-attention network, the first feature map according to at least two subarea images corresponding to one mediastinum subarea image of the two adjacent mediastinum subarea images; and obtaining a second feature map through a self-attention network according to the first feature map and at least two sub-subarea images corresponding to the other mediastinal subarea image in the two adjacent mediastinal subarea images.

In an embodiment of the present application, when two adjacent mediastinal subarea images have overlapping portions, the overlapping portion of the first feature map corresponds to the first overlapping feature map, the overlapping portion of the second feature map corresponds to the second overlapping feature map, and the second obtaining module 820 is further configured to, when obtaining the total feature map according to the feature map corresponding to each mediastinal subarea image, perform weighted summation on the first overlapping feature map and the second overlapping feature map to obtain a total overlapping feature map; and combining the total feature map, the feature map except the first overlapped feature map in the first feature map and the feature map except the second overlapped feature map in the second feature map to obtain the total feature map.

In an embodiment of the present application, the first obtaining module 810 is further configured to perform windowing operation on the region of interest image corresponding to the lung medical image by using a mediastinum window to obtain a mediastinum region; windowing is carried out on an image of a region of interest corresponding to the lung medical image by utilizing a bone window, so as to obtain a bone region, wherein the bone region comprises a thoracic vertebra and a vertebra; dividing the mediastinum area by taking a straight line between the thoracic vertebra and the vertebra as a dividing reference to obtain at least two mediastinum dividing images.

In an embodiment of the present application, when the first obtaining module 810 partitions the mediastinum area with a straight line between the thoracic vertebra and the vertebra as a partition reference to obtain at least two sub-partition images, the first obtaining module is further configured to use an area image corresponding to a straight line near one side of the thoracic vertebra with a first preset length as an upper mediastinum partition image; taking an area image corresponding to a straight line of a second preset length, which is close to one side of the vertebra, as a lower mediastinum partition image; the region image between the upper mediastinum partition image and the lower mediastinum partition image is a middle mediastinum partition image.

In an embodiment of the present application, the upper mediastinal subarea image and the lower mediastinal subarea image use the same branch network for lesion detection, and the middle mediastinal subarea image uses another branch network for lesion detection.

In an embodiment of the present application, the apparatus 800 further includes: a third obtaining module 850 configured to perform thresholding operation on the lung medical image to obtain a lung field mask; expanding the lung field mask outwards to contain the thoracic vertebrae to obtain a mask of the region of interest; and cutting the lung medical image by using the region-of-interest mask to obtain a region-of-interest image.

Fig. 9 is a schematic structural diagram of a training apparatus 900 for a neural network model provided in an embodiment of the present application. As shown in fig. 9, the training apparatus 900 of fig. 9 may include: a fourth acquisition module 910 and a training module 920. These modules are described in detail below.

The fourth acquisition module 910 is configured to partition a mediastinal region of a lung medical sample image, resulting in at least two mediastinal partitioned sample images, wherein the lung medical sample image is labeled with a lesion label.

The training module 920 is configured to obtain a lesion prediction result of a mediastinal region of the lung medical sample image according to at least two mediastinal regional sample images corresponding to each mediastinal regional sample image through at least two branch networks in the neural network model, wherein one branch network is at least used for predicting a mediastinal lesion in an anatomical structure of one mediastinal regional sample image; calculating a loss function value according to the lesion prediction result and the lesion label; and updating parameters of the neural network model according to the loss function value.

According to the training device of the neural network model, firstly, the mediastinum area of the lung medical sample image is partitioned to obtain at least two mediastinum partition sample images, then, according to the at least two mediastinum partition sample images corresponding to each mediastinum partition sample image, a lesion prediction result of the mediastinum area of the lung medical sample image is obtained through at least two branch networks in the neural network model, finally, a loss function value is calculated according to the lesion prediction result and a lesion label, and parameters of the neural network model are updated according to the loss function value. One branch network in the neural network model obtained by training is at least used for detecting the lesion in the anatomical structure of one mediastinum subarea image, that is, different branch networks are adopted for detecting the lesion for different mediastinum subarea images so as to realize the optimal lesion detection performance, thereby improving the accuracy of mediastinum lesion detection.

Fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device 1000 shown in fig. 10 may include a memory 1010 and a processor 1020. Memory 1010 may be used to store executable code. The processor 1020 may be used to execute executable code stored in the memory 1010 to implement the steps in the various methods described previously. In some embodiments, the electronic device 1000 may also include a network interface 1030, and data exchange of the processor 1000 with external devices may be accomplished through the network interface 1030.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy Disk, a hard Disk, a magnetic tape), an optical medium (e.g., a digital video disc (Digital Video Disc, DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether or not these functions are implemented in hardware

Or software, depending on the particular application of the solution and design constraints. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the present disclosure

The scope of the application.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the single sheet

The division of elements is merely a logical function division, and there may be other division manners in actual implementation, for example, multiple single 0 elements or components may be combined or integrated into another system, or some features may be omitted or not performed. Another one

The points of the disclosure or discussion that are coupled or directly coupled or communicatively coupled to each other may be an indirect coupling or communicative connection via some interfaces, devices or elements, whether in electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate and are shown as units

May or may not be a physical unit, may be located in one place, or may be distributed over a plurality of 5 network elements. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.

The above description is only of the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think of the changes or substitutions within the technical scope of the present application

Are encompassed within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, comprising:

partitioning a mediastinum region of the lung medical image to obtain at least two mediastinum partition images, wherein different mediastinum partition images comprise different anatomical structures of the mediastinum region;

And according to the at least two mediastinal subarea images, performing lesion detection on the different anatomical structures through at least two branch networks in a neural network model to obtain a lesion detection result of a mediastinal area of the lung medical image, wherein one branch network is at least used for detecting the mediastinal lesion in the anatomical structure of one mediastinal subarea image.

2. The method according to claim 1, wherein each branch network comprises a self-attention network, and wherein the performing lesion detection on the different anatomical structures through at least two branch networks in a neural network model according to the at least two mediastinal subarea images to obtain a lesion detection result of a mediastinal area of the lung medical image comprises:

according to at least two sub-subarea images corresponding to each mediastinal subarea image, obtaining a characteristic diagram corresponding to each mediastinal subarea image through the self-attention network;

acquiring a total feature map according to the feature map corresponding to each mediastinum subarea image;

and performing function operation or convolution operation on the total feature map to obtain a lesion detection result of the mediastinal region of the lung medical image.

3. The method of claim 2, wherein prior to deriving a feature map for each mediastinal subzone image from at least two sub-subzone images for each mediastinal subzone image via the self-attention network, the method further comprises:

Expanding one mediastinal subarea image in two adjacent mediastinal subarea images to the direction of the other mediastinal subarea image to obtain an expanded mediastinal subarea image, wherein the expanded mediastinal subarea image and the other mediastinal subarea image have an overlapping part;

partitioning the expanded mediastinal subarea image and the other mediastinal subarea image to obtain at least two subarea images corresponding to the expanded mediastinal subarea image and at least two subarea images corresponding to the other mediastinal subarea image.

4. The method according to claim 2, wherein the obtaining, by the self-attention network, the feature map corresponding to each mediastinal subarea image according to the at least two subarea images corresponding to each mediastinal subarea image includes:

obtaining a first feature map through the self-attention network according to at least two sub-subarea images corresponding to one mediastinal subarea image in two adjacent mediastinal subarea images;

and obtaining a second feature map through the self-attention network according to the first feature map and at least two sub-subarea images corresponding to the other mediastinal subarea image in the two adjacent mediastinal subarea images.

5. The method of claim 2, wherein when the two adjacent mediastinal subareas have overlapping portions, the overlapping portion of the first feature map corresponds to a first overlapping feature map and the overlapping portion of the second feature map corresponds to a second overlapping feature map, wherein the obtaining a total feature map from the feature maps corresponding to each mediastinal subarea image comprises:

carrying out weighted summation on the first overlapped characteristic diagram and the second overlapped characteristic diagram to obtain an overlapped characteristic diagram;

and merging the total feature map, the feature maps except the first overlapped feature map in the first feature map and the feature maps except the second overlapped feature map in the second feature map to obtain the total feature map.

6. The method according to any one of claims 1 to 5, wherein the sectioning of the mediastinal region of the medical image of the lung to obtain at least two mediastinal sectioning images comprises:

windowing an image of a region of interest corresponding to the lung medical image by using a mediastinum window to obtain the mediastinum region;

windowing an image of a region of interest corresponding to the medical image of the lung by using a bone window to obtain a bone region, wherein the bone region comprises a thoracic vertebra and a vertebra;

And partitioning the mediastinum area by taking a straight line between the thoracic vertebra and the vertebra as a partitioning reference to obtain the at least two mediastinum partition images.

7. The method of claim 6, wherein partitioning the mediastinum region with respect to a line between the thoracic vertebra and the spine to obtain the at least two sub-partitioned images comprises:

taking an area image corresponding to the straight line, which is close to one side of the thoracic vertebra, of a first preset length as an upper mediastinum partition image;

taking an area image corresponding to the straight line, which is close to one side of the vertebra, of a second preset length as a lower mediastinum partition image;

taking the region image between the upper mediastinum partition image and the lower mediastinum partition image as a middle mediastinum partition image,

the upper mediastinum subarea image and the lower mediastinum subarea image adopt the same branch network for lesion detection, and the middle mediastinum subarea image adopts another branch network for lesion detection.

8. The method as recited in claim 6, further comprising:

thresholding is carried out on the lung medical image to obtain a lung field mask;

Expanding the lung field mask outwards to contain the thoracic vertebrae to obtain a region-of-interest mask;

and cutting the lung medical image by using the region-of-interest mask to obtain the region-of-interest image.

9. A method for training a neural network model, comprising:

partitioning a mediastinum area of a lung medical sample image to obtain at least two mediastinum partitioned sample images, wherein the lung medical sample image is marked with a lesion label;

obtaining a lesion prediction result of a mediastinum region of the lung medical sample image through at least two branch networks in a neural network model according to at least two mediastinum partition sample images corresponding to each mediastinum partition sample image, wherein one branch network is at least used for predicting a mediastinum lesion in an anatomical structure of one mediastinum partition sample image;

calculating a loss function value according to the lesion prediction result and the lesion label;

and updating parameters of the neural network model according to the loss function value.

10. An image processing apparatus, comprising:

the device comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is configured to partition a mediastinum region of a lung medical image to obtain at least two mediastinum partition images, and different mediastinum partition images comprise different anatomical structures of the mediastinum region;

And the second acquisition module is configured to perform lesion detection on the different anatomical structures through at least two branch networks in a neural network model according to the at least two mediastinal subarea images to obtain a lesion detection result of a mediastinal region of the 5-lung medical image, wherein one branch network is at least used for detecting mediastinal lesions in the anatomical structure of one mediastinal subarea image.

11. A training device for a neural network model, comprising:

the fourth acquisition module is configured to partition a mediastinum area of the lung medical sample image to obtain at least two mediastinum partition sample images, wherein the lung medical sample image is marked with a lesion label; the training module is configured to obtain a lesion prediction result of a mediastinum region of the lung medical sample image through at least two branch networks in a neural network model according to at least two mediastinum partition sample images corresponding to each mediastinum partition sample image, wherein one branch network is at least used for predicting a mediastinum lesion in an anatomical structure of one mediastinum partition sample image; calculating a loss function value according to the lesion prediction result and the lesion label; and updating parameters of the neural network model 5 according to the loss function value.

12. An electronic device, comprising:

a processor; and

a memory in which computer program instructions are stored which, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 9.