KR20220009451A - Neural network training method and apparatus, electronic device and storage medium - Google Patents

Abstract

The present application relates to a neural network training method and apparatus, an electronic device, and a storage medium, and training a neural network model, classifying an image according to the neural network model obtained by training, the neural network training method comprising: a location of a target region in a sample image obtaining information and category information; dividing and obtaining at least one sample image block according to the location information of the target area in the sample image; classifying at least one sample image block according to the category information to obtain sample image blocks of N categories; and inputting N categories of sample image blocks into the neural network for training.

Description

Neural network training method and apparatus, electronic device and storage medium

Cross-referencing of related applications

This application was filed based on a Chinese patent application with an application number of 202010148544.8 and an filing date of March 5, 2020, and claims the priority of the Chinese patent application, all contents of the Chinese patent application are incorporated herein by reference. do.

The present application relates to the field of computer technology, and more particularly, to a method and apparatus for training a neural network, an electronic device, and a storage medium.

Methods such as machine learning are widely used in the image processing field, and may be applied to, for example, classification and image detection of general images or 3D images. For example, in the processing of medical images, it is possible to classify diseases and detect lesion regions through a machine learning method.

In medical image processing, classification and detection of lung medical images (eg, lung computed tomography (CT)) plays an important role in examination and diagnosis of lesions such as pneumonia and lung cancer. Early detection of preoperative lesion features is of great clinical importance and can provide guidance for clinical decisions. However, due to the lack of radiological features typical of early cancer (clear bubbles, pleural constriction, etc.), it is difficult for a clinician or radiologist to accurately distinguish and diagnose the category of subtype Ground-Glass Nodule (GGN) on CT images. is difficult

Embodiments of the present application provide a method and apparatus for training a neural network, an electronic device, and a storage medium.

An embodiment of the present application provides a neural network training method, wherein the neural network training method trains a neural network model, classifies an image according to a neural network model obtained by training, and the neural network training method includes: a location of a target region in a sample image obtaining information and category information; dividing and obtaining at least one sample image block according to the location information of the target area among the sample images; classifying the at least one sample image block according to the category information to obtain sample image blocks of N categories, where N is an integer, and N≥1; and inputting the N categories of sample image blocks into a neural network for training.

According to the neural network training method of the embodiment of the present application, it is possible to obtain a fine classification of a sample image block, and train the neural network so that the neural network can fine-classify an image, thereby improving classification efficiency and accuracy.

In some embodiments of the present application, the sample image is a medical imaging photograph.

In some embodiments of the present application, the obtaining of the location information and category information of the target area in the sample image includes performing location tracking on the target area of the medical image to obtain location information of the target area. step; acquiring a pathological picture related to the medical image, wherein the pathological picture is a picture including pathological information that has been diagnosed; and determining category information of the target region of the medical image according to pathological information of each target region of the pathological picture.

In some embodiments of the present application, the training by inputting the N categories of sample image blocks to the neural network includes inputting and processing any one sample image block to the neural network, so as to obtain category prediction information and prediction target of the sample image block. acquiring an area; determining a classification loss according to at least the category prediction information and the category information to which the sample image block belongs; determining a division loss according to the prediction target region and position information of the sample image block; and training the neural network according to the classification loss and the segmentation loss.

In some embodiments of the present application, the determining of the classification loss according to the category prediction information and the category information to which the sample image block belongs includes, according to the category prediction information and the category information to which the sample image block belongs, a first determining a classification loss; determining a second classification loss according to the category prediction information and category information of a category center of a category to which the sample image block belongs; and performing weighted summation processing on the first classification loss and the second classification loss to obtain the classification loss.

In this way, the categorical features of the sample image blocks of the same category are more concentrated in the training process, and the feature distance between the category information of the sample image blocks of different categories becomes larger, which is advantageous for improving classification performance and improving classification accuracy. can be improved

In some embodiments of the present application, the determining of the division loss according to the location information of the prediction target area and the sample image block includes: according to a first ratio, the number of pixels of the prediction target area occupied in the sample image block; determining a first weight of the prediction target area and a second weight of a sample background area in the sample image block; and determining the division loss according to the first weight, the second weight, the prediction target region, and the position information of the sample image block.

In some embodiments of the present application, a first weight of the prediction target area and a second weight of a sample background area in the sample image block are determined according to a first ratio that the number of pixels of the prediction target area occupies in the sample image block The method may include: determining a second ratio of a sample background area in the sample image block according to a first ratio in which the number of pixels of the prediction target area occupies in the sample image block; and determining the second ratio as the first weight and determining the first ratio as the second weight.

In this way, it is possible to balance the error in the target region and the error in the non-target region, and to help optimize network parameters, thereby improving training efficiency and training effect.

In some embodiments of the present application, the category information includes invasive prostate cancer atypical adenomatous proliferative nodule, adenocarcinoma in situ, minimally invasive adenocarcinoma nodule, and invasive adenocarcinoma nodule.

In some embodiments of the present application, the neural network includes a shared feature extraction network, a classification network, and a segmentation network, and the neural network training method includes inputting an image block to be processed into the shared feature extraction network and processing it to be processed. obtaining a target feature of an image block, wherein the shared feature extraction network includes M shared feature extraction blocks, and an input feature of an i-th shared feature extraction block includes output features of the first i-1 shared feature extraction blocks and i and M are integers, and 1<i≤M - ; inputting the target feature into the classification network and performing classification processing to obtain category information of the image block to be processed; and inputting the target feature into the segmentation network and performing segmentation processing to obtain a target area among the image blocks to be processed.

In this way, the target feature can be obtained through the shared feature extraction network, and the shared feature extraction block of the shared feature extraction network acquires the output features of all the shared feature extraction blocks before it, and adds its output features to all subsequent It can be entered into the shared feature extraction block. By reinforcing the gradient flow in the network, it is possible to alleviate the gradient annihilation phenomenon, and at the same time improve feature extraction and learning ability, and it helps to more precisely classify and segment the input image block to be processed. It is possible to obtain more detailed category information and a target area of an image block to be processed, and improve image processing efficiency.

In some embodiments of the present application, the step of inputting an image block to be processed into the shared feature extraction network and processing to obtain a target feature of the image block to be processed includes: a first feature extraction process for the image block to be processed to obtain a first characteristic of the image block to be processed; The first feature is input to a first shared feature extraction block to obtain an output feature of the first shared feature extraction block, and the output feature of the first shared feature extraction block is added to subsequent M-1 shared feature extraction blocks. outputting; inputting the output features of the first j-1 shared feature extraction blocks into the j-th shared feature extraction block to obtain the output features of the j-th shared feature extraction block - j is an integer, and 1 < j < M - ; performing a second feature extraction process on the output feature of the M-th shared feature extraction block to obtain a second feature of the image block to be processed; and performing a pooling process on the second feature to obtain the target feature.

In some embodiments of the present application, the neural network training method includes: obtaining a first image by performing pre-processing on an image to be processed; determining location information of the target area in the first image by performing location tracking on the target area of the first image; and dividing and obtaining at least one image block to be processed according to the location information of the target region in the first image.

An embodiment of the present application provides an apparatus for training a neural network, the apparatus for training a neural network trains a neural network model, and classifies an image according to a neural network model obtained by training, and the apparatus for training a neural network includes location information of a target region in a sample image and an acquiring module, configured to acquire category information; a first dividing module, configured to divide and obtain at least one sample image block according to the location information of the target region among the sample images; a classification module, configured to classify the at least one sample image block according to the category information to obtain sample image blocks of N categories, wherein N is an integer and N≥1; and a training module configured to train by inputting the N categories of sample image blocks into a neural network.

An embodiment of the present application provides an electronic device, the electronic device comprising: a processor; and a memory configured to store a computer program executable by the processor, wherein the processor is configured to execute the neural network training method described above by the computer program.

An embodiment of the present application provides a storage medium in which a computer program configured to perform the above-described neural network training method when executed is stored.

An embodiment of the present application provides a computer program including computer readable code, and when the computer readable code is executed in an electronic device, a processor of the electronic device is configured to implement the above-described neural network training method.

It should be understood that the foregoing general description and the following detailed description are illustrative and interpretative only and not limiting of the present application. Other features and aspects of the present application will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Here, the accompanying drawings are included in the specification and form a part of the present specification, and the accompanying drawings illustrate embodiments consistent with the present application, and together with the specification describe the technical solutions of the present application.
1 is a schematic system architecture of a neural network training method provided by an embodiment of the present application.
2 is an implementation flowchart of a neural network training method provided by an embodiment of the present application.
3 is an application schematic diagram of a neural network training method provided by an embodiment of the present application.
4 is a schematic diagram of a neural network training apparatus provided by an embodiment of the present application.
5 is a schematic diagram of an electronic device provided by an embodiment of the present application.
6 is a schematic diagram of another electronic device provided by an embodiment of the present application.

Hereinafter, various exemplary embodiments, features and aspects of the present application will be described in detail with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals indicate elements having the same or similar functions. Although various aspects of the embodiment are shown in the accompanying drawings, the accompanying drawings are not drawn according to proportions unless specifically indicated.

The word 'exemplary' used in its entirety herein means 'used as an example, embodiment, or description'. Any embodiment described herein as 'exemplary' is not necessarily to be construed as superior or superior to another embodiment.

In this specification, the term 'and/or' only describes a related relationship of a related object, and indicates that three relationships may exist. For example, A and/or B may represent these three cases in which A exists alone, A and B exist simultaneously, and B exists alone. Also, in the present specification, 'at least one' refers to any one of a plurality or any combination of at least two of the plurality. For example, including at least one of A, B, and C may indicate including any one or a plurality of elements selected from the set consisting of A, B, and C.

Additionally, in order to better explain the present application, numerous specific details are provided in the following embodiments. It should be understood that a person skilled in the art may practice the present application without some details. In some instances, methods, means, elements, and circuits well known to those of ordinary skill in the art have not been described in detail in order to emphasize the subject matter of the present application.

Before describing the embodiments of the present application in more detail, deficiencies in the related art will be first described.

In a related art, a method such as machine learning is widely used in the image processing field, and may be applied to, for example, classification and image detection of a general image or a 3D image.

Lung cancer is one of the most common malignant tumors in our country, and its mortality rate is the number one among all cancer deaths, regardless of whether it is urban or rural, male or female, where adenocarcinoma accounts for about 40% of all lung cancers. Screening is performed using medical images (eg lung CT and low-dose helix CT), and an increasing number of early lung adenocarcinomas are found and appearing as Ground-Glass Nodule (GGN). Adenocarcinoma is divided into invasive prostate cancer, Atypical Adenomatous Hyperplasia Of Preinvasive Adenocarcinoma (AAHOPA), Adenocarcinoma In Situ (AIS), Minimally Invasive Adenocarcinoma (MIA), and Invasive Adenocarcinoma (IA). . The GGN category of adenocarcinoma includes invasive prostate cancer atypical adenomatous proliferative nodules, adenocarcinoma in situ nodules, minimally invasive adenocarcinoma nodules, and invasive adenocarcinoma nodules. Survival time is significantly reduced as tumor size increases, indicating that early detection and diagnosis are effective and important ways to reduce patient mortality. Therefore, early detection of invasive features before surgery is clinically important and can provide guidance for clinical decisions.

In medical image processing, classification and detection of lung medical images (eg, lung CT) plays an important role in medical screening and diagnosis of diagnoses such as pneumonia and lung cancer. In the related technology, a disease category may be determined and a disease site may be detected through a method such as machine learning. For example, it is possible to predict whether the input nodule image is a malignant tumor or a benign tumor. However, the related art did not classify the prediction results in detail. In addition, it is difficult for clinical experts or radiologists to accurately distinguish and diagnose subtype GGN categories from CT images due to the lack of typical radiological features (clear bubbles, pleural constriction, etc.) of early cancer. In this case, AI-based computer-aided diagnosis is a more effective method to evaluate the invasiveness of nodules, and is expected to play an important role in clinical evaluation work.

Referring to FIG. 1 , FIG. 1 is a schematic system architecture of a neural network training method provided by an embodiment of the present application. As shown in FIG. 1 , the system architecture includes a CT device 100 , a server 200 , a network 300 , and a terminal device 400 . In order to implement and support one exemplary application, the CT device 100 may be connected to the terminal device 400 via the network 300 , and the terminal device 400 may be connected to the server 200 via the network 300 . connected to, the CT device 100 may collect a CT image. For example, it may be a terminal capable of scanning a layer of a certain thickness of a human body, such as an X-ray CT device or a γ-ray CT device. The terminal device 400 may be a device having a screen display function, such as a notebook computer, a tablet PC, a desktop computer, or a dedicated message device. The network 300 may be a wide area network or a LAN, or a combination of both, and implements data transmission using a wireless link.

The server 200 converts the pulmonary nodule region that has undergone each pathology proof among the acquired training medical image photos through a three-dimensional classification framework designed based on the neural network training method provided by the embodiment of the present application into a small image block. After cropping, training data is obtained by classifying image blocks, the training data is input to the neural network to perform detailed classification on the training medical image, and training is performed, and a trained neural network model is obtained after training is completed. The medical image may be a lung CT image of a patient or examinee collected by the CT device 100 of an apparatus such as a hospital or an examination center. The server 200 may acquire a medical image image collected by the CT device 100 from the terminal device 400 as a training medical image image, may acquire a training medical image image from the CT device, or may acquire a training medical image image from the network. Medical imaging pictures may also be obtained.

The server 200 may be an independent physical server, a server cluster consisting of a plurality of physical servers or a distributed system, or a cloud server based on cloud technology. Cloud technology refers to a hosting technology that integrates a series of resources such as hardware, software, and network within a wide area network or LAN to realize data calculation, storage, processing and sharing. In an embodiment of the present application, when the server 200 is a cloud server, the provided artificial intelligence cloud service includes a neural network model, and the neural network trains the neural network based on the fine-classified training data to fine-classify the medical image. can

As an example, the server 200 receives a medical image to be processed (eg, a lung CT image), and then performs classification, segmentation, etc. processing on the medical image according to the trained neural network, so that the microclassified lesion acquire an area Thereafter, the server 200 returns and displays the obtained microclassified lesion area to the terminal device 400 so that the medical staff can see it.

In some embodiments of the present application, the server 200 may transmit the trained neural network to the terminal device 400 after training is completed, and the terminal device 400 collects the collected medical image to be processed (eg, Lung CT image) is classified, segmented, etc. to obtain a microclassified lesion area, and the obtained microclassified lesion area is displayed on its own display screen so that medical staff can see it.

In some embodiments of the present application, the system architecture of the neural network training method includes a CT device 100 , a network 300 , and a terminal device 400 , and training by training a training medical image by the terminal device 400 . After acquiring the neural network, a microclassified lesion area is obtained and obtained by performing classification, segmentation, etc. processing on the medical image to be processed (eg, lung CT image) collected by the terminal device 400 again. The microclassified lesion area is displayed on its display screen so that the medical staff can see it.

An embodiment of the present application provides a neural network training method, the neural network training method is applied to a neural network training device, the neural network training device may be a server, training a neural network model, and training an image according to the acquired neural network model classify The method provided by the embodiment of the present application may be implemented through a computer program, and when the computer program is executed, each step of the neural network training method provided by the embodiment of the present application is completed. In some embodiments, the computer program may be executed by a processor. 2 is an implementation flowchart of a neural network training method provided by an embodiment of the present application, and as shown in FIG. 2 , the neural network training method includes the following steps.

In step S11, location information and category information of the target area in the sample image are acquired.

In step S12, at least one sample image block is obtained by dividing the sample image according to the location information of the target region.

In step S13, the at least one sample image block is classified according to the category information to obtain sample image blocks of N categories, where N is an integer, and N≥1.

In step S14, the N categories of sample image blocks are input to the neural network for training.

According to the neural network training method provided by the embodiment of the present application, it is possible to obtain a fine classification of a sample image block, and train the neural network so that the neural network can fine-classify the image, thereby improving classification efficiency and accuracy.

In some embodiments of the present application, the neural network training method may be performed by a terminal device or other processing device. A terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a mobile phone, a wireless telephone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle device, a wearable device, etc. can The other processing device may be a server or a cloud server or the like. In some embodiments of the present application, the neural network training method may be implemented by calling a computer program stored in a memory through a processor.

In some embodiments of the present application, the sample image is a medical imaging picture, such as a lung CT image. The sample image block may be an image block including a target region in the sample image. In an embodiment, the sample image may be a 3D medical image annotated (eg, category annotation and segmentation annotation), and the sample image block may be an image block in which a nodule is included in the 3D medical image.

In some embodiments of the present application, in step S11 , location information and category information of a target region among the sample images may be determined so as to obtain a sample image block for training the neural network and annotate the sample image block. Step S11 may include: performing location tracking on the target area of the medical image to obtain location information of the target area; acquiring a pathological picture related to the medical image; and determining category information of the target region of the medical image according to pathological information of each target region of the pathological picture. The pathological picture is a picture including pathological information that has undergone diagnosis, and may be acquired from a medical image database, or may be manually annotated by an expert such as a doctor in the terminal and then transmitted to the neural network training apparatus.

In some embodiments of the present application, a 3D image of 1×1×1 resolution may be obtained by performing resampling processing on the sample image. In addition, the 3D image is divided. For example, some regions outside the lung parenchyma may be present in the lung 3D medical image, and lesions such as lung nodules are often present inside the lung parenchyma, so the image block in which the lung parenchyma exists is cut out (i.e. segmented) and the image Normalization processing is performed on the block. In addition, location information of the target area may be obtained by performing location tracking on a target area (eg, a lesion area) in the normalized 3D image. For example, the location information of the target area may be determined through a convolutional neural network for location tracking, or location information of the target area may be confirmed through an expert such as a doctor, and the embodiment of the present application is not limited to the location tracking method. does not

In some embodiments of the present application, the medical image may include a related pathological photograph, and may be used to determine a category of a lesion in the medical image. For example, a category of lesions may include a Ground-Glass Nodule (GGN). Adenocarcinoma is divided into invasive prostate cancer, Atypical Adenomatous Hyperplasia Of Preinvasive Adenocarcinoma (AAHOPA), Adenocarcinoma In Situ (AIS), Minimally Invasive Adenocarcinoma (MIA), and Invasive Adenocarcinoma (IA). However, the embodiments of the present application are not limited to the category of the lesion.

In some embodiments of the present application, pathological information of each target region may be acquired according to the pathological photograph. For example, the pathological picture may be a picture that has undergone a professional diagnosis, and an analysis description for each lesion may be included. By acquiring pathological information of each target region according to the pathological picture, category information of each target region of the medical image may be determined.

In some embodiments of the present application, an image block including a lesion region is cut out from a medical image photograph (ie, a sample image block is cut out), and sample image blocks of N categories are obtained according to category information of the target region can do. For example, through statistics on nodule size, the size of a sample image block can be determined to be 64×64×64, and after cropping and classification, sample image blocks of 4 types (AAHOPA, AIS, MIA and IA) to acquire

In some embodiments of the present application, the cost is high because the number of medical image pictures is small and the difficulty of annotation is large, and when a 3D image is divided into a plurality of 2D images, spatial information is lost and performance is deteriorated. The number of samples can be increased by performing manipulations such as rotation, translation, mirroring, and zooming on the sample image block. In addition, by training the neural network using the increased sample image blocks, the generalization ability of the neural network can be improved and overfitting can be prevented. In some embodiments of the present application, positive and negative samples may be balanced. In an example, there is a large difference in the number of samples of malignant nodules, such as invasive prostate cancer atypical adenomatous proliferative nodules, in situ adenocarcinoma nodules, minimally invasive adenocarcinoma nodules, and invasive adenocarcinoma nodules. The number of negative samples can be balanced. Examples of the present application do not limit the method of increasing the number of samples.

In some embodiments of the present application, the sample image block may be divided and input to the neural network. Step S14 may include inputting one sample image block to the neural network and processing it to obtain category prediction information and a prediction target region of the sample image block; determining a classification loss according to at least the category prediction information and the category information to which the sample image block belongs; determining a division loss according to the prediction target region and position information of the sample image block; and training the neural network according to the classification loss and the segmentation loss.

In some embodiments of the present application, the neural network may include a shared feature extraction network, a classification network, and a segmentation network. By performing feature extraction on the sample image block through the shared feature extraction network, sample target features of the sample image block may be obtained, and category prediction information of the sample image block may be obtained through the classification network. There may be an error in the category prediction information, and the classification loss of the neural network may be determined through the sample prediction category information and the category annotation information of the sample image block.

In some embodiments of the present application, the determining of the classification loss according to the category prediction information and the annotation information of the sample image block includes, according to the category prediction information and the annotation information of the sample image block, a second classification loss. determining; determining a second classification loss according to the category prediction information and category information of a category center of a category to which the sample image block belongs; and performing weighted summation processing on the first classification loss and the second classification loss to obtain the classification loss.

In some embodiments of the present application, the annotation information of the sample image block may include category annotation information. For example, the category annotation information may be information indicating a category of a nodule among the sample image blocks. In an embodiment, the category prediction information may be category information displayed in the form of a vector or the like. It is possible to determine a probability distribution in which the image block to be processed, which is represented by the vector, belongs to each category through a probability dictionary or the like, and further to determine a category to which the image block to be processed belongs. In addition, the vector of the category prediction information may directly indicate the probability of the image block to be processed, and in an embodiment, each element of the vector indicates the probability of the category to which the image block to be processed belongs.

을 결정할 수 있다. 예를 들어, softmaxloss 손실함수에 따라 제1 분류 손실

을 계산할 수 있다. 실예에서, 이하 공식(1)을 통해 제1 분류 손실

을 결정할 수 있다. In some embodiments of the present application, the first classification loss may be determined according to the category prediction information and the category annotation information of the sample image block. For example, determine the feature distance (eg, Euclidean distance, cosine distance, etc.) between the vector of category prediction information and the vector of category annotation information, and the first classification loss according to the feature distance

can be decided For example, the first classification loss according to the softmaxloss loss function

can be calculated. In the example, the first classification loss through the following formula (1)

can be decided

(1);

(One);

는 i 번째 샘플 이미지 블록의 카테고리 예측 정보를 나타내고,

는 i 번째 샘플 이미지 블록이 속한 카테고리를 나타내고, n은 카테고리 수를 나타낸다.

는 완전 연결 계층 중 제

카테고리의 가중치를 나타내고,

는 완전 연결 계층 중 j 번째 카테고리의 가중치를 나타내고, m은 매번 신경망에 입력된 샘플 이미지 블록 수를 나타내고,

는 i 번째 샘플 이미지 블록이 속한 카테고리 바이어스 항목을 나타내고,

는 j 번째 카테고리의 바이어스 항목을 나타낸다. here,

represents the category prediction information of the i-th sample image block,

denotes the category to which the i-th sample image block belongs, and n denotes the number of categories.

is the first of the fully connected layers

indicates the weight of the category,

represents the weight of the j-th category among fully connected layers, m represents the number of sample image blocks input to the neural network each time,

represents the category bias item to which the i-th sample image block belongs,

denotes a bias item of the j-th category.

만 사용하여 훈련된 분류 네트워크의 분류 효과가 좋지 않다. In some embodiments of the present application, when training using the above-described first classification loss, the feature distance between classes of category information of different categories can be expanded, and further, the classification network can distinguish sample image blocks of different categories. make it possible However, the differences between nodules of multiple categories of the lung are not clear (e.g., the differences in the morphology of nodules in in situ and minimally invasive adenocarcinoma are not significant), and because the morphology is different between nodules of the same type (e.g., , the types of malignant nodules such as invasive adenocarcinoma are different), the feature distance between classes of category information is small and the feature distance within the class is large, so the first classification is lost

The classification effect of a classification network trained using only

In some embodiments of the present application, for the above-described problem, a classification network may be trained through a second classification loss. In an embodiment, category information of a category center of each category among a plurality of sample image blocks may be determined. For example, a class center feature may be acquired by performing a weighted average on category information of class centers of a plurality of sample image blocks or performing clustering processing on category information of sample image blocks. The embodiment of the present application does not limit the category information of the class center.

를 결정할 수 있다. 예를 들어, centerloss 손실함수에 따라 제2 분류 손실

를 계산할 수 있다. 제2 분류 손실

를 통해 분류 네트워크를 훈련하면, 동일한 유형의 샘플 이미지 블록의 카테고리 정보의 클래스 내 특징 거리를 단축하여, 동일한 유형의 특징 정보가 특징 공간에서 더 집중되도록 할 수 있으며, 샘플 이미지 블록의 카테고리를 결정하는데 도움이 된다. 실예에서, 이하 공식(2)를 통해 제2 분류 손실

을 결정할 수 있다.In some embodiments of the present application, the second classification loss may be determined according to the category prediction information of the sample image block and the category annotation information of the category center of the category to which it belongs. For example, determine the feature distance between the category prediction information and the category information of the class center, and the second classification loss according to the feature distance

can be decided For example, the second classification loss according to the centerloss loss function

can be calculated. Second Classification Loss

By training the classification network through , it is possible to shorten the within-class feature distance of the category information of the sample image block of the same type, so that the feature information of the same type is more concentrated in the feature space, and to determine the category of the sample image block. It helps. In the example, the second classification loss through the following formula (2)

can be decided

(2);

(2);

는 i 번째 샘플 이미지 블록이 속한 카테고리의 범주 센터의 카테고리 주석 정보이다. here,

is category annotation information of the category center of the category to which the i-th sample image block belongs.

In some embodiments of the present application, the classification loss may be jointly determined through the first classification loss and the second classification loss. For example, a weighted summation process may be performed on the first classification loss and the second classification loss to obtain a classification loss. For example, the weight ratio of the first classification loss and the second classification loss is 1:0.8, and after performing weighted summation according to the above-described weight ratio, the classification loss may be obtained. The embodiment of the present application does not limit the weight ratio.

In this way, the categorical features of sample image blocks of the same category are more concentrated in the training process, and the distance between category information of sample image blocks of different categories becomes larger, which is advantageous for improving classification performance and improving classification accuracy. can do it

In some embodiments of the present application, segmentation processing may be performed on a sample target feature through a segmentation network to obtain a prediction target area in a sample image block. There may be an error in the prediction target region, and a division loss may be determined according to an error between the prediction target region and the annotation target region of the sample image block, and further, training may be performed through the division loss.

In some embodiments of the present application, the determining of the division loss according to the annotation information of the image of the prediction target area and the sample image block includes: the number of pixels of the prediction target area in a first ratio occupied by the sample image block Accordingly, determining a first weight of the prediction target region and a second weight of a sample background region in the sample image block according to the first weight, the second weight, and annotation information of the prediction target region and the sample image block , determining the splitting loss.

In some embodiments of the present application, the annotation information may include an annotated segmentation region, and the segmentation loss may be directly determined according to an error between the prediction target region and the annotated segmentation region. However, in general, the diameter of the nodule is between 5 millimeters (mm) and 30 mm, and the difference in the ratio between the area where the nodule is located and other areas in the sample image block is large. , the ratio of the error of the prediction target region to the segmentation loss is small, which does not help in the optimization control of the neural network, so the training efficiency is low and the training effect is bad.

In some embodiments of the present application, weight processing may be performed on pixels of a target area and pixels of a non-target area (ie, a sample background area). In an embodiment, the first weight of the prediction target area and the second weight of the sample background area in the sample image block may be determined according to a first ratio that the number of pixels of the prediction target area occupies in the sample image block. Further, when determining the division loss, weighting may be performed on the pixels of the two regions to balance the loss of the target region and the loss of the non-target region.

In some embodiments of the present application, a first weight of the prediction target area and a second weight of a sample background area in the sample image block are determined according to a first ratio that the number of pixels of the prediction target area occupies in the sample image block determining a second ratio of a sample background area in the sample image block according to a first ratio that the number of pixels of the prediction target area occupies in the sample image block; and using the second ratio as the first weight and determining the first ratio as a second weight.

In some embodiments of the present application, the sample image block may include a prediction target area and a background area, and a ratio occupied by the number of pixels of the prediction target area may be statistic to determine the ratio occupied by the sample background area. For example, if the first ratio occupied by the number of pixels in the prediction target region is 0.2, the second ratio occupied by the number of pixels in the sample background region is 0.8. Examples of the present application do not limit the first ratio and the second ratio.

In some embodiments of the present application, in order to balance the prediction target area and the sample background area, the second ratio is determined as the first weight of the prediction target area, and the first ratio is determined as the second weight of the sample background area. For example, if the first ratio occupied by the number of pixels in the prediction target region is 0.2, the first weight of the prediction target region is 0.8, and when the second ratio occupied by the number of pixels in the sample background region is 0.8, the second weight of the sample background region is 0.2 to be.

을 결정한다. 예를 들어, weightedDiceloss 손실함수에 따라 분할 손실

을 계산할 수 있다. 실예에서, 이하 공식 (3)을 통해 분할 손실

을 결정할 수 있다.In some embodiments of the present application, the division loss may be determined according to the first weight, the second weight, the prediction target area, and the annotation target area of the sample image block. In an embodiment, the division loss may be determined according to a difference between the prediction target region and the target region among the annotation information. For example, weighting is performed on pixel points in the prediction target area, and the weight is a first weight, weighting is performed on pixel points in the sample background area, and the weight is a second weight, and division loss after weighting

to decide For example, split loss according to the weightedDiceloss loss function

can be calculated. In the example, the division loss through the following formula (3)

can be decided

(3);

(3);

이고,

=1이면, 제k 픽셀 위치가 예측 목표 영역임을 나타내고,

=0이면, 제k 픽셀 위치가 샘플 배경 영역임을 나타내고,

는 분할 네트워크가 제k 픽셀 위치에서의 출력을 나타내고, W는 제1 가중치를 나타내고, Y는 제k 픽셀 위치의 분할 주석을 나타낸다. here,

ego,

=1, indicates that the k-th pixel position is the prediction target area,

=0, indicates that the kth pixel position is the sample background area,

denotes the output of the segmentation network at the kth pixel position, W denotes the first weight, and Y denotes the segmentation annotation at the kth pixel position.

In this way, it is possible to balance the error of the target region and the error of the non-target region, it is advantageous for network parameter optimization, and training efficiency and training effect can be improved.

을 결정할 수 있다. In some embodiments of the present application, a shared feature extraction network, a segmentation network, and an aggregate network loss of the classification network may be determined according to the classification loss and the segmentation loss. For example, weighted summation processing may be performed on classification loss and segmentation loss to obtain a comprehensive network loss. In the example, the total network loss according to the formula (4) below

can be decided

(4);

(4);

은

의 가중치를 나타내고,

는

의 가중치를 나타내고,

은

의 가중치를 나타낸다. 예를 들어,

=1.2,

=0.8,

=2이며, 본 출원의 실시예는 분류 손실 및 분할 손실의 가중치에 대해 한정하지 않는다.here,

silver

represents the weight of

Is

represents the weight of

silver

represents the weight of For example,

=1.2,

=0.8,

=2, and the embodiment of the present application does not limit the weighting of the classification loss and the division loss.

In some embodiments of the present application, a network parameter of the above-described neural network may be adjusted inversely through a comprehensive network loss. For example, by adjusting the network parameters through gradient descent, the network parameters can be optimized, and the segmentation and classification accuracy can be improved.

In some embodiments of the present application, the above-described training method may be repeatedly performed several times, and is trained according to a set learning rate. ^{In an example, you can train with a learning rate of 0.001*1.1 x} (where x represents the training period) in all 20 training cycles, and in subsequent training the learning rate is halved in each of the 40th, 80th and 120th training cycles. can be reduced to In the initial stage of training, the training efficiency is improved so that the network parameters are greatly optimized, and in the subsequent training, the learning rate is gradually reduced to fine-tune the network parameters, improve the accuracy of the neural network, and improve the accuracy of classification processing and segmentation processing. can

In some embodiments of the present application, when a training condition is satisfied, training may be completed to obtain a trained shared feature extraction network, a segmentation network, and a classification network. The training condition may include a training order. That is, when the preset training order is reached, the training condition is satisfied. The training condition may include that the overall network loss is less than or equal to a preset threshold or converges in a preset section. That is, when the overall network loss is less than or equal to a preset threshold or converges to a preset section, the accuracy of the neural network can be considered to meet the usage requirements, and training can be completed. The embodiments of the present application do not limit the training conditions.

In some embodiments of the present application, after training is completed, the trained neural network may be tested. For example, the 3D image block including the nodule region in the 3D lung 3D medical image may be input to the above-described neural network, and the accuracy of the output segmentation result and classification result may be statistic. For example, if the accuracy of the segmentation result and the classification result is confirmed by comparing the 3D image block with annotation information, the training effect of the neural network may be determined. If the accuracy is higher than the preset threshold, it can be considered that the training effect is good and the neural network performance is good, and it can be used for processing to obtain the category of the image block to be processed and segment the target region. If the accuracy does not reach the preset threshold, the training effect is considered poor, and training can continue using other sample image blocks.

In some embodiments of the present application, when the trained neural network does not know both the target region and the category among the image blocks to be processed, it can acquire the category and target region of the image block to be processed, and knows the category of the image block to be processed. In this case, only the target region among the image blocks to be processed may be acquired, or if the target region among the image blocks to be processed is known, the category of the image block to be processed may be acquired. The embodiments of the present application do not limit the method of using the neural network.

In some embodiments of the present application, a neural network trained through the above-described training method may be used to determine a lesion region and a lesion category among image blocks to be processed. The neural network includes a shared feature extraction network, a classification network and a segmentation network, and the neural network training method includes: inputting and processing an image block to be processed into a shared feature extraction network to obtain a target feature of the image block to be processed; The shared feature extraction network includes M shared feature extraction blocks, an input feature of an i-th shared feature extraction block includes output features of the first i-1 shared feature extraction blocks, i and M are integers, and 1 <i≤M - ; inputting the target feature into a classification network and performing classification processing to obtain category information of the image block to be processed; further comprising steps.

In this way, the target feature can be obtained by applying the shared feature extraction network, and the shared feature extraction block of the shared feature extraction network can acquire the output features of all previous shared feature extraction blocks, and its output features can be entered into all subsequent shared feature extraction blocks. By reinforcing the gradient flow in the network, it is possible to alleviate the gradient extinction phenomenon, at the same time improve feature extraction and learning capabilities, and help to more precisely classify and segment the input image block to be processed. It is possible to obtain more detailed category information and a target area of an image block to be processed, and improve image processing efficiency.

In some embodiments of the present application, the image block to be processed may be a partial region of the image to be processed. In an embodiment, some area may be cropped from the image to be processed. For example, an area including a target object may be cut out. For example, the image to be processed is a medical image, and a region including a lesion may be cut out from the medical image. For example, the image to be processed may be a lung three-dimensional medical image (eg, a lung CT image), and the image block to be processed is It may be a 3D image block. The embodiments of the present application do not limit the types of images to be processed and image blocks to be processed.

In some embodiments of the present application, in a medical image (eg, a 3D medical image of the lung), the size and resolution of the medical image are high, and in the medical image, the area of normal tissue is large, so the medical image is It is possible to improve the processing efficiency by performing pre-processing on the lesion and cutting and processing the area containing the lesion.

In some embodiments of the present application, the neural network training method performs pre-processing on an image to be processed to obtain a first image Position tracking is performed on a target area of the first image, and a target area of the first image The method further includes: determining location information of , and dividing and obtaining at least one image block to be processed according to location information of a target region in the first image.

In some embodiments of the present application, first, preprocessing may be performed on an image to be processed to improve processing efficiency. For example, preprocessing such as resampling and normalization can be performed. In an embodiment, the resampling process may be performed on the lung 3D medical image to obtain a 3D image with a resolution of 1×1×1 (that is, each pixel represents the content of a cube with a size of 1 mm×1 mm×1 mm) . In addition, by reducing the size of the resampled 3D image (for example, some non-pulmonary regions may exist in the lung 3D medical image), the region in which the lungs are located can be cropped, saving computation and improving processing efficiency. .

In an embodiment, processing efficiency may be improved by normalizing the cropped 3D image to normalize the pixel value of each pixel in the 3D image within a value range of 0 to 1. After performing normalization processing, the first image is acquired. Examples of the present application do not limit the method of pretreatment.

In some embodiments of the present application, a target area in the first image may be detected. For example, a target region in the first image may be detected through a convolutional neural network for position detection. In an embodiment, a region including a nodule may be detected in a lung 3D medical image using a convolutional neural network.

In some embodiments of the present application, an image block to be processed may be obtained by cropping the target area. For example, an image block to be processed may be obtained by cutting out a region including a nodule in the lung 3D medical image. In an embodiment, the size of the image block to be processed may be determined and cropped according to the size of the nodule. For example, the size of the image block to be processed may be determined as 64×64×64 through statistics on the nodule size, and after cropping, one or a plurality of image blocks to be processed may be obtained.

In some embodiments of the present application, category information of an image block to be processed may be determined and a target region may be segmented through the neural network. For example, the image block to be processed is a cropped image block including nodules in the lung 3D medical image. The type of nodule (eg, AAHOPA, AIS, MIA, and IA) among the image blocks to be processed may be determined through the neural network, and the region in which the nodule is located may be divided.

In some embodiments of the present application, target features of an image block to be processed may be extracted through a shared feature extraction network for use in classification and segmentation processing. The step of inputting the image block to be processed into the shared feature extraction network to process it to obtain a target feature of the image block to be processed includes: performing a first feature extraction processing on the image block to be processed, so that the first feature of the image block to be processed Input the first feature into the first shared feature extraction block to obtain the output feature of the first shared feature extraction block, and add the output feature of the first shared feature extraction block to the subsequent M-1 shared feature extraction blocks Step of outputting the output features of the first j-1 shared feature extraction blocks to the j-th shared feature extraction block to obtain the output features of the j-th shared feature extraction block To the output features of the M-th shared feature extraction block performing a second feature extraction process on the image block to obtain a second feature of the image block to be processed; and performing a pooling process on the second feature to obtain the target feature.

In some embodiments of the present application, first, a first feature extraction process is performed to include, for example, a three-dimensional convolutional layer, a batch normalization layer, and an activation layer. The first feature may be obtained by performing the first feature extraction process through the network module. The embodiment of the present application does not limit the network layer of the first feature extraction process.

In some embodiments of the present application, the shared feature extraction network may include a plurality of shared feature extraction blocks, and the shared feature extraction block may include a plurality of network layers (eg, convolutional layer, activation layer, etc.). Also, the embodiment of the present application is not limited to the network layer included in the shared feature extraction block. The first feature may be processed through a plurality of shared feature extraction blocks. In an embodiment, the number of shared feature extraction blocks is M, and the first feature may be input into the first shared feature extraction block. That is, the first shared feature extraction block may use the first feature as an input feature, and perform feature extraction processing on the input feature to obtain an output feature. The output features of the first shared feature extraction block can be shared by all subsequent shared feature extraction blocks. That is, the output feature of the first shared feature extraction block may be output to the subsequent M-1 shared feature extraction blocks and become input features of the subsequent M-1 shared feature extraction blocks.

In some embodiments of the present application, the input feature of the second shared feature extraction block is an output feature of the first shared feature extraction block, and the second shared feature extraction block performs feature extraction processing on the input feature, The output feature may be output to subsequent third to M-th shared feature extraction blocks to be input features of the third to M-th shared feature extraction blocks.

In some embodiments of the present application, the input features of the third shared feature extraction block are output features of the first shared feature extraction block and output features of the second shared feature extraction block. The output features of the first shared feature extraction block and the output features of the second shared feature extraction block are combined with the third shared feature after feature integration (e.g., integrating by calculating average, maximum, etc. or withholding all feature channels). input into the extraction block (i.e., the input features of the third shared feature extraction block may be the features after the output features of the first shared feature extraction block and the output features of the second shared feature extraction block are integrated), or the third shared feature extraction block The feature extraction block has both the output features of the first shared feature extraction block and the output features of the second shared feature extraction block as input features (for example, the third shared feature extraction block may include a feature integration layer; Feature integration processing can be performed by the corresponding layer or all feature channels can be withheld, and subsequent processing can be directly performed on features of all feature channels, that is, the output feature of the first shared feature extraction block and the second shared feature Subsequent processing is performed on output features of the feature extraction block), and feature extraction processing is performed on input features (e.g., direct feature extraction processing is performed on features of all feature channels, or features on integrated features performing extraction processing), to obtain the output feature of the third shared feature extraction block. The output feature of the third shared feature extraction block may be output to the fourth to M-th shared feature extraction blocks, and may be input features of the fourth to M-th shared feature extraction blocks.

In some embodiments of the present application, taking the j-th (j is an integer, and 1<j<M) shared feature extraction block as an example, the output feature of the first j-1 shared feature extraction blocks is the j-th shared feature as an input feature may be input to the feature extraction block. After feature integration is performed on the output features of the first j-1 shared feature extraction blocks, the integrated features are used as input features of the j-th shared feature extraction block, or output features of the first j-1 shared feature extraction blocks are used. It can be directly as the input feature of the j-th shared feature extraction block (e.g., perform integration within the j-th shared feature extraction block, or perform subsequent processing directly on the features of all feature channels. That is, the first j -1 perform subsequent processing on the output features of the shared feature extraction block). The j-th shared feature extraction block performs feature extraction processing on the input feature, obtains the output feature of the j-th shared feature extraction block, and sets the corresponding output feature as the input feature of the j+1-th to M-th shared feature extraction block can be done with

In some embodiments of the present application, the M-th shared feature extraction block may acquire output features of the M-th shared feature extraction block according to output features of the first M-1 shared feature extraction blocks. In addition, by performing the second feature extraction processing through a subsequent network layer of the shared feature extraction network (eg, the Nth shared feature extraction block through a network module including a three-dimensional convolution layer, a batch normalization layer, and an activation layer) performing a second feature extraction process on the output features of ), and a second feature may be obtained. The embodiment of the present application does not limit the network layer of the second feature extraction process.

In some embodiments of the present application, a pooling process may be performed on the second feature. For example, the target feature may be obtained by performing a pooling process on the second feature through the average pooling layer. The embodiments of the present application do not limit the types of pooling processing.

In some embodiments of the present application, the above-described processing may be performed multiple times. For example, it may include a plurality of shared feature extraction networks. The first shared feature extraction network has a first feature as an input feature, and after performing feature extraction processing, second feature extraction processing, and pooling processing of the shared feature extraction block, the output feature of the first shared feature extraction network can be obtained In the second shared feature extraction network, the output features of the first shared feature extraction network are input features, and after the feature extraction processing of the shared feature extraction block, the second feature extraction processing, and the pooling processing are performed, the second shared feature extraction The output feature of the network may be acquired, ..., processed through a plurality of shared feature extraction networks, and the output feature of the last (eg, fourth) shared feature extraction network may be set as the target feature. Embodiments of the present application do not limit the number of shared feature extraction networks.

In this way, the target feature can be obtained through the shared feature extraction network, and the shared feature extraction block of the shared feature extraction network acquires the output features of all the shared feature extraction blocks before it, and adds its output features to all subsequent It can be entered into the shared feature extraction block. By reinforcing the gradient flow in the network, it is possible to alleviate the gradient annihilation phenomenon and at the same time improve the feature extraction and learning ability, and it is advantageous for more precisely classifying and segmenting the input image block to be processed.

In some embodiments of the present application, category information of an image block to be processed may be determined according to a target feature (eg, an image block to be processed may be an image block including a lesion such as a nodule among lung 3D medical images) ), it is possible to determine the category of the nodule according to the target characteristics. In an embodiment, the category of a nodule may be determined as an atypical adenomatous proliferative nodule, an in situ adenocarcinoma nodule, a minimally invasive adenocarcinoma nodule, or an invasive adenocarcinoma nodule.

In some embodiments of the present application, classification processing may be performed on a target feature through a classification network to obtain category information of an image block to be processed. In an embodiment, the classification network may include a plurality of network layers (eg, a convolutional layer, a global average pooling layer, a fully connected layer, etc.), and the above-described network layer is Classification processing can be performed on the target feature, and category information can be output. The category information may be category information displayed in the form of a vector or the like. It is possible to determine a probability distribution that the image block to be processed, which is represented by the vector, belongs to each category through a probability dictionary or the like, and further determines category information of the image block to be processed. Alternatively, the vector of category information may directly indicate the probability of the image block to be processed. In an embodiment, each element of the vector may represent a probability of a category to which an image block to be processed each belongs. For example, (0.8, 0.8, 0.1) indicates that the probability that the image block to be processed belongs to the first category is 0.8, the probability that it belongs to the second category is 0.1, and the probability that the image block to be processed belongs to the third category is 0.1, and the probability is The largest category may be determined as the category of the image block to be processed. That is, category information of the image block to be processed may be determined as the first category. The embodiment of the present application does not limit the method of displaying category information.

In some embodiments of the present application, category information of an image block to be processed may be determined according to a target feature (for example, an image block to be processed may be an image block including a lesion such as a nodule among lung 3D medical images) ), the location of the nodule can be determined according to the target characteristics and the region where the tooth is located can be divided.

In some embodiments of the present application, segmentation processing may be performed through a segmentation network to obtain a target area among image blocks to be processed. For example, the target area may be divided. In an embodiment, the split network may include a plurality of network layers, such as an upsample layer and a fully connected layer. In an embodiment, the target feature is a feature map obtained through processing such as feature extraction and pooling for an image block to be processed in the shared feature extraction network, and the resolution of the target feature may be lower than that of the image block to be processed. The upsampling process is performed through the upsampling layer to reduce the number of feature channels of the target feature and improve the resolution, so that the feature map output by the segmentation network and the resolution of the image block to be processed match. For example, if the shared feature extraction network performs the pooling process 4 times, the upsampling process is performed 4 times through the upsampling layer so that the feature map output by the segmentation network and the resolution of the image block to be processed match can In addition, the target region may be segmented from the output feature map of the segmentation network. For example, the target area in which the nodule is located may be determined through the contour line or the contour surface. Embodiments of the present application do not limit the network layer of the divided network.

In some embodiments of the present application, after segmenting the target region in the image block to be processed, the position of the target region in the image to be processed may be determined. For example, according to the position of the image block to be processed in the image to be processed and the position of the target region among the image blocks to be processed, the position of the target region in the image to be processed may be restored. In an embodiment, in the lung medical image, the position of the nodule among the image blocks to be processed may be divided, and the position of the nodule may be restored in the pulmonary medical image.

According to the neural network training method provided by the embodiment of the present application, it is possible to obtain a fine classification of a sample image block, and train the neural network so that the neural network can fine-classify the image, thereby improving classification efficiency and accuracy. In addition, the target feature can be obtained through the shared feature extraction network, and the shared feature extraction block of the shared feature extraction network acquires the output features of all previous shared feature extraction blocks, and extracts its output features from all subsequent shared features By entering a block, it is possible to alleviate the gradient disappearance phenomenon by strengthening the gradient flow in the network, and at the same time, it can improve the feature extraction and learning ability, and it helps to more precisely classify and segment the input image block to be processed. do. In addition, it is possible to obtain more detailed category information and target area of an image block to be processed, and improve image processing efficiency. In addition, as the category information of the sample image blocks of the same category is more concentrated in the training process, the feature distance between the category information of the sample image blocks of different categories can be increased, and the error of the target area and the error of the non-target area are balanced This can help to improve classification performance and improve classification accuracy.

3 is an application schematic diagram of a neural network training method provided by an embodiment of the present application. As shown in FIG. 3 , the sample image 31 is a medical image, and the sample image block 32 is an image block including a lesion (eg, nodule) cut out from the medical image. In addition, the sample image block may include category annotations. For example, the sample image block may include four categories: AAHOPA, AIS, MIA, and IA.

In some embodiments of the present application, a sample image block 32 is input to the neural network 33 , and a shared feature extraction network 331 included in the neural network 33 performs feature extraction on each batch of sample image blocks. Thus, the sample target feature of the sample image block can be obtained, and the category prediction information of the sample image block can be obtained through the classification network 332 included in the neural network 33, and formulas (1) and (2) are Through this, it is possible to determine the classification loss of the neural network. In some embodiments of the present application, the segmentation network 333 included in the neural network 33 may obtain a prediction target region among the sample image blocks 32, and may determine the segmentation loss of the neural network according to formula (3). have. We can perform weighted summation on the division loss and classification loss to obtain the integrative network loss of the neural network, and train the neural network with the integrative network loss. The trained neural network may be used to determine a lesion region and a lesion category among image blocks of a medical image.

In some embodiments of the present application, the image to be processed may be a three-dimensional lung medical image (eg, lung CT image), and the image block to be processed is a lesion region (eg, including a nodule) cut out from the image to be processed. area) may be a 3D image block.

In some embodiments of the present application, a resampling process is performed on a 3D medical image to obtain a 3D image of 1×1×1 resolution, a region in which the lung is located may be cut out, and further normalization is performed on the region in which the lung is located can do. In some embodiments of the present application, a region in which the nodule is located among the regions in which the lung is located may be detected and a plurality of image blocks to be processed including the region in which the nodule is located may be cut out to a size of 64×64×64.

In some embodiments of the present application, a target feature of the image block to be processed may be obtained by performing feature extraction processing by dividing a plurality of image blocks to be processed. For example, first, a first feature extraction process is performed (eg, a first feature extraction process is performed through a network module including a three-dimensional convolutional layer, a batch normalization layer, and an activation layer), and the first feature can be obtained.

In some embodiments of the present application, the first feature may be input to the shared feature extraction network. The shared feature extraction network may include a plurality of shared feature extraction blocks. In an embodiment, the number of shared feature extraction blocks is M, and the first feature can be input into the first shared feature extraction block and processed, and the output features of the first shared feature extraction block are added to the subsequent M-1 shared feature extraction blocks. can be entered. The input feature of the second shared feature extraction block is the output feature of the first shared feature extraction block, and the second shared feature extraction block may output the output feature to subsequent third to M-th shared feature extraction blocks. The input features of the third shared feature extraction block are the output features of the first shared feature extraction block and the output features of the second shared feature extraction block, and the output features of the third shared feature extraction block are the fourth to Mth shared feature extraction blocks can be output to Similarly, the output features of the first j-1 shared feature extraction blocks may be input to the j-th shared feature extraction block, and the output features of the j-th shared feature extraction block may be input to the j+1-th to M-th shared feature extraction blocks. can be output. The M-th shared feature extraction block may acquire output features of the M-th shared feature extraction block according to the output features of the first M-1 shared feature extraction blocks, and perform a second feature extraction process. For example, the second feature may be obtained by performing second feature extraction processing on the output feature of the N-th shared feature extraction block through a network module including a three-dimensional convolutional layer, a batch normalization layer, and an activation layer. have. In some embodiments of the present application, a pooling (eg, average pooling) process may be performed on the second feature to obtain a target feature.

In some embodiments of the present application, the above-described processing may be performed multiple times (eg, 4 times). For example, it may include a plurality of shared feature extraction networks. A target feature may be obtained through processing of a plurality of cascaded shared feature extraction networks.

In some embodiments of the present application, the classification network may perform classification processing on the target feature to obtain category information of the image block to be processed. For example, the classification network may obtain category information of an image block to be processed through a convolutional layer, a global average pooling layer, and a fully connected layer.

In some embodiments of the present application, the segmentation network may perform segmentation processing on the target feature to obtain a target area (ie, a region in which a nodule is located). In an embodiment, the segmentation network performs the upsampling process 4 times through the upsampling layer so as to match the resolution of the image block to be processed with the feature map output by the segmentation network, and selects the target region from the output feature map of the segmentation network. can be divided

In some embodiments of the present application, when the above-described neural network does not know both the target region and the category of the image block to be processed, it may acquire the category and the target region of the image block to be processed (eg, the region in which the nodule is located) segment and obtain the category of the nodule), if the category of the image block to be processed is known, only the target region of the image block to be processed may be acquired (for example, by dividing the region where the nodule is located), or When the target area among the image blocks to be processed is known, the category of the image block to be processed (eg, the category of the nodule is determined) may be acquired.

In some embodiments of the present application, the image processing method may be used for segmentation and classification of a lesion region in a medical image, such as a lung CT image, and may improve clinical work efficiency and reduce diagnosis omissions and misdiagnosis. Also, it may be used for classification of other images and segmentation of a target region, and the embodiments of the present application are not limited to the field of application of the image processing method.

It should be understood that each of the above-described method embodiments mentioned in the present application may be combined with each other to form a combined embodiment without violating the principle logic. In addition, the embodiments of the present application further provide devices, electronic devices, computer-readable storage media, and programs, and all of the above may implement any one method provided by the embodiments of the present application, and corresponding technical solutions and For the description, reference is made to the corresponding description in the Methods section. A person skilled in the art will know that the description order of each step in the above-described method embodiment does not mean a strict execution order, does not constitute any limitation on the implementation process, and the execution order of each step is determined by its function and possible inherent logic understand that it should be

4 is a schematic diagram of a neural network training apparatus provided by an embodiment of the present application. As shown in FIG. 4 , the neural network training apparatus includes: an acquisition module 11 configured to acquire location information and category information of a target region in a sample image; a first dividing module 12, configured to divide and obtain at least one sample image block according to the location information of the target region among the sample images; a classification module 13, configured to classify the at least one sample image block according to the category information to obtain sample image blocks of N categories, where N is an integer, and N≥1; and a training module 14, configured to train by inputting the N categories of sample image blocks into a neural network.

In some embodiments of the present application, the sample image is a medical imaging photograph.

In some embodiments of the present application, the acquisition module 11 also performs location tracking on a target area of the medical image to obtain location information of the target area and obtains a pathological photograph related to the medical image. to acquire - the pathological photograph is a photograph containing the diagnosed pathological information; and determine category information of the target region of the medical image according to the pathological information of each target region of the pathological picture.

In some embodiments of the present application, the training module 14 also inputs any one sample image block to the neural network and processes it to obtain category prediction information and prediction target region of the sample image block, and at least the category prediction information and a classification loss is determined according to the category information to which the sample image block belongs, a division loss is determined according to the prediction target region and location information of the sample image block, and the neural network is constructed according to the classification loss and the division loss. designed to train

In some embodiments of the present application, the training module 14 is further configured to determine a first classification loss according to the category prediction information and the category information to which the sample image block belongs, and determine whether the category prediction information and the sample image block are determine a second classification loss according to the category information of the category center of the category to which it belongs, and perform weighted summing processing on the first classification loss and the second classification loss to obtain the classification loss.

In some embodiments of the present application, the training module 14 is further configured to: according to a first ratio that the number of pixels of the prediction target area occupies in the sample image block, in the first weight of the prediction target area and the sample image block and determine a second weight of the sample background region, and determine the division loss according to the first weight, the second weight, the prediction target region, and the position information of the sample image block.

In some embodiments of the present application, the training module 14 is further configured to determine, according to a first ratio in which the number of pixels of the prediction target region occupies in the sample image block, a second ratio of a sample background area in the sample image block. and determine the second ratio as the first weight, and determine the first ratio as the second weight.

In some embodiments of the present application, the category information includes invasive prostate cancer atypical adenomatous proliferative nodule, adenocarcinoma in situ, minimally invasive adenocarcinoma nodule, and invasive adenocarcinoma nodule.

In some embodiments of the present application, the neural network includes a shared feature extraction network, a classification network, and a segmentation network, and the neural network training apparatus inputs and processes an image block to be processed into the shared feature extraction network to process an image to be processed an acquiring module, configured to acquire a target feature of a block, wherein the shared feature extraction network includes M shared feature extraction blocks, and an input feature of an i-th shared feature extraction block obtains an output feature of the first i-1 shared feature extraction blocks wherein i and M are integers, and 1<i≤M; a classification module, configured to input the target feature into the classification network and perform classification processing to obtain category information of the image block to be processed and a partitioning module configured to obtain

In some embodiments of the present application, the acquiring module is further configured to perform a first feature extraction process on the to-be-processed image block to acquire a first feature of the to-be-processed image block and to share the first feature with the first input to the feature extraction block to obtain the output features of the first shared feature extraction block, output the output features of the first shared feature extraction block to subsequent M-1 shared feature extraction blocks, and output the first j-1 shared features input the output feature of the feature extraction block to the j-th shared feature extraction block to obtain the output feature of the j-th shared feature extraction block, where j is an integer, and 1<j<M; perform a second feature extraction process on the output feature of the M-th shared feature extraction block to obtain a second feature of the to-be-processed image block, and perform a pooling process on the second feature to obtain the target feature is composed

In some embodiments of the present application, the neural network training apparatus is a pre-processing module configured to perform pre-processing on an image to be processed to obtain a first image Position tracking is performed on a target area of the first image, and the first image Position tracking module, configured to determine position information of a target region in the first image, a second dividing module configured to divide and obtain at least one image block to be processed according to position information of a target region in the first image.

In some embodiments, a function or a module included in the apparatus provided by the embodiment of the present application may be configured to perform the method described in the above method embodiment, the implementation of which follows the description of the above method embodiment. Please refer to

An embodiment of the present application further provides a computer-readable storage medium in which a computer program for performing the above-described method is stored when executed. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

An embodiment of the present application further provides an electronic device, wherein the electronic device includes a memory configured to store a computer program executable by a processor processor, wherein the processor is configured to perform the above-described method through the computer program . The electronic device may be provided as a terminal, server, or other type of device.

An embodiment of the present application further provides a computer program product including a computer readable code, wherein when the computer readable code is executed in a device, the processor of the device executes the instructions of the neural network training method provided by any one of the embodiments described above. run

Embodiments of the present application further provide another computer program product, configured to store computer-readable instructions that, when executed, cause a computer to perform operations of the neural network training method provided by any one of the above-described embodiments.

5 is a schematic diagram of an electronic device provided by an embodiment of the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a message transmitting/receiving device, a game machine, a tablet PC, a medical device, an exercise device, a PDA, and the like. Referring to FIG. 5 , the electronic device 800 includes a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , and an Input/Output (I). /O) one or more of an interface 812 , a sensor component 814 , and a communication component 816 .

The processing component 802 generally controls the overall operation of the electronic device 800 , such as operations related to displays, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 executing instructions to complete all or some steps of the method described above. Further, processing component 802 may include one or more modules to facilitate interaction with other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802 .

Memory 804 is configured to store various types of data to support operation in electronic device 800 . Examples of such data include commands, contact data, phone book data, messages, pictures, moving pictures, and the like of any application or method operating in the electronic device 800 . Memory 804 may be implemented in any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), PROM (Programmable Read Only Memory, PROM), ROM (Read Only Memory, ROM), magnetic storage device, flash, disk, and optical disk.

The power component 806 supplies power to various components of the electronic device 800 , and includes a power management system, one or more power sources, and other components related to generating, managing, and allocating power for the electronic device 800 . may include

The multimedia component 808 includes a screen that provides one output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TouchPanel, TP). When the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to detect touch, sliding and gestures on the touch panel. The touch sensor may not only detect the boundary of a touch or slide action, but also detect a duration and pressure associated with the touch or slide action. In some embodiments, multimedia component 808 includes one front camera and/or one rear camera. When the electronic device 800 is in an operation mode such as a photographing mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras may be a single fixed optical lens system or may have focus and optical zoom capabilities.

The audio component 810 is configured to output and/or input an audio signal. For example, the audio component 810 includes one microphone (Microphone, MIC), and when the electronic device 800 is in an operation mode such as a call mode, a recording mode, and a voice recognition mode, the microphone receives an external audio signal. configured to receive The received audio signal may be stored in memory 804 or transmitted by communication component 816 . In some embodiments, the audio component 810 further includes one speaker configured to output an audio signal.

I/O interface 812 provides an interface between processing component 802 and a peripheral interface module, which may be a keyboard, click wheel, button, or the like. Such buttons include, but are not limited to, a home page button, a volume button, a start button, and a lock button.

The sensor component 814 includes one or a plurality of sensors configured to provide the terminal device 800 with an assessment of the status of each side. For example, the sensor component 814 may detect an on/off state of the electronic device 800 and a relative position of the component. For example, the component is a display device and a keypad of the electronic device 800 . The sensor component 814 may also include a change in the position of the electronic device 800 or one component of the electronic device 800 , whether the user and the electronic device 800 are in contact, the orientation or acceleration/deceleration of the electronic device 800 , and the electronic device A temperature change of 800 may be detected. The sensor component 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor component 814 may further include an optical sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, and may be used in imaging applications. In some embodiments, the sensor component 814 may include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate communication between the electronic device 800 and another device in a wired or wireless manner. The electronic device 800 may access a wireless network based on a communication standard such as WiFi, 2G or 3G, or a combination thereof. In some embodiments of the present application, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system through a broadcast channel. In some embodiments of the present application, the communication component 816 may further include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module has a radio frequency identification (Radio Frequency) technology, an infrared communication standard (Infrared Data Association, IrDA) technology, an Ultra Wide Band (UWB) technology, and a Bluetooth (BT) technology. and other technologies.

In some embodiments of the present application, the electronic device 800 includes one or a plurality of Application Specific Integrated Circuits (ASICs), digital signal processors (DSPs), and digital signal processing devices (Digital Signal Processes). Device, DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a controller, a microcontroller, a microprocessor or other electronic device to perform the method described above. can be configured.

In some embodiments of the present application, a non-volatile computer-readable storage medium (eg, a memory 804 containing computer program instructions) is further provided. The above-described computer program instructions may be executed by the processor 820 of the electronic device 800 to complete the above-described method.

6 is a schematic diagram of another electronic device provided by an embodiment of the present application. For example, the electronic device 1900 may be provided as a server. Referring to FIG. 6 , the electronic device 1900 includes a processing component 1922 , and the processing component 1922 may further include one or a plurality of processors and a memory resource representative of the memory 1932 . . Memory resources are configured to store instructions (eg, applications) that can be executed by processing component 1922 . Applications stored in memory 1932 may include one or more modules each corresponding to a set of instructions. Further, processing component 1922 is configured to execute instructions to perform the method described above.

The electronic device 1900 includes one power component 1926 configured to perform power management of the electronic device 1900 , one wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and one may further include an input output (I/O) interface 1958 of The electronic device 1900 may operate based on an operating system stored in the memory 1932 , such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

In some embodiments of the present application, a non-volatile computer-readable storage medium (eg, a memory 1932 containing computer program instructions) is further provided. The computer program instructions described above may be executed by the processing component 1922 of the electronic device 1900 to complete the method described above.

The present application may be a system, method and/or computer program product. The computer program product may include a computer readable storage medium having recorded thereon computer readable program instructions configured to cause a processor to implement each aspect of the present application.

The computer-readable storage medium may be a tangible device that holds and stores instructions used by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. Computer-readable storage media include portable computer disks, hard disks, RAM (Random Access Memory, RAM), ROM (ROM), EPROM (EPROM or Flash), SRAM, CD-ROM (Compact Disk-Read Only Memory, CD-ROM) , a Digital Video Disc (DVD), a memory stick, a floppy disk, a punch card on which instructions are stored, or a mechanical encoding device such as a protruding structure in a groove, and any suitable combination thereof. As used herein, a computer-readable storage medium includes radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or electrical signals transmitted over electric wires. It is not interpreted as a signal itself at the same moment.

The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing/processing device, or may be downloaded to an external computer or external storage device through a network such as the Internet, a LAN, a wide area network and/or a wireless network. have. The network may include copper transport cables, fiber optic transport, wireless transport, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and communicates the computer readable program instructions for storage in a computer readable storage medium in each computing/processing device.

The computer program instructions configured to perform the operations of the present application may include assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or one or more programming instructions. It can be source code or object code written in any combination of languages. The programming languages include object-oriented programming languages such as Smalltalk, C++, and the like, and general procedural programming languages such as "C" language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a separate software package, partly on the user's computer, partly on a remote computer, or on a completely remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any kind of network (including a LAN or wide area network), or it may be connected to an external computer (eg, via the Internet using an Internet service provider). connected). In some embodiments, the electronic circuitry is personalized using state information in computer readable program instructions. For example, Programmable Logic Circuits, FPGAs or Programmable Logic Arrays (PLAs). The electronic circuitry may execute computer readable program instructions to implement each aspect of the present application.

Here, each aspect of the present application has been described with reference to flowcharts and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present application. It will be understood that each block in the flowcharts and/or block diagrams and combinations of blocks in the flowcharts and/or block diagrams may all be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device, such that the instructions, when executed by the processor of the computer or other programmable data processing device, are either in flow diagrams and/or block diagrams. or create a machine to create a device that implements the functions/operations specified in a plurality of blocks. These computer readable instructions may be stored in a computer readable storage medium. These instructions cause the computer, programmable data processing device, and/or other device to operate in a particular manner. Accordingly, a computer readable storage medium having instructions stored thereon includes a product including instructions for implementing each aspect of a function/action defined in one or a plurality of blocks of the flowchart and/or block diagrams.

The computer readable program instructions may also be loaded into a computer, other programmable data processing device, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing device, or other device to create a computer-implemented process. can do. Accordingly, instructions executed in a computer, other programmable data processing device, or other device may implement functions/operations specified in one or a plurality of blocks in the flowchart and/or block diagram.

The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to a plurality of embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent one module, program segment, or part of an instruction. Some of the modules, program segments or instructions include one or a plurality of executable instructions configured to implement defined logical functions. In some alternative implementations, the functions shown in the blocks may also occur in an order other than that shown in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, sometimes in reverse order depending on the function involved. It should also be noted that each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, is a dedicated hardware-based system for performing specified functions or operations, or a combination of dedicated hardware and computer instructions. can be implemented.

The computer program product may be implemented in hardware, software, or a combination thereof. In the embodiment of the present application, the computer program product may be implemented as a computer storage medium. In another embodiment, the computer program product may be implemented as a software product, such as a Software Development Kit (SDK).

In the above, each embodiment of the present application has been described, but the above description is exemplary, not exhaustive, and is not limited to each disclosed embodiment. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of each described embodiment. The terminology used herein is selected to best describe the principle, practical application, or technical improvement of the related art of each embodiment, or to enable others skilled in the art to understand each embodiment disclosed herein.

Industrial Applicability

The present application relates to a method and apparatus for training a neural network, an electronic device, and a storage medium, wherein the method for training a neural network includes obtaining location information and category information of a target area from among sample images. According to the location information of the target area, at least one sample Classifying at least one sample image block according to the category information to obtain N categories of sample image blocks by dividing the image blocks include According to the neural network training method of the embodiment of the present application, it is possible to obtain a fine classification of a sample image block, and train the neural network so that the neural network can fine-classify an image, thereby improving classification efficiency and accuracy.

Claims (25)

A method for training a neural network, comprising:
The neural network training method trains a neural network model, classifies images according to the neural network model obtained by training, and the neural network training method comprises:
obtaining location information and category information of a target area in a sample image;
dividing and obtaining at least one sample image block according to the location information of the target area among the sample images;
classifying the at least one sample image block according to the category information to obtain sample image blocks of N categories, where N is an integer, and N≥1;
and training by inputting the N categories of sample image blocks into the neural network. According to claim 1,
The sample image is a neural network training method, characterized in that the medical image. 3. The method of claim 2,
The step of obtaining location information and category information of the target area in the sample image includes:
obtaining location information of the target area by performing location tracking on the target area of the medical image;
acquiring a pathological picture related to the medical image, wherein the pathological picture is a picture including pathological information that has been diagnosed;
and determining category information of the target region of the medical image according to pathological information of each target region of the pathological picture. According to claim 1,
The step of training by inputting the N categories of sample image blocks into the neural network,
inputting one sample image block to the neural network and processing it to obtain category prediction information and a prediction target region of the sample image block;
determining a classification loss according to at least the category prediction information and the category information to which the sample image block belongs;
determining a division loss according to the prediction target region and position information of the sample image block;
and training the neural network according to the classification loss and the segmentation loss. 5. The method of claim 4,
Determining a classification loss according to the category prediction information and the category information to which the sample image block belongs includes:
determining a first classification loss according to the category prediction information and the category information to which the sample image block belongs;
determining a second classification loss according to the category prediction information and category information of a category center of a category to which the sample image block belongs;
and performing weighted summation processing on the first classification loss and the second classification loss to obtain the classification loss. 5. The method of claim 4,
Determining a division loss according to the location information of the prediction target area and the sample image block includes:
determining a first weight of the prediction target area and a second weight of a sample background area in the sample image block according to a first ratio that the number of pixels of the prediction target area occupies in the sample image block;
and determining the division loss according to the first weight, the second weight, the prediction target region, and the position information of the sample image block. 7. The method of claim 6,
determining the first weight of the prediction target area and the second weight of the sample background area in the sample image block according to a first ratio that the number of pixels of the prediction target area occupies in the sample image block;
determining a second ratio of a sample background area in the sample image block according to a first ratio in which the number of pixels of the prediction target area occupies in the sample image block;
and determining the second ratio as the first weight and determining the first ratio as the second weight. 8. The method according to any one of claims 1 to 7,
The category information is a neural network training method, characterized in that it includes an invasive prostate cancer atypical adenomatous proliferative node, an in situ adenocarcinoma node, a minimally invasive adenocarcinoma node, and an invasive adenocarcinoma node. 9. The method according to any one of claims 1 to 8,
the neural network includes a shared feature extraction network, a classification network and a segmentation network;
The neural network training method is
inputting and processing an image block to be processed into the shared feature extraction network to obtain a target feature of the image block to be processed; The input features of the block include the output features of the first i-1 shared feature extraction blocks, where i and M are integers, and 1<i≤M;
inputting the target feature into the classification network and performing classification processing to obtain category information of the image block to be processed;
The method further comprising the step of inputting the target feature into the segmentation network and performing segmentation processing to obtain a target region among the image blocks to be processed. 10. The method of claim 9,
The step of inputting the image block to be processed into the shared feature extraction network and processing it to obtain the target feature of the image block to be processed comprises:
performing a first feature extraction process on the image block to be processed to obtain a first feature of the image block to be processed;
The first feature is input to the first shared feature extraction block to obtain the output feature of the first shared feature extraction block, and the output feature of the first shared feature extraction block is added to the subsequent M-1 shared feature extraction blocks. outputting;
inputting the output features of the first j-1 shared feature extraction blocks into the j-th shared feature extraction block to obtain the output features of the j-th shared feature extraction block - j is an integer, and 1 < j < M - ;
performing a second feature extraction process on the output feature of the M-th shared feature extraction block to obtain a second feature of the image block to be processed;
and performing a pooling process on the second feature to obtain the target feature. 10. The method of claim 9,
obtaining a first image by performing pre-processing on the image to be processed;
determining location information of the target area in the first image by performing location tracking on the target area of the first image;
The method of training a neural network according to claim 1, further comprising: dividing and obtaining at least one image block to be processed according to the location information of the target region in the first image. A neural network training device comprising:
The neural network training device trains a neural network model, and classifies the image according to the neural network model obtained by training, the neural network training device comprising:
an acquiring module, configured to acquire location information and category information of the target region in the sample image;
a first dividing module, configured to divide and obtain at least one sample image block according to location information of a target region among the sample images;
a classification module, configured to classify the at least one sample image block according to the category information to obtain sample image blocks of N categories, wherein N is an integer and N≥1;
and a training module configured to train by inputting the N categories of sample image blocks into the neural network. 13. The method of claim 12,
The sample image is a neural network training apparatus, characterized in that the medical image. 14. The method of claim 13,
The acquisition module is also
performing location tracking on the target area of the medical image to obtain location information of the target area;
acquiring a pathological picture related to the medical image, wherein the pathological picture is a picture including pathological information that has been diagnosed;
and determine category information of the target region of the medical image according to the pathological information of each target region of the pathological picture. 13. The method of claim 12,
The training module also includes:
input one sample image block to the neural network and process it to obtain category prediction information and a prediction target region of the sample image block;
determine a classification loss according to at least the category prediction information and the category information to which the sample image block belongs;
determine a division loss according to the location information of the prediction target area and the sample image block;
and train the neural network according to the classification loss and the segmentation loss. 16. The method of claim 15,
The training module also includes:
determine a first classification loss according to the category prediction information and the category information to which the sample image block belongs;
determine a second classification loss according to the category prediction information and category information of a category center of a category to which the sample image block belongs;
and performing weighted summing processing on the first classification loss and the second classification loss to obtain the classification loss. 16. The method of claim 15,
The training module also includes:
determine a first weight of the prediction target area and a second weight of a sample background area in the sample image block according to a first ratio that the number of pixels of the prediction target area occupies in the sample image block;
and determine the division loss according to the first weight, the second weight, the prediction target region, and the position information of the sample image block. 18. The method of claim 17,
The training module also includes:
determine a second ratio of a sample background area in the sample image block according to a first ratio that the number of pixels of the prediction target area occupies in the sample image block;
and determine the second ratio as the first weight, and determine the first ratio as the second weight. 19. The method according to any one of claims 12 to 18,
The category information is a neural network training apparatus, characterized in that it includes an invasive prostate cancer atypical adenomatous proliferative node, an in situ adenocarcinoma node, a minimally invasive adenocarcinoma node, and an invasive adenocarcinoma node. 20. The method according to any one of claims 12 to 19,
the neural network includes a shared feature extraction network, a classification network and a segmentation network;
The neural network training device,
an acquiring module, configured to input and process an image block to be processed into the shared feature extraction network to obtain a target feature of the image block to be processed, wherein the shared feature extraction network includes M shared feature extraction blocks, the i-th sharing The input features of the feature extraction block include the output features of the first i-1 shared feature extraction blocks, where i and M are integers, and 1<i≤M;
a classification module configured to input the target feature into the classification network and perform classification processing to obtain category information of the image block to be processed;
and a segmentation module configured to input the target feature into the segmentation network and perform segmentation processing to obtain a target region among the image blocks to be processed. 21. The method of claim 20,
The acquisition module is also
performing a first feature extraction process on the image block to be processed to obtain a first feature of the image block to be processed;
The first feature is input to the first shared feature extraction block to obtain the output feature of the first shared feature extraction block, and the output feature of the first shared feature extraction block is added to the subsequent M-1 shared feature extraction blocks. output;
input the output features of the first j-1 shared feature extraction blocks into the j-th shared feature extraction block to obtain the output features of the j-th shared feature extraction block, where j is an integer, and 1<j<M;
performing a second feature extraction process on the output feature of the M-th shared feature extraction block to obtain a second feature of the image block to be processed;
and performing a pooling process on the second feature to obtain the target feature. 21. The method of claim 20,
a pre-processing module, configured to perform pre-processing on the image to be processed to obtain a first image;
a location tracking module configured to perform location tracking on a target area of the first image to determine location information of a target area in the first image;
The neural network training apparatus according to claim 1, further comprising: a second division module configured to divide and obtain at least one image block to be processed according to the location information of the target region in the first image. As an electronic device,
processor;
a memory configured to store a computer program executable by the processor;
12. Electronic device, characterized in that the processor is configured to execute the neural network training method according to any one of claims 1 to 11 through the computer program. A computer-readable storage medium having a computer program stored therein, comprising:
12. Computer-readable storage medium, characterized in that the computer program is configured to execute the neural network training method according to any one of claims 1 to 11 when the computer program is executed. A computer program comprising computer readable code, comprising:
A computer program, characterized in that when the computer readable code is executed in an electronic device, a processor in the electronic device is configured to implement the method of training a neural network according to any one of claims 1 to 11.

Images