WO2021233021A1 - 一种图像区域分割模型训练方法、分割方法和装置 - Google Patents

一种图像区域分割模型训练方法、分割方法和装置 Download PDF

Info

Publication number
WO2021233021A1
WO2021233021A1 PCT/CN2021/087128 CN2021087128W WO2021233021A1 WO 2021233021 A1 WO2021233021 A1 WO 2021233021A1 CN 2021087128 W CN2021087128 W CN 2021087128W WO 2021233021 A1 WO2021233021 A1 WO 2021233021A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
sample image
vertices
annotation information
structure data
Prior art date
Application number
PCT/CN2021/087128
Other languages
English (en)
French (fr)
Inventor
张军
田宽
颜克洲
姚建华
韩骁
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Publication of WO2021233021A1 publication Critical patent/WO2021233021A1/zh
Priority to US17/707,045 priority Critical patent/US20220222932A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06V10/7747—Organisation of the process, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/162—Segmentation; Edge detection involving graph-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
    • G06V20/695—Preprocessing, e.g. image segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • This application relates to the field of artificial intelligence, in particular to image region segmentation technology.
  • the image segmentation technology is a technology that divides an image into a number of specific areas with unique properties and proposes objects of interest from them. For example, in a medical image segmentation scene, the lesion in the medical image can be segmented for further analysis.
  • Deep learning is now widely used in the field of image region segmentation.
  • image-level labeling methods are used in related technologies to complete image region segmentation using weakly-supervised modes.
  • the supervision signal is too weak, and it is difficult to accurately segment the target area of interest.
  • the present application provides an image region segmentation model training method, image region segmentation method and device, which improve the accuracy of the trained image region segmentation model on the premise of realizing large-scale and rapid labeling, and then When the image region segmentation model obtained by training is used for image region segmentation, the accuracy of image segmentation is improved.
  • an embodiment of the present application provides a method for training an image region segmentation model, which is executed by a data processing device, and the method includes:
  • the sample image set includes at least one sample image, each sample image has its corresponding first annotation information;
  • the target sample image in the sample image set For the target sample image in the sample image set, generate graph structure data corresponding to the target sample image; the graph structure data includes a plurality of vertices, and each vertex includes at least one pixel in the target sample image Point; the target sample image is any sample image in the sample image set;
  • the second annotation information of the vertex is determined according to the graph structure data corresponding to the target sample image and the first annotation information corresponding to the target sample image;
  • the granularity is smaller than the granularity of the first annotation information;
  • the graph convolutional network model is a part of the graph region segmentation model;
  • an embodiment of the present application provides a device for training an image region segmentation model.
  • the device includes an acquiring unit, a generating unit, a determining unit, and a training unit:
  • the acquiring unit is configured to acquire a sample image set, the sample image set includes at least one sample image, and each sample image has its corresponding first annotation information;
  • the generating unit is configured to generate graph structure data corresponding to the target sample image for the target sample image in the sample image set;
  • the graph structure data includes a plurality of vertices, and each of the vertices includes At least one pixel in the target sample image;
  • the target sample image is any sample image in the sample image set;
  • the determining unit is configured to determine the second annotation information of the vertex according to the graph structure data corresponding to the target sample image and the first annotation information corresponding to the target sample image through a graph convolutional network model;
  • the granularity of the second annotation information is smaller than the granularity of the first annotation information;
  • the graph convolutional network model is a part of the graph region segmentation model;
  • the training unit is configured to train the image region segmentation model according to the second annotation information.
  • an embodiment of the present application provides an image region segmentation method, which is executed by a data processing device, and the method includes:
  • the graph structure data corresponding to the image to be segmented is generated through the image region segmentation model; the graph structure data includes a plurality of vertices, and each of the vertices includes at least one pixel in the image to be segmented; the image region segmentation
  • the model is trained based on the second annotation information, the second annotation information is determined based on the graph structure data corresponding to the sample image and the first annotation information corresponding to the sample pattern; the granularity of the second annotation information is smaller than the first annotation information corresponding to the sample pattern. State the granularity of the first annotation information;
  • the target region in the image to be segmented is obtained by segmentation according to the graph structure data corresponding to the image to be segmented.
  • an embodiment of the present application provides an image region segmentation device, which includes an acquisition unit, a generation unit, and a segmentation unit:
  • the acquiring unit is used to acquire an image to be divided
  • the generating unit is configured to generate graph structure data corresponding to the image to be divided through an image region segmentation model;
  • the graph structure data includes a plurality of vertices, and each of the vertices includes at least one pixel in the image to be divided Point;
  • the image region segmentation model is obtained by training according to the second annotation information, the second annotation information is determined according to the image structure data corresponding to the sample image and the first annotation information corresponding to the sample pattern; the first 2.
  • the granularity of the annotation information is smaller than the granularity of the first annotation information;
  • the segmentation unit is configured to obtain the target area in the image to be segmented by segmenting the image region segmentation model according to the graph structure data corresponding to the image to be segmented.
  • an embodiment of the present application provides a medical device that includes an image acquisition module, an image processing module, and an image display module:
  • the image acquisition module is used to obtain an image to be segmented; the image to be segmented is a pathological image including biological tissue;
  • the image processing module is configured to generate graph structure data corresponding to the image to be divided through an image region segmentation model;
  • the graph structure data includes a plurality of vertices, and each of the vertices includes at least one of the images to be divided One pixel;
  • the image region segmentation model is obtained by training according to the second annotation information, and the second annotation information is determined according to the graph structure data corresponding to the sample image and the first annotation information corresponding to the sample pattern;
  • the granularity of the second annotation information is smaller than the granularity of the first annotation information;
  • the lesions in the image to be segmented are obtained by segmentation according to the graph structure data corresponding to the image to be segmented through the image region segmentation model;
  • the image display module is used to display the lesion.
  • an embodiment of the present application provides a device including a processor and a memory:
  • the memory is used to store program code and transmit the program code to the processor
  • the processor is configured to execute the method described in the first aspect or the third aspect according to instructions in the program code.
  • an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect or the third aspect.
  • embodiments of the present application provide a computer program product, including instructions, which when run on a computer, cause the computer to execute the method described in the first aspect or the third aspect.
  • a sample image set is obtained.
  • the sample image set includes at least one sample image, and each sample image has its corresponding first annotation information, which may be Large-scale annotation information such as image level is so easy to implement large-scale and rapid annotation.
  • first annotation information such as image level is so easy to implement large-scale and rapid annotation.
  • For the target sample image in the sample image set (it can be any sample image in the sample image set), generate graph structure data corresponding to the target sample image.
  • the graph structure data includes multiple vertices, and each vertex includes the target sample image At least one pixel in.
  • the second annotation information of the vertex is determined according to the graph structure data corresponding to the target sample image and the first annotation information corresponding to the target sample image.
  • the granularity of the second annotation information is smaller than the granularity of the first annotation information. Since the vertex is actually a super pixel point, it includes at least one pixel point, that is, the second annotation information is a super-pixel-level annotation. In this way, the image region segmentation model including the graph convolutional network model is divided according to the second annotation information. For training, the intervention based on pixel-level annotation can achieve strong supervision and improve the accuracy of the trained image region segmentation model.
  • this application converts image-level annotation information into super-pixel-level annotation information based on the graph convolutional network model on the premise of realizing large-scale and rapid annotation, thereby achieving strong supervised model training, improving the accuracy of the model, and then When using the trained model for image region segmentation, the accuracy of image segmentation is improved.
  • FIG. 1 is a schematic diagram of an application scenario of an image region segmentation model training method provided by an embodiment of the application
  • FIG. 2 is a flowchart of an image region segmentation model training method provided by an embodiment of the application
  • FIG. 3 is a system flowchart of an image region segmentation model training method provided by an embodiment of the application
  • FIG. 4 is a flowchart of an image region segmentation method provided by an embodiment of the application.
  • FIG. 5 is a system flowchart of an image region segmentation method provided by an embodiment of this application.
  • FIG. 6A is a segmentation effect diagram obtained by segmenting an image region segmentation method provided by an embodiment of the application.
  • Figure 6B is a segmentation effect diagram obtained by segmentation through a supervised algorithm
  • FIG. 7 is a flowchart of an image region segmentation method provided by an embodiment of the application.
  • FIG. 8 is a structural diagram of an image region segmentation model training device provided by an embodiment of the application.
  • FIG. 9 is a structural diagram of an image region segmentation device provided by an embodiment of the application.
  • FIG. 10 is a structural diagram of a terminal device provided by an embodiment of this application.
  • FIG. 11 is a structural diagram of a server provided by an embodiment of the application.
  • Deep learning is now widely used in the field of image segmentation.
  • accurate pixel-level annotation is often required, but pixel-level manual annotation is extremely time-consuming and labor-intensive. For example, it usually takes 5-30 minutes to manually label a lesion in a 2048*2048 case picture. Therefore, generating a large number of labeled sample images becomes very expensive and time-consuming.
  • the weakly supervised segmentation method came into being.
  • the weakly supervised segmentation method may adopt, for example, a Class Activation Mapping (CAM) algorithm.
  • CAM Class Activation Mapping
  • the weakly supervised segmentation methods in related technologies usually use image-level tags (usually image categories) related to the segmentation task to train the classification model, and then use the trained classification model to determine the target area that needs to be segmented, such as in the medical field. Lesions. Because image-level labels are relatively rough compared to pixel-level labels, it is difficult to accurately label images, and there is no direct pixel-level intervention. As a result, the image region segmentation results obtained by the weakly-supervised method training model are often not accurate enough. .
  • an embodiment of the present application provides a method for training an image region segmentation model.
  • the method still uses image-level annotation for sample images, but during the model training process, image-level annotation information is based on the graph convolutional network model (first Annotation information) is converted into super-pixel-level annotation information (second annotation information), and the second annotation information is used to train the image region segmentation model, so as to achieve strong supervision of model training and improve the accuracy of the image region segmentation model obtained by training. Therefore, when the image area segmentation model is used for image area segmentation, the accuracy of image segmentation can be improved.
  • the image region segmentation model training method and image region segmentation method provided by the embodiments of the application can be applied to a data processing device with a graphics processing unit (GPU).
  • the data processing device may be a terminal device, such as It can be a computer, a personal digital assistant (PDA), a tablet computer, a smart phone, etc.
  • PDA personal digital assistant
  • the data processing device may also be a server.
  • the server may be an independent server, a cluster server or a cloud server.
  • the server can complete the training of the image region segmentation model, and then use the server to interact with the terminal device to segment the target region of the image to be segmented obtained from the terminal device, and return the segmentation result to the terminal equipment.
  • FIG. 1 is a schematic diagram of an application scenario of an image region segmentation model training method provided by an embodiment of the application.
  • the terminal device 101 is included in this application scenario.
  • the terminal device 101 acquires a sample image set, and the sample image set includes a plurality of sample images with first annotation information.
  • the sample image includes the target area that the user is interested in, and the sample image can be various types of images, such as pathological images, traffic monitoring images, and so on.
  • the above-mentioned first labeling information is labeling information with larger granularity, such as image-level labeling information.
  • the target sample image in the sample image set For the target sample image in the sample image set, generate graph structure data corresponding to the target sample image, which includes multiple vertices, and each vertex includes at least one pixel in the target sample image.
  • the target sample image may be in the sample image set. Any sample image.
  • the graph structure data is composed of at least one pixel in the target sample image
  • the graph structure data composed of these pixels is usually non-standard geometric data, so the graph structure data needs to be processed through the graph convolutional network model. That is, in this embodiment, the graph convolutional network model in the image region segmentation model can be used to determine the second annotation information of the vertices in the graph structure data according to the graph structure data and the first annotation information.
  • the vertices are actually superpixels, which include at least one pixel in the target sample image, that is, the second annotation information is a superpixel-level annotation, and the granularity of the second annotation information is significantly smaller than the granularity of the first annotation information.
  • the image region segmentation model including the graph convolutional network model is trained according to the second annotation information, and the intervention based on superpixel-level annotation can achieve strong supervision and improve the accuracy of the model.
  • the method provided in the embodiments of the present application can be applied to a variety of application scenarios, such as segmentation of medical images in the medical field to distinguish various tissues of the human body, segmentation of target regions (such as lesions) in pathological images, and traffic Field of vehicle recognition and so on.
  • Fig. 2 shows a flowchart of a method for training an image region segmentation model, and the method includes:
  • the sample image set includes a plurality of sample images, and each sample image has its corresponding first annotation information.
  • the first annotation information is annotation information with a larger granularity, such as image-level annotation information.
  • Image-level labeling information only needs to label the entire sample image, and one sample image corresponds to a label. It does not need to label each pixel in the sample image, which can save labor and time costs due to labeling, and facilitate large-scale rapid labeling.
  • the image-level annotation information may include multiple types.
  • the image-level annotation information may be, for example, the area proportion of the target area in the sample image, that is, the first annotation information includes the area proportion of the target area in the sample image.
  • the target area may be an area that may be of interest to the user in the sample image, such as a lesion in a pathological image; the area proportion of the target area in the sample image may be manually estimated from the sample image.
  • mark At the same time as the area ratio, the estimated error value can also be marked, so that the estimated error value can be considered in the subsequent model training process, and the accuracy of model training can be improved.
  • graph structure data corresponding to the target sample image (the target sample image can be any sample image in the sample image set) is generated.
  • the vertices of the graph structure data include at least one pixel in the target sample image, and which pixels each vertex includes can be determined by clustering the pixels in the target sample image, and the basis of the clustering can be the characteristics of the pixels, that is, Pixels with similar characteristics can be clustered together as vertices.
  • the image region segmentation model may include Fully Convolutional Networks (FCN).
  • FCN Fully Convolutional Networks
  • the way to generate graph structure data can be to extract the features of the target sample image through the full convolutional network to obtain the features corresponding to each pixel in the target sample image, and then, according to the features corresponding to the pixel points, compare the features of the target sample image The pixel points are clustered to obtain the superpixel division result, and the graph structure data is constructed according to the superpixel division result, and each superpixel point in the superpixel division result is used as a vertex in the graph structure data.
  • FCN Fully Convolutional Networks
  • Figure 3 shows the system flow of the image region segmentation model training method, which mainly includes a feature extraction module 301 and a classification module 302. Among them, the feature extraction module 301 is used to convert the sample image
  • the target sample images in the set are input to the full convolutional network, and the characteristics of each pixel in the target sample image are obtained.
  • the feature corresponding to each pixel can be represented by an N-dimensional feature vector, where N is the number of channels of the full convolutional network.
  • the embodiment of the present application can select a conventional fully convolutional network, such as U-Net, which is a network structure of a fully convolutional network.
  • U-Net is a network structure of a fully convolutional network.
  • other network structures can also be selected.
  • the network structure is not limited.
  • each pixel has its corresponding feature
  • the pixels in the target sample image can be clustered according to the feature of the pixel, and the pixels with similar features are clustered together to form a superpixel.
  • the features of pixels belonging to the same cell are generally similar and can be clustered together to form a super pixel; the features of pixels belonging to the same part (for example, all belong to the mouth) are generally similar and can be clustered together Form a super pixel, and so on.
  • the clustering method may include multiple types, such as simple linear iterative clustering (SLIC), normalized segmentation algorithm, gradient-based algorithm, etc., which are not limited in the embodiment of the present application.
  • SLIC simple linear iterative clustering
  • normalized segmentation algorithm normalized segmentation algorithm
  • gradient-based algorithm etc.
  • the superpixel division needs to use the features of the pixels output by the full convolutional network as the basis for division. Therefore, as long as the features change, the division of superpixels Will change accordingly.
  • the parameters of the full convolutional network in the image region segmentation model will be adjusted, which will result in changes in the output characteristics, which will further drive changes in the results of the superpixel segmentation. In other words, in this embodiment What you get is dynamic super pixel points, which are more helpful to improve the accuracy of model training.
  • the graph structure data can be constructed according to the superpixel division result, and each superpixel point in the superpixel division result is regarded as a vertex in the graph structure data and enters the subsequent classification module 302.
  • edges In addition to multiple vertices (Nodes) included in the graph structure data, there may also be edges and weights between some vertices.
  • An edge indicates that two vertices are adjacent in space and have a certain association relationship; the weight of an edge indicates the degree of association between the two vertices, and the greater the weight, the greater the degree of association.
  • the method of constructing graph structure data according to the result of superpixel division may be to determine the edge between the vertices according to the position information of the pixel points included in any two vertices. Normally, if two super pixels (vertices) are spatially connected, then an edge connection can be established between the two super pixels. The way of judging whether two super-pixels (vertices) are spatially connected may be judging whether there are adjacent pixels among the pixels included in the two super-pixels.
  • the weight of the edge is determined according to the Euclidean distance between the corresponding features of the first vertex and the second vertex.
  • the larger the Euclidean distance the smaller the degree of association and the smaller the weight. Setting the weights in this way can more accurately reflect the degree of association between superpixels, thereby helping to improve the accuracy of subsequent processing.
  • the method for determining the weights of edges can also be simplified.
  • the weights of all edges can be set to a uniform value.
  • the weights of each edge can be set to 1 uniformly.
  • the feature corresponding to the vertex is the average value of the feature of the vertex including the pixel points. If the feature of the pixel is an N-dimensional feature vector, the feature of the vertex is also an N-dimensional feature vector.
  • S203 Determine the second annotation information of the vertex according to the graph structure data corresponding to the target sample image and the first annotation information corresponding to the target sample image through the graph convolutional network model.
  • the graph structure data can be input to the graph convolutional network model to train the image region segmentation model including the graph convolutional network model .
  • the graph convolutional network model can be composed of five linear graph convolutional layers, and the number of neurons in each linear layer is 64, 128, 256, 64, 1, respectively. Except for the last layer, a nonlinear activation layer (ReLu) is added to each layer.
  • ReLu nonlinear activation layer
  • the graph convolutional network model can determine the second annotation information of the vertex according to the graph structure data and the first annotation information. Since the graph structure data includes multiple vertices, the vertex includes at least one pixel. , The second annotation information is pixel-level annotation information, and the granularity of the second annotation information is smaller than the granularity of the first annotation information.
  • the first labeling information is a pre-labeled real label
  • the second labeling information is a pseudo label, which is the graph convolutional network model included in the image region segmentation model to predict the graph structure data based on the first
  • the second annotation information can provide a stronger supervision signal for the pixel-level annotation information obtained by the information conversion. Therefore, the image region segmentation model can be trained according to the second annotation information.
  • the full convolutional network uses U-Net
  • the feature dimension N can be set to 24
  • the number of superpixels is not fixed
  • each iteration uses a random number between 200-8000.
  • the adaptive moment estimation (Adam) optimization method can be used, and the learning rate is set to 0.001.
  • the sample images are pathological images, such as HER2 images.
  • HER2 is also called the proto-oncogene human epidermal growth factor receptor 2, which is used to check breast cancer.
  • These sample images come from 50 full-scan glass slides, which are used to train the image region segmentation model.
  • 226 sample data are used for model training, and the other 79 sample images are used as test data.
  • DICE coefficient is a similarity measurement function, usually used for calculation The similarity of the two samples
  • the DICE coefficient obtained is 0.86. It can be seen that the training method of the image region segmentation model provided by the embodiment of the present application has approximated a supervised algorithm, which proves the effectiveness and feasibility of the algorithm.
  • a sample image set is obtained.
  • the sample image set includes at least one sample image.
  • the sample image has its corresponding first annotation information, which may be an image. Labeling information with large granularity, such as large scale, is so easy to implement large-scale and fast labeling.
  • For the target sample image in the sample image set (it can be any sample image in the sample image set), generate graph structure data corresponding to the target sample image.
  • the graph structure data includes multiple vertices, and each vertex includes the target sample image At least one pixel.
  • the second annotation information of the vertex is determined according to the graph structure data corresponding to the target sample image and the first annotation information corresponding to the target sample image.
  • the granularity of the second annotation information is smaller than the granularity of the first annotation information. Since the vertex is actually a super pixel point, it includes at least one pixel point, that is, the second annotation information is a super-pixel-level annotation. In this way, the image region segmentation model including the graph convolutional network model is divided according to the second annotation information. For training, the intervention based on pixel-level annotation can achieve strong supervision and improve the accuracy of the model.
  • this application converts image-level annotation information into super-pixel-level annotation information based on the graph convolutional network model on the premise of realizing large-scale and fast annotation, thereby achieving strong supervised model training and improving the image region segmentation trained
  • the accuracy of the model can further improve the accuracy of image segmentation when the image area segmentation model is used for image area segmentation.
  • the first labeling information may include the area proportion of the target area in the sample image. If the pixel points in the sample image and the map structure data are evenly distributed, the area proportion may also reflect the map structure data The proportion of the number of vertices belonging to the target area in all vertex data. If the number of vertices belonging to the target area is determined, it can be determined which vertices belong to the target area and which vertices belong to the background area, thereby determining the second label information of the vertices .
  • a possible implementation manner for determining the second annotation information may be to obtain the prediction result of each vertex according to the graph structure data through the graph convolutional network model.
  • the background area is the area other than the target area in the image sample, and the target area can also be called the foreground area.
  • the second label information of the vertex is determined according to the number of the first vertices, the number of the second vertices, and the prediction result.
  • the prediction result is the probability value of each vertex belonging to the target area predicted by the graph convolutional network model based on the graph structure data. The higher the probability value corresponding to the vertex, the more likely the vertex belongs to the target area.
  • the prediction result can be sorted according to the magnitude of the probability value, for example, sorting according to the order of the probability value from the largest to the smallest, or from the smallest to the largest.
  • the second label information with only the first number of vertices indicates that it belongs to the target area (for example, the second label information is set to 1), and the remaining number of second vertices
  • the second annotation information of the vertex indicates that it belongs to the background area (for example, the second annotation information is set to 0). Since the higher the probability value, the more likely the vertices belong to the target area. Then, if the prediction results are sorted in descending order of the probability value, you can select the first vertices with the highest number of vertices among all vertices.
  • the second label information indicates that it belongs to the target area (for example, the second label information is set to 1), and the second label information of the remaining number of second vertices indicates that it belongs to the background area (for example, the second label information is set to 0).
  • the area ratio is p
  • the total number of vertices is M
  • the number of vertices corresponding to the target area is p ⁇ M
  • the number of vertices corresponding to the background area is (1-p) ⁇ M.
  • the second label information of the first p ⁇ M vertices in the prediction result can be set to 1, and the next (1-p) ⁇ M vertices
  • the second label information of is set to 0.
  • the estimated error value may also be marked while marking the area proportion, that is, the first marking information also includes the estimated error value.
  • the number of first vertices and the number of second vertices are determined.
  • the method has also changed.
  • the number of first vertices and the number of second vertices not only the area ratio, but also the estimated error value should be considered. That is, the number of first vertices and the number of second vertices can be determined according to the area ratio, the estimated error value, and the total number of vertices in the graph structure data.
  • the area ratio is p and the total number of vertices is M
  • the estimated number of first vertices is p ⁇ M
  • the number of second vertices is (1-p) ⁇ M.
  • the estimated error value is q
  • the number of confirmed first vertices is (p-q) ⁇ M
  • the number of confirmed second vertices is (1-p-q) ⁇ M.
  • the second label information of the (pq) ⁇ M vertices in the prediction result can be set to 1, and the sorting is first (pq)
  • the second label information of (1-pq) ⁇ M vertices after ⁇ M vertices is set to 0.
  • the remaining uncertain vertices can be ignored and not labeled, so as to avoid estimation errors leading to inaccurate model training.
  • the second annotation information can be used as a supervision signal.
  • a loss function can be constructed according to the prediction result of each vertex and the second annotation information, so as to calculate the loss function according to the loss function.
  • the image region segmentation model is trained.
  • the training process may include training the graph convolutional network model and the full convolutional network in the image region segmentation model until the value of the loss function is the smallest, and the training is stopped.
  • the construction of the loss function can include multiple methods, for example, the loss function can be constructed by means of mutual information entropy, mean square error (Mean Squared Error, MSE), etc., to apply a strong supervision signal.
  • MSE mean squared Error
  • an embodiment of the present application also provides an image region segmentation method. Referring to FIG. 4, the method includes:
  • S401 Acquire an image to be divided.
  • the image to be segmented can be various types of images, such as pathological images, traffic monitoring images, and so on. Taking the image to be segmented is a pathological image as an example, the target area to be obtained through image area segmentation may be a lesion for subsequent diagnosis. Among them, the image to be divided can be referred to as shown in 501 in FIG. 5.
  • S402 Generate graph structure data corresponding to the image to be divided through the image region segmentation model.
  • the image to be segmented is input into the image region segmentation model (as shown in FIG. 5), and the graph structure data corresponding to the image to be segmented is obtained.
  • the graph structure data includes a plurality of vertices, and each of the vertices includes at least one pixel in the image to be divided.
  • the image region segmentation model used in the embodiment of the present application is obtained by training according to the method provided in the embodiment shown in FIG. 2, and will not be repeated here.
  • segmentation is performed according to the image structure data corresponding to the image to be segmented to obtain a target region.
  • the segmentation effect diagram can be seen as shown in 502 in FIG. 5, where white represents the target area, and black represents the background area.
  • FIG. 6A and FIG. 6B The comparison of the segmentation effect obtained by image segmentation by the image region segmentation method provided by the embodiment of the application and the segmentation effect obtained by the supervised algorithm for image segmentation can be seen in FIG. 6A and FIG. 6B.
  • the picture in FIG. 6A is through
  • the image region segmentation method provided by the embodiment of the application is a segmentation effect map obtained by segmentation.
  • the picture in FIG. 6B is a segmentation effect map obtained through a supervised algorithm.
  • the segmentation results are similar. It can be seen that the image region segmentation method provided by the embodiments of the present application can achieve strong supervision and improve the accuracy of the model.
  • a medical device carries a pre-trained image region segmentation model.
  • the medical device can collect pathological images of patients and perform lesion segmentation on the pathological images.
  • the method includes:
  • the medical device collects a pathological image of the patient.
  • the medical device generates graph structure data corresponding to the pathological image through the image region segmentation model.
  • the medical device uses the image region segmentation model to segment the image structure data to obtain the lesion.
  • the medical device displays the lesion on the display screen.
  • the doctor performs observation and analysis according to the displayed lesions to diagnose the disease.
  • the image region segmentation model training method and image region segmentation method provided by the embodiments of the application can be applied to a variety of scenarios. Taking the medical field as an example, the image region segmentation model training method and image region segmentation method can be applied to medical devices.
  • the image region segmentation model training method can be used to train the image region segmentation model, and then the pathological image can be segmented using the image region segmentation model obtained by training to obtain a target region such as a lesion for subsequent analysis and processing.
  • an embodiment of the present application also provides a medical device, which may be a medical imaging device, such as an X-ray machine, a computer tomography (Computed Tomography, CT) device, or a magnetic resonance imaging (Magnetic Resonance Imaging, abbreviation).
  • MRI Magnetic Resonance Imaging
  • the medical equipment includes an image acquisition module, an image processing module, and an image display module:
  • the image acquisition module is used to obtain an image to be segmented; the image to be segmented is a pathological image including biological tissue;
  • the image processing module is configured to generate graph structure data corresponding to the image to be divided through an image region segmentation model;
  • the graph structure data includes a plurality of vertices, and each of the vertices includes at least one of the images to be divided One pixel;
  • the image region segmentation model is obtained by training according to the second annotation information, and the second annotation information is determined according to the graph structure data corresponding to the sample image and the first annotation information corresponding to the sample pattern;
  • the granularity of the second annotation information is smaller than the granularity of the first annotation information;
  • the lesions in the image to be segmented are obtained by segmenting the image region segmentation model according to the graph structure data;
  • the image display module is used to display the lesion so that the doctor can observe, analyze, etc. through the lesion displayed by the medical device.
  • the image acquisition module is configured to acquire a sample image set, the sample image set includes a plurality of sample images, and the sample images have corresponding first annotation information;
  • the image processing module is used for:
  • the target sample image in the sample image set For the target sample image in the sample image set, generate graph structure data corresponding to the target sample image; the graph structure data includes a plurality of vertices, and each vertex includes at least one pixel in the target sample image Point; the target sample image is any sample image in the sample image set;
  • the second annotation information of the vertex is determined according to the graph structure data corresponding to the target sample image and the first annotation information corresponding to the target sample image; the granularity of the second annotation information is less than The granularity of the first annotation information; the graph convolutional network model is a part of the graph region segmentation model;
  • the first annotation information includes the area proportion of the target area in the sample image
  • the image processing module is configured to:
  • the number of first vertices corresponding to the target area and the number of second vertices corresponding to the background area are determined, where the background area is divided by the sample image The area outside the target area;
  • the second label information of the vertices is determined.
  • the first annotation information further includes an estimated error value
  • the image processing module is configured to:
  • the image processing module is used for:
  • the image region segmentation model includes a fully convolutional network
  • the image processing module is used for:
  • the graph structure data is constructed according to the superpixel division result, and each superpixel point in the superpixel division result is used as a vertex in the graph structure data.
  • the image processing module is used for:
  • the first vertex and the second vertex can be any two vertex;
  • the weight of the edge is determined according to the Euclidean distance between the features corresponding to the first vertex and the second vertex.
  • the feature corresponding to the vertex is an average value of the features of the pixels included in the vertex.
  • an embodiment of the present application also provides an image region segmentation model training device.
  • the device includes an acquiring unit 801, a generating unit 802, a determining unit 803, and a training unit 804:
  • the acquiring unit 801 is configured to acquire a sample image set, the sample image set includes at least one sample image, and each sample image has its corresponding first annotation information;
  • the generating unit 802 is configured to generate graph structure data corresponding to the target sample image for the target sample image in the sample image set;
  • the graph structure data includes a plurality of vertices, each of the vertices Includes at least one pixel in the target sample image;
  • the target sample image is any sample image in the sample image set;
  • the determining unit 803 is configured to determine the second annotation information of the vertex according to the graph structure data corresponding to the target sample image and the first annotation information corresponding to the target sample image through a graph convolutional network model;
  • the granularity of the second annotation information is smaller than the granularity of the first annotation information;
  • the graph convolutional network model is a part of the graph region segmentation model;
  • the training unit 804 is configured to train the image region segmentation model according to the second annotation information.
  • the first annotation information includes the area proportion of the target area in the sample image
  • the determining unit 803 is configured to:
  • the number of first vertices corresponding to the target area and the number of second vertices corresponding to the background area are determined.
  • the second label information of the vertices is determined.
  • the first labeling information further includes an estimated error value
  • the determining unit 803 is configured to:
  • the training unit 804 is configured to:
  • the image region segmentation model includes a fully convolutional network
  • the generating unit 802 is configured to:
  • the graph structure data is constructed according to the superpixel division result, and each superpixel point in the superpixel division result is used as a vertex in the graph structure data.
  • the generating unit 802 is configured to:
  • the weight of the edge is determined according to the Euclidean distance between the corresponding features of the first vertex and the second vertex.
  • the feature corresponding to the vertex is an average value of the features of the pixels included in the vertex.
  • an embodiment of the present application also provides an image region segmentation device based on artificial intelligence.
  • the device includes an acquisition unit 901, a generation unit 902, and a segmentation unit 903:
  • the acquiring unit 901 is configured to acquire an image to be divided
  • the generating unit 902 is configured to generate graph structure data corresponding to the image to be divided through an image region segmentation model; the graph structure data includes a plurality of vertices, and each vertex includes at least one of the images to be divided Pixel points; the image region segmentation model is obtained by training according to the second annotation information, the second annotation information is determined according to the graph structure data corresponding to the sample image and the first annotation information corresponding to the sample pattern; The granularity of the second annotation information is smaller than the granularity of the first annotation information;
  • the segmentation unit 903 is configured to obtain the target area in the image to be segmented by segmenting the image region segmentation model according to the graph structure data corresponding to the image to be segmented.
  • the image to be segmented is a pathological image
  • the target area is a lesion
  • an embodiment of the present application also provides a device, which can implement the image region segmentation model training method or the image region segmentation method described above.
  • the device will be introduced below in conjunction with the drawings.
  • an embodiment of the present application provides a device 1000.
  • the device 1000 may also be a terminal device.
  • the terminal device may include a computer, a tablet computer, a mobile phone, and a personal digital assistant (Personal Digital Assistant, for short). PDA), Point of Sales (POS), on-board computer, etc., taking the terminal device as a mobile phone as an example:
  • FIG. 10 shows a block diagram of a part of the structure of a mobile phone related to a terminal device provided in an embodiment of the present application.
  • the mobile phone includes: Radio Frequency (RF) circuit 1010, memory 1020, input unit 1030, display unit 1040, sensor 1050, audio circuit 1060, wireless fidelity (wireless fidelity, WiFi for short) module 1070, processing Adapter 1080, and power supply 1090 and other components.
  • RF Radio Frequency
  • the processor 1080 included in the terminal device also has the following functions:
  • sample image set including a plurality of sample images, each of the sample images has its corresponding first annotation information
  • the target sample image in the sample image set For the target sample image in the sample image set, generate graph structure data corresponding to the target sample image; the graph structure data includes a plurality of vertices, and each vertex includes at least one pixel in the target sample image Point; the target sample image is any sample image in the sample image set;
  • the second annotation information of the vertex is determined according to the graph structure data corresponding to the target sample image and the first annotation information corresponding to the target sample image;
  • the granularity is smaller than the granularity of the first annotation information;
  • the graph convolutional network model is a part of the graph region segmentation model;
  • the graph structure data corresponding to the image to be segmented is generated through the image region segmentation model; the graph structure data includes a plurality of vertices, and each of the vertices includes at least one pixel in the image to be segmented; the image region segmentation
  • the model is trained based on the second annotation information, the second annotation information is determined based on the graph structure data corresponding to the sample image and the first annotation information corresponding to the sample pattern; the granularity of the second annotation information is smaller than the first annotation information corresponding to the sample pattern. State the granularity of the first annotation information;
  • the target region in the image to be segmented is obtained by segmenting according to the graph structure data corresponding to the image to be segmented.
  • FIG. 11 is a structural diagram of the server 1100 provided by the embodiment of the present application.
  • the above Central Processing Units (CPU for short) 1122 for example, one or more processors
  • memory 1132 for example, one or more storage media 1130 for storing application programs 1142 or data 1144 (for example, one or one storage medium for storing data 1144) equipment).
  • the memory 1132 and the storage medium 1130 may be short-term storage or persistent storage.
  • the program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of command operations on the server.
  • the central processing unit 1122 may be configured to communicate with the storage medium 1130, and execute a series of instruction operations in the storage medium 1130 on the server 1100.
  • the server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and/or one or more operating systems 1141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
  • operating systems 1141 such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
  • the steps executed by the server in the foregoing embodiment can be executed based on the server structure shown in FIG. 11.
  • the embodiments of the present application also provide a computer-readable storage medium, where the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in each of the foregoing embodiments.
  • the embodiments of the present application also provide a computer program product including instructions, which when run on a computer, cause the computer to execute the method described in each of the foregoing embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)

Abstract

一种图像区域分割模型训练方法、分割方法及装置。在模型训练过程中，获取包括至少一个样本图像的样本图像集合，样本图像具有其对应的第一标注信息，该第一标注信息为可以是图像级等粒度较大的标注信息。针对样本图像集合中的目标样本图像，生成目标样本图像对应的图结构数据，图结构数据中每个顶点包括目标样本图像中至少一个像素点。通过图卷积网络模型，根据图结构数据和第一标注信息确定顶点的第二标注信息，第二标注信息的粒度小于第一标注信息的粒度。由于顶点实际上是超像素点，第二标注信息是超像素级的标注，在训练过程中，基于超像素级标注的干预可以实现较强监督，提高模型的精确性，进而提高图像分割的精确性。

Description

一种图像区域分割模型训练方法、分割方法和装置
本申请要求于2020年05月18日提交中国专利局、申请号为202010419791.7、申请名称为“一种图像区域分割模型训练方法、分割方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能领域，特别涉及图像区域分割技术。
背景技术
随着计算机技术的发展，图像分割技术的应用越来越广泛，例如，医学图像分割以及自然图像分割等。其中，图像分割技术是把图像分成若干个特定的、具有独特性质的区域，并从中提出感兴趣目标的技术。例如，在医学图像分割场景中，可以对医学图像中的病灶进行分割，以用于进一步分析。
深度学习如今在图像区域分割领域得到了广泛的应用，为了减少产生大量标注图像所耗费的时间和人工成本，相关技术中采用图像级标注的方式，利用弱监督的模式完成图像区域分割。
然而，采用图像级标签的弱监督，监督信号过弱，很难精确分割出来感兴趣的目标区域。
发明内容
为了解决上述技术问题，本申请提供了一种图像区域分割模型训练方法、图像区域分割方法和装置，在实现大规模快速标注的前提下，提高了所训练的图像区域分割模型的精确性，进而在使用训练得到的该图像区域分割模型进行图像区域分割时，提高图像分割的精确性。
本申请实施例公开了如下技术方案：
第一方面，本申请实施例提供一种图像区域分割模型训练方法，由数据处理设备执行，所述方法包括：
获取样本图像集合，所述样本图像集合包括至少一个样本图像，每个所述样本图像具有其对应的第一标注信息；
针对所述样本图像集合中的目标样本图像，生成所述目标样本图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述目标样本图像中至少一个像素点；所述目标样本图像为所述样本图像集合中的任一个样本图像；
通过图卷积网络模型，根据所述目标样本图像对应的图结构数据和所述目标样本图像对应的所述第一标注信息，确定所述顶点的第二标注信息；所述第二标注信息的粒度小于所述第一标注信息的粒度；所述图卷积网络模型为所述图形区域分割模型的一部分；
根据所述第二标注信息对所述图像区域分割模型进行训练。
第二方面，本申请实施例提供一种图像区域分割模型训练装置，所述装置包括获取单元、生成单元、确定单元和训练单元：
所述获取单元，用于获取样本图像集合，所述样本图像集合包括至少一个样本图像，每个所述样本图像具有其对应的第一标注信息；
所述生成单元，用于所述针对所述样本图像集合中的目标样本图像，生成所述目标样本图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述目标样本图像中至少一个像素点；所述目标样本图像为所述样本图像集合中的任一个样本图像；
所述确定单元，用于通过图卷积网络模型，根据所述目标样本图像对应的图结构数据和所述目标样本图像对应的所述第一标注信息，确定所述顶点的第二标注信息；所述第二标注信息的粒度小于所述第一标注信息的粒度；所述图卷积网络模型为所述图形区域分割模型的一部分；
所述训练单元，用于根据所述第二标注信息对所述图像区域分割模型进行训练。
第三方面，本申请实施例提供一种图像区域分割方法，由数据处理设备执行，所述方法包括：
获取待分割图像；
通过图像区域分割模型生成所述待分割图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述待分割图像中至少一个像素点；所述图像区域分割模型是根据第二标注信息训练得到的，所述第二标注信息是根据样本图像对应的图结构数据和所述样本图样对应的第一标注信息确定的；所述第二标注信息的粒度小于所述第一标注信息的粒度；
通过所述图像区域分割模型，根据所述待分割图像对应的图结构数据分割得到所述待分割图像中的目标区域。
第四方面，本申请实施例提供一种图像区域分割装置，所述装置包括获取单元、生成单元和分割单元：
所述获取单元，用于获取待分割图像；
所述生成单元，用于通过图像区域分割模型生成所述待分割图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述待分割图像中至少一个像素点；所述图像区域分割模型是根据第二标注信息训练得到的，所述第二标注信息是根据样本图像对应的图结构数据和所述样本图样对应的第一标注信息确定的；所述第二标注信息的粒度小于所述第一标注信息的粒度；
所述分割单元，用于通过所述图像区域分割模型，根据所述待分割图像对应的图结构数据分割得到所述待分割图像中的目标区域。
第五方面，本申请实施例提供一种医疗器械，所述医疗器械包括图像采集模组、图像处理模组和图像显示模组：
所述图像采集模组，用于获取待分割图像；所述待分割图像为包括生物组织的病理图像；
所述图像处理模组，用于通过图像区域分割模型生成所述待分割图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述待分割图像中至少一个像素点；所述图像区域分割模型是根据第二标注信息训练得到的，所述第二标注信息是根据样本图像对应的图结构数据和所述样本图样对应的第一标注信息确定的；所述第二标注信息的粒度小于所述第一标注信息的粒度；通过所述图像区域分割模型，根据所述待分割图像对应的图结构数据分割得到所述待分割图像中的病灶；
所述图像显示模组，用于显示所述病灶。
第六方面，本申请实施例提供一种设备，所述设备包括处理器以及存储器：
所述存储器用于存储程序代码，并将所述程序代码传输给所述处理器；
所述处理器用于根据所述程序代码中的指令执行第一方面或第三方面所述的方法。
第七方面，本申请实施例提供一种计算机可读存储介质，所述计算机可读存储介质用于存储程序代码，所述程序代码用于执行第一方面或第三方面所述的方法。
第八方面，本申请实施例提供一种计算机程序产品，包括指令，当其在计算机上运行时，使得计算机执行第一方面或第三方面所述的方法。
由上述技术方案可以看出，本申请在模型训练过程中，获取样本图像集合，样本图像集合包括至少一个样本图像，每个样本图像具有其对应的第一标注信息，该第一标注信息可以是图像级等粒度较大的标注信息，如此易于实现大规模快速标注。针对样本图像集合中的目标样本图像(可以为样本图像集合中任一个样本图像)，生成该目标样本图像对应的图结构数据，图结构数据中包括多个顶点，每个顶点包括该目标样本图像中至少一个像素点。通过图卷积网络模型，根据目标样本图像对应的图结构数据和目标样本图像对应的第一标注信息，确定顶点的第二标注信息，第二标注信息的粒度小于第一标注信息的粒度。由于顶点实际上是超像素点，其包括至少一个像素点，也就是说，第二标注信息是超像素级的标注，这样，根据第二标注信息对包括图卷积网络模型的图像区域分割模型进行训练，基于像素级标注的干预可以实现较强监督，提高所训练的图像区域分割模型的精确性。可见，本申请在实现大规模快速标注的前提下，基于图卷积网络模型将图像级标注信息转化为超像素级标注信息，从而实现较强监督的模型训练，提高了模型的精确性，进而在使用训练得到的模型进行图像区域分割时，提高图像分割的精确性。
附图说明
图1为本申请实施例提供的一种图像区域分割模型训练方法的应用场景示意图；
图2为本申请实施例提供的一种图像区域分割模型训练方法的流程图；
图3为本申请实施例提供的图像区域分割模型训练方法的系统流程图；
图4为本申请实施例提供的一种图像区域分割方法的流程图；
图5为本申请实施例提供的一种图像区域分割方法的系统流程图；
图6A为本申请实施例提供的图像区域分割方法进行分割得到的分割效果图；
图6B为通过有监督算法进行分割得到的分割效果图；
图7为本申请实施例提供的一种图像区域分割方法的流程图；
图8为本申请实施例提供的一种图像区域分割模型训练装置的结构图；
图9为本申请实施例提供的一种图像区域分割装置的结构图；
图10为本申请实施例提供的一种终端设备的结构图；
图11为本申请实施例提供的一种服务器的结构图。
具体实施方式
下面结合附图，对本申请的实施例进行描述。
深度学习如今在图像分割领域得到了广泛的应用，为了训练好的图像区域分割模型，往往需要精确的像素级别标注，但是像素级人工标注极其耗时耗力。比如人工标注一张2048*2048的病例图片中的病灶，往往需要5-30分钟。因此，产生大量带有标注的样本图像变得非常昂贵和耗时。鉴于此，基于弱监督分割(Weakly supervised segmentation)方法应运而生。弱监督分割方法例如可以采用类激活映射(Class Activation Mapping，CAM)算法。
而相关技术中的弱监督分割方法，通常使用与分割任务相关的图像级标签(往往为图像类别)训练分类模型，进而，利用训练得到的分类模型确定需要分割的目标区域，例如在医疗领域的病灶。由于图像级标签相对于像素级标签的标注较为粗略，难以对图像进行准确的标注，没有直接地进行像素级干预，导致通过该弱监督方法训练得到的模型分割出的图像区域分割结果往往不够精确。
为此，本申请实施例提供一种图像区域分割模型训练方法，该方法仍然对样本图像采用图像级标注，但是在模型训练过程中，会基于图卷积网络模型将图像级标注信息(第一标注信息)转化为超像素级标注信息(第二标注信息)，利用第二标注信息对图像区域分割模型进行训练，从而实现较强监督的模型训练，提高了训练得到的图像区域分割模型的精确性，进而在使用该图像区域分割模型进行图像区域分割时，可以提高图像分割的精确性。
本申请实施例提供的图像区域分割模型训练方法和图像区域分割方法，可以应用于具有图形处理器(Graphics Processing Unit，GPU)的数据处理设备，该数据处理设备可以是终端设备，该终端设备例如可以是计算机、个人 数字助理(Personal Digital Assistant，简称PDA)、平板电脑、智能手机等。
该数据处理设备还可以是服务器，在实际部署时，服务器可以是独立服务器，也可以是集群服务器或是云服务器。在这种情况下，可以通过服务器完成图像区域分割模型的训练，然后，利用服务器与终端设备进行交互，对从终端设备处获取到的待分割图像进行目标区域分割，并将分割结果返回至终端设备。
为了便于理解本申请的技术方案，下面结合实际应用场景对本申请实施例提供的图像区域分割模型训练方法进行介绍。
参见图1，图1为本申请实施例提供的图像区域分割模型训练方法的应用场景示意图。以数据处理设备是终端设备为例，该应用场景中包括终端设备101。
终端设备101获取样本图像集合，该样本图像集合包括多个具有第一标注信息的样本图像。样本图像中包括用户感兴趣的目标区域，样本图像可以是各种类型的图像，例如病理图像、交通监控图像等等。为了避免标注样本图像耗费大量人力和时间成本，便于实现大规模快速标注，上述第一标注信息是粒度较大的标注信息，例如图像级标注信息。
针对样本图像集合中的目标样本图像，生成该目标样本图像对应的图结构数据，其中包括多个顶点，每个顶点包括目标样本图像中至少一个像素点，该目标样本图像可以为样本图像集合中任一个样本图像。
由于图结构数据是由目标样本图像中至少一个像素点构成的，这些像素点构成的图结构数据通常是非标准几何的数据，因此需要通过图卷积网络模型对图结构数据进行处理。即，在本实施例中，可以通过图像区域分割模型中的图卷积网络模型，根据图结构数据和第一标注信息确定图结构数据中顶点的第二标注信息。
顶点实际上是超像素点，其包括目标样本图像中至少一个像素点，也就是说，第二标注信息是超像素级的标注，第二标注信息的粒度明显小于第一标注信息的粒度。这样，根据第二标注信息对包括图卷积网络模型的图像区域分割模型进行训练，基于超像素级标注的干预可以实现较强监督，提高模型的精确性。
需要说明的是，本申请实施例提供的方法可以应用于多种应用场景，例如在医疗领域对医学图像进行分割以区分人体各个组织，对病理图片中的目标区域(例如病灶)进行分割，交通领域的车辆识别等等。
接下来，将结合附图对本申请实施例提供的图像区域分割模型训练方法进行介绍。
参见图2，图2示出了一种图像区域分割模型训练方法的流程图，所述方法包括：
S201、获取样本图像集合。
样本图像集合包括多个样本图像，每个样本图像具有其对应的第一标注信息。在本实施例中，第一标注信息是粒度较大的标注信息，例如图像级标注信息。图像级标注信息仅需要对样本图像整体进行标注，一个样本图像对应一个标签，无需针对样本图像中每个像素点进行标注，可以节省因标注耗费的人力和时间成本，便于实现大规模快速标注。
图像级标注信息可以包括多种，在本实施例中，图像级标注信息例如可以是目标区域在样本图像中的面积占比，即第一标注信息包括目标区域在样本图像中的面积占比。其中，目标区域可以是样本图像中用户可能感兴趣的区域，例如病理图像中的病灶等；目标区域在样本图像中的面积占比可以是人工根据样本图像估计得到，为了避免估计误差，在标注面积占比的同时，还可以标注估计误差值，从而在后续模型训练过程中考虑该估计误差值，提高模型训练的精确性。
S202、针对所述样本图像集合中的目标样本图像，生成所述目标样本图像对应的图结构数据。
在得到样本图像集合后，生成目标样本图像(该目标样本图像可以为样本图像集合中任一样本图像)对应的图结构数据。图结构数据的顶点包括目标样本图像中至少一个像素点，每个顶点包括哪些像素点可以通过对目标样本图像中的像素点进行聚类确定，而聚类的依据可以是像素点的特征，即可以将特征相似的像素点聚类在一起作为顶点。
在一些实施例中，图像区域分割模型中可以包括全卷积网络(Fully convolutional networks，FCN)。生成图结构数据的方式可以是，通过全卷积网络对目标样本图像进行特征提取，得到该目标样本图像中每个像素点对应的特征，然后，根据像素点对应的特征对目标样本图像中的像素点进行聚类，得到超像素划分结果，根据超像素划分结果构建图结构数据，该超像素划分结果中每个超像素点作为图结构数据中的一个顶点。
生成图结构数据的方式可以参见图3所示，图3示出了图像区域分割模型训练方法的系统流程，主要包括特征提取模块301和分类模块302，其中，特征提取模块301用于将样本图像集合中的目标样本图像输入到全卷积网络，得到目标样本图像中每个像素点的特征。每个像素点对应的特征可以用N维的特征向量表示，N为全卷积网络的通道数。
其中，本申请实施例可以选取常规的全卷积网络，比如U-Net，U-Net是全卷积网络的一种网络结构，当然还可以选取其他网络结构，本实施例对全卷积网络的网络结构不做限定。
由于每个像素点都得到其对应的特征，故可以根据像素点的特征对目标样本图像中的像素点进行聚类，将特征相似的像素点聚类在一起构成一个超像素点(superpixel)，得到超像素划分结果。例如，属于同一个细胞的像素点 的特征一般比较相似，可以聚类在一起形成一个超像素点；属于同一部位(例如都属于嘴部)的像素点的特征一般比较相似，可以聚类在一起形成一个超像素点，等等。
其中，超像素点划分需要指定超像素点的个数，本申请实施例采用随机数的方式赋值。聚类方法可以包括多种，例如简单的线性迭代聚类(simple linear iterative clustering，SLIC)、归一分割算法、基于梯度上升的算法等等，本申请实施例对此不做限定。
需要说明的是，由于本实施例中采用全卷积网络提取像素点的特征，超像素划分需要利用全卷积网络输出的像素点的特征作为划分依据，因此只要特征变化，超像素点的划分也会相应地改变。而在图像区域分割模型训练过程中，会对图像区域分割模型中的全卷积网络进行参数调整，从而导致输出的特征变化，进一步带动超像素划分结果发生变化，也就是说，在本实施例中得到的是动态超像素点，此类动态超像素点更有助于提高模型训练的精确性。
在得到超像素划分结果后，可以根据超像素划分结果构建图结构数据，将超像素划分结果中每个超像素点作为图结构数据中的一个顶点，进入后续分类模块302。
图结构数据中除了包括多个顶点(Node)，一些顶点之间可能还存在边(edge)以及边的权重(weight)。边表示两个顶点之间在空间位置上相邻，具有一定的关联关系；边的权重表示两个顶点之间的关联程度，权重越大，关联程度越大。
因此，在一些实施例中，根据超像素划分结果构建图结构数据的方式，可以是根据任意两个顶点所包括的像素点的位置信息确定顶点之间的边。通常情况下，如果两个超像素点(顶点)为空间相连，那么这两个超像素点之间即可建立起边的连接。判断两个超像素点(顶点)是否空间相连的方式，可以是判断这两个超像素点包括的像素点中是否存在相邻像素点。
针对任一条边，若该条边连接第一顶点和第二顶点，则根据第一顶点和第二顶点各自对应的特征之间的欧氏距离，确定边的权重。欧式距离越大，关联程度越小，权重越小。通过该方式设置权重，更能够准确地体现出超像素点之间的关联程度，从而有利于提高后续处理的准确性。
当然，还可以简化边的权重的确定方式，在一些情况下，可以将所有边的权重设置为统一的数值，例如可以将各个边的权重统一设置为1。
可以理解的是，针对图结构数据中的每个顶点，顶点对应的特征为该顶点包括像素点的特征的平均值。若像素点的特征为N维特征向量，则顶点的特征也为N维特征向量。
S203、通过图卷积网络模型，根据所述目标样本图像对应的图结构数据和所述目标样本图像对应的第一标注信息，确定所述顶点的第二标注信息。
在得到图结构数据后，即得到图卷积网络模型所需的数据后，可以将图结构数据输入至图卷积网络模型，以便对包括图卷积网络模型在内的图像区域分割模型进行训练。其中，图卷积网络模型可以由五个线性图卷积层构成，每个线性层的神经元数量分别为64，128，256，64，1。除了最后一层，为每个层加入非线性激活层(ReLu)。
参见图3中302所示，图卷积网络模型可以根据图结构数据和第一标注信息确定顶点的第二标注信息，由于图结构数据中包括多个顶点，顶点中包括至少一个像素点，因此，第二标注信息是像素级标注信息，第二标注信息的粒度小于第一标注信息的粒度。
S204、根据所述第二标注信息对所述图像区域分割模型进行训练。
在本实施例中，第一标注信息是预先标注的真实标签，而第二标注信息是伪标签，是图像区域分割模型中包括的图卷积网络模型对图结构数据进行预测，根据第一标注信息转换得到的像素级别的标注信息，与第一标注信息相比，第二标注信息能够提供较强的监督信号，因此，可以根据第二标注信息对图像区域分割模型进行训练。
在本实施例中，全卷积网络采用U-Net，特征的维度N可以设为24，超像素点个数不固定，每个迭代采用200-8000之间的随机数。可以采用适应性矩估计(Adaptive moment estimation，Adam)优化方法，学习率设为0.001。
若获取到305个样本图像，样本图像是病理图像，例如HER2图像，HER2又称为原癌基因人类表皮生长因子受体2，用于检查乳腺癌。这些样本图像来自50个全片扫描玻片，用于进行图像区域分割模型的训练。其中，采用226个样本数据用于模型训练，另外79个样本图像作为测试数据。利用训练数据基于本申请实施例提供的方法训练得到图像区域分割模型，利用测试数据对图像区域分割模型进行测试，得到分割结果的DICE系数(DICE系数是一种相似度度量函数，通常用于计算两个样本的相似度)为0.84。而采用有监督的算法(基于U-Net的全监督)训练图像区域分割模型，得到的DICE系数为0.86。可见，本申请实施例提供的图像区域分割模型的训练方法已经逼近有监督算法，证明了算法的有效性和可行性。
由上述技术方案可以看出，本申请在模型训练过程中，获取样本图像集合，样本图像集合包括至少一个样本图像，样本图像具有其对应的第一标注信息，该第一标注信息为可以是图像级等粒度较大的标注信息，如此易于实现大规模快速标注。针对样本图像集合中的目标样本图像(可以为样本图像集合中任一个样本图像)，生成该目标样本图像对应的图结构数据，图结构数据中包括多个顶点，每个顶点包括目标样本图像中至少一个像素点。通过图卷积网络模型，根据目标样本图像对应的图结构数据和目标样本图像对应的第一标注信息，确定顶点的第二标注信息，第二标注信息的粒度小于第一标注信息的粒度。由于顶点实际上是超像素点，其包括至少一个像素点，也 就是说，第二标注信息是超像素级的标注，这样，根据第二标注信息对包括图卷积网络模型的图像区域分割模型进行训练，基于像素级标注的干预可以实现较强监督，提高模型的精确性。可见，本申请在实现大规模快速标注的前提下，基于图卷积网络模型将图像级标注信息转化为超像素级标注信息，从而实现较强监督的模型训练，提高了所训练的图像区域分割模型的精确性，进而在使用该图像区域分割模型进行图像区域分割时，能够提高图像分割的精确性。
在一种可能的实现方式中，第一标注信息可以包括目标区域在样本图像中的面积占比，若样本图像和图结构数据中像素点分布均匀，则面积占比也可以反映出图结构数据中属于目标区域的顶点数量在所有顶点数据中的占比，若确定出属于目标区域的顶点数量，则可以确定出哪些顶点属于目标区域，哪些顶点属于背景区域，从而确定顶点的第二标注信息。
因此，确定第二标注信息的一种可能的实现方式可以是，通过图卷积网络模型，根据图结构数据得到每个顶点的预测结果。根据面积占比和图结构数据中顶点的总数量，确定目标区域对应的第一顶点数量(即属于目标区域的顶点数量)以及背景区域对应的第二顶点数量(即属于背景区域的顶点数量)，背景区域为图像样本中除目标区域外的区域，目标区域也可以称为前景区域。接着，根据第一顶点数量、第二顶点数量和预测结果确定顶点的第二标注信息。
通常情况下，预测结果为图卷积网络模型根据图结构数据预测得到的每个顶点属于目标区域的概率值，顶点对应的概率值越高，则表示该顶点越有可能属于目标区域。在得到预测结果后，可以按照概率值的大小进行排序，例如按照概率值从大到小的顺序进行排序，或从小到大的顺序进行排序。由于属于目标区域的顶点数据为第一顶点数量，那么，仅有第一顶点数量个顶点的第二标注信息表示其属于目标区域(例如第二标注信息设为1)，剩余第二顶点数量个顶点的第二标注信息表示其属于背景区域(例如第二标注信息设为0)。由于概率值越高，顶点越有可能属于目标区域，那么，若预测结果按照概率值从大到小的顺序进行排序，可以在所有顶点中，选择排序靠前的第一顶点数量个顶点的第二标注信息表示其属于目标区域(例如第二标注信息设为1)，剩余的第二顶点数量个顶点的第二标注信息表示其属于背景区域(例如第二标注信息设为0)。
例如面积占比为p，顶点的总数量为M，那么目标区域对应的顶点数量(第一顶点数量)为p×M，背景区域对应的顶点数量(第二顶点数量)为(1-p)×M。鉴于此，若预测结果按照概率值从大到小的顺序进行排序，可以将预测结果中的前p×M个顶点的第二标注信息设为1，将后(1-p)×M个顶点的第二标注信息设为0。
在一些实施例中，为了避免估计误差，在标注面积占比的同时，还可以 标注估计误差值，即第一标注信息还包括估计误差值，此时，确定第一顶点数量和第二顶点数量的方式也有所改变，确定第一顶点数量和第二顶点数量时不仅要考虑面积占比，也要考虑估计误差值。即，可以根据面积占比、估计误差值和图结构数据中顶点的总数量，确定第一顶点数量以及第二顶点数量。
例如面积占比为p，顶点的总数量为M，那么，估计出的第一顶点数量为p×M，第二顶点数量为(1-p)×M。若估计误差值为q，考虑到估计误差值，确信的第一顶点数量为(p-q)×M，确信的第二顶点数量为(1-p-q)×M。鉴于此，若预测结果按照概率值从大到小的顺序进行排序，可以将预测结果中排序靠前的(p-q)×M个顶点的第二标注信息设为1，将排序在前(p-q)×M个顶点后的(1-p-q)×M个顶点的第二标注信息设为0。而对于余下的不确信的顶点可以忽略，不进行标注，从而避免估计误差导致模型训练的不精确。
在根据第二标注信息对图像区域分割模型进行训练的过程中，可以将第二标注信息作为监督信号，例如可以根据每个顶点的预测结果和第二标注信息构建损失函数，从而根据损失函数对图像区域分割模型进行训练。该训练过程可以包括对图像区域分割模型中的图卷积网络模型、全卷积网络进行训练，直到损失函数的值最小，停止训练。
其中，构建损失函数可以包括多种方式，例如可以通过互信息熵、均方误差(MeanSquaredError，MSE)等方式构建损失函数，施加强的监督信号。
基于前述实施例提供的图像区域分割模型的训练方法，本申请实施例还提供一种图像区域分割方法，参见图4，所述方法包括：
S401、获取待分割图像。
待分割图像可以是各种类型的图像，例如病理图像、交通监控图像等等。以待分割图像是病理图像为例，则通过图像区域分割所要获取的目标区域可以为病灶，以用于后续诊断。其中，待分割图像可以参见图5中501所示。
S402、通过图像区域分割模型生成所述待分割图像对应的图结构数据。
将待分割图像输入至图像区域分割模型(如图5所示)，得到该待分割图像对应的图结构数据。其中，图结构数据中包括多个顶点，每个所述顶点包括待分割图像中至少一个像素点。
本申请实施例中所采用的图像区域分割模型是根据图2所示实施例所提供的方法训练得到的，此处不再赘述。
S403、通过所述图像区域分割模型，根据所述待分割图像对应的图结构数据分割得到目标区域。
通过图像区域分割模型对图结构数据进行预测得到预测结果，从而确定该图结构数据中哪些顶点属于目标区域，哪些顶点属于背景区域，进而确定 待分割图像中哪些像素点属于目标区域，哪些像素点属于背景区域，从而分割得到目标区域。分割效果图可以参见图5中502所示，其中，白色表示目标区域，黑色表示背景区域。
通过本申请实施例提供的图像区域分割方法进行图像分割得到的分割效果与通过有监督算法进行图像分割得到的分割效果对比图，可以参见图6A和图6B所示，图6A中的图片为通过本申请实施例提供的图像区域分割方法进行分割得到的分割效果图，图6B中的图片为通过有监督算法进行分割得到的分割效果图，通过两张效果图的比对可以看出，二者的分割结果相似，可见通过本申请实施例提供的图像区域分割方法可以实现较强监督，提高模型的精确性。
接下来，将结合实际应用场景对本申请实施例提供的图像区域分割方法进行介绍。以医疗场景为例，在该应用场景中医疗器械中承载有预先训练好的图像区域分割模型，该医疗器械可以采集病人的病理图像，并对病理图像进行病灶分割。参见图7，所述方法包括：
S701、医疗器械采集病人的病理图像。
S702、医疗器械通过图像区域分割模型生成病理图像对应的图结构数据。
S703、医疗器械通过图像区域分割模型，根据图结构数据分割得到病灶。
S704、医疗器械通过显示屏显示病灶。
S705、医生根据显示的病灶进行观察、分析，以诊断疾病。
本申请实施例提供的图像区域分割模型训练方法和图像区域分割方法可以应用于多种场景，以医疗领域为例，图像区域分割模型训练方法和图像区域分割方法可以应用于医疗器械上，医疗器械可以利用该图像区域分割模型训练方法训练图像区域分割模型，进而利用训练得到的图像区域分割模型对病理图像进行区域分割，得到目标区域例如病灶，以便用于后续分析、处理。
为此，本申请实施例还提供一种医疗器械，该医疗器械可以为医学影像设备，如X光机、电子计算机断层扫描(Computed Tomography，简称CT)设备、磁共振成像(Magnetic Resonance Imaging，简称MRI)设备等，所述医疗器械包括图像采集模组、图像处理模组和图像显示模组：
所述图像采集模组，用于获取待分割图像；所述待分割图像为包括生物组织的病理图像；
所述图像处理模组，用于通过图像区域分割模型生成所述待分割图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述 待分割图像中至少一个像素点；所述图像区域分割模型是根据第二标注信息训练得到的，所述第二标注信息是根据样本图像对应的图结构数据和所述样本图样对应的第一标注信息确定的；所述第二标注信息的粒度小于所述第一标注信息的粒度；通过所述图像区域分割模型，根据所述图结构数据分割得到所述待分割图像中的病灶；
所述图像显示模组，用于显示所述病灶，以便医生可以通过医疗器械显示的病灶进行观察、分析等。
在一种可能的是实现方式中，所述图像采集模组，用于获取样本图像集合，所述样本图像集合包括多个样本图像，所述样本图像具有其对应的第一标注信息；
所述图像处理模组，用于：
针对所述样本图像集合中的目标样本图像，生成所述目标样本图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述目标样本图像中至少一个像素点；所述目标样本图像为所述样本图像集合中的任一个样本图像；
通过图卷积网络模型，根据所述目标样本图像对应的图结构数据和所述目标样本图像对应的第一标注信息，确定所述顶点的第二标注信息；所述第二标注信息的粒度小于所述第一标注信息的粒度；所述图卷积网络模型为所述图形区域分割模型的一部分；
根据所述第二标注信息对所述图像区域分割模型进行训练。
在一种可能的是实现方式中，所述第一标注信息包括目标区域在样本图像中的面积占比，所述图像处理模组，用于：
通过图卷积网络模型，根据所述图结构数据得到每个顶点的预测结果；
根据所述面积占比和所述图结构数据中顶点的总数量，确定所述目标区域对应的第一顶点数量以及背景区域对应的第二顶点数量，所述背景区域为所述样本图像中除所述目标区域外的区域；
根据所述第一顶点数量、所述第二顶点数量和所述预测结果，确定所述顶点的第二标注信息。
在一种可能的是实现方式中，所述第一标注信息还包括估计误差值，所述图像处理模组，用于：
根据所述面积占比、所述估计误差值和所述图结构数据中顶点的总数量，确定所述目标区域对应的第一顶点数量以及背景区域对应的第二顶点数量。
在一种可能的是实现方式中，所述图像处理模组，用于：
根据每个顶点的所述预测结果和第二标注信息构建损失函数；
根据所述损失函数对所述图像区域分割模型进行训练。
在一种可能的是实现方式中，所述图像区域分割模型中包括全卷积网 络，所述图像处理模组，用于：
通过全卷积网络对所述目标样本图像进行特征提取，得到所述目标样本图像中每个像素点对应的特征；
根据所述像素点对应的特征对所述目标样本图像中的像素点进行聚类，得到超像素划分结果；
根据所述超像素划分结果构建所述图结构数据，所述超像素划分结果中每个超像素点作为所述图结构数据中的一个顶点。
在一种可能的是实现方式中，所述图像处理模组，用于：
根据第一顶点和第二顶点各自包括的像素点的位置信息，确定所述第一顶点与所述第二顶点之间的边；所述第一顶点和所述第二顶点可以为任意两个顶点；
根据所述第一顶点和所述第二顶点各自对应的特征之间的欧氏距离，确定边的权重。
在一种可能的是实现方式中，针对所述图结构数据中每个顶点，顶点对应的特征为所述顶点所包括像素点的特征的平均值。
基于前述图2所对应的实施例，本申请实施例还提供一种图像区域分割模型训练装置，参见图8，所述装置包括获取单元801、生成单元802、确定单元803和训练单元804：
所述获取单元801，用于获取样本图像集合，所述样本图像集合包括至少一个样本图像，每个所述样本图像具有其对应的第一标注信息；
所述生成单元802，用于所述针对所述样本图像集合中的目标样本图像，生成所述目标样本图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述目标样本图像中至少一个像素点；所述目标样本图像为所述样本图像集合中的任一个样本图像；
所述确定单元803，用于通过图卷积网络模型，根据所述目标样本图像对应的图结构数据和所述目标样本图像对应的所述第一标注信息确定所述顶点的第二标注信息；所述第二标注信息的粒度小于所述第一标注信息的粒度；所述图卷积网络模型为所述图形区域分割模型的一部分；
所述训练单元804，用于根据所述第二标注信息对所述图像区域分割模型进行训练。
在一种可能的实现方式中，所述第一标注信息包括目标区域在样本图像中的面积占比，所述确定单元803，用于：
通过图卷积网络模型，根据所述图结构数据得到每个顶点的预测结果；
根据所述面积占比和所述图结构数据中顶点的总数量，确定所述目标区域对应的第一顶点数量以及背景区域对应的第二顶点数量，所述背景区域为所述样本图像中出所述目标区域外的区域；
根据所述第一顶点数量、所述第二顶点数量和所述预测结果，确定所述顶点的第二标注信息。
在一种可能的实现方式中，所述第一标注信息还包括估计误差值，所述确定单元803，用于：
根据所述面积占比、所述估计误差值和所述图结构数据中顶点的总数量，确定所述目标区域对应的第一顶点数量以及背景区域对应的第二顶点数量。
在一种可能的实现方式中，所述训练单元804，用于：
根据每个顶点的所述预测结果和第二标注信息构建损失函数；
根据所述损失函数对所述图像区域分割模型进行训练。
在一种可能的实现方式中，所述图像区域分割模型中包括全卷积网络，所述生成单元802，用于：
通过全卷积网络对所述目标样本图像进行特征提取，得到所述目标样本图像中每个像素点对应的特征；
根据所述像素点对应的特征对所述目标样本图像中的像素点进行聚类，得到超像素划分结果；
根据所述超像素划分结果构建所述图结构数据，所述超像素划分结果中每个超像素点作为所述图结构数据中的一个顶点。
在一种可能的实现方式中，所述生成单元802，用于：
根据第一顶点和第二顶点各自包括的像素点的位置信息，确定所述第一顶点与所述第二顶点之间的边；所述第一顶点和所述第二顶点为任意两个顶点；
根据第一顶点和第二顶点各自对应的特征之间的欧氏距离确定边的权重。
在一种可能的实现方式中，针对所述图结构数据中每个顶点，顶点对应的特征为所述顶点包括的像素点的特征的平均值。
基于前述图4所对应的实施例，本申请实施例还一种基于人工智能的图像区域分割装置，参见图9，所述装置包括获取单元901、生成单元902和分割单元903：
所述获取单元901，用于获取待分割图像；
所述生成单元902，用于通过图像区域分割模型生成所述待分割图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述待分割图像中至少一个像素点；所述图像区域分割模型是根据第二标注信息训练得到的，所述第二标注信息是根据样本图像对应的图结构数据和所述样本图样对应的第一标注信息确定的；所述第二标注信息的粒度小于所述第一标注信息的粒度；
所述分割单元903，用于通过所述图像区域分割模型，根据所述待分割 图像对应的图结构数据分割得到所述待分割图像中的目标区域。
在一种可能的实现方式中，所述待分割图像为病理图像，所述目标区域为病灶。
本申请实施例还提供了一种设备，该设备可以实现上文中的图像区域分割模型训练方法或图像区域分割方法。下面结合附图对该设备进行介绍。请参见图10所示，本申请实施例提供了一种的设备1000，该设备1000还可以是终端设备，该终端设备可以为包括计算机、平板电脑、手机、个人数字助理(Personal Digital Assistant，简称PDA)、销售终端(Point of Sales，简称POS)、车载电脑等，以终端设备为手机为例：
图10示出的是与本申请实施例提供的终端设备相关的手机的部分结构的框图。参考图10，手机包括：射频(Radio Frequency，简称RF)电路1010、存储器1020、输入单元1030、显示单元1040、传感器1050、音频电路1060、无线保真(wireless fidelity，简称WiFi)模块1070、处理器1080、以及电源1090等部件。本领域技术人员可以理解，图10中示出的手机结构并不构成对手机的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
在本实施例中，该终端设备所包括的处理器1080还具有以下功能：
获取样本图像集合，所述样本图像集合包括多个样本图像，每个所述样本图像具有其对应的第一标注信息；
针对所述样本图像集合中的目标样本图像，生成所述目标样本图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述目标样本图像中至少一个像素点；所述目标样本图像为所述样本图像集合中的任一个样本图像；
通过图卷积网络模型，根据所述目标样本图像对应的图结构数据和所述目标样本图像对应的所述第一标注信息，确定所述顶点的第二标注信息；所述第二标注信息的粒度小于所述第一标注信息的粒度；所述图卷积网络模型为所述图形区域分割模型的一部分；
根据所述第二标注信息对所述图像区域分割模型进行训练。
或，
获取待分割图像；
通过图像区域分割模型生成所述待分割图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述待分割图像中至少一个像素点；所述图像区域分割模型是根据第二标注信息训练得到的，所述第二标注信息是根据样本图像对应的图结构数据和所述样本图样对应的第一标注信息确定的；所述第二标注信息的粒度小于所述第一标注信息的粒度；
通过所述图像区域分割模型，根据所述待分割图像对应的图结构数据分 割得到所述待分割图像中的目标区域。
本申请实施例还提供服务器，请参见图11所示，图11为本申请实施例提供的服务器1100的结构图，服务器1100可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上中央处理器(Central Processing Units，简称CPU)1122(例如，一个或一个以上处理器)和存储器1132，一个或一个以上存储应用程序1142或数据1144的存储介质1130(例如一个或一个以上海量存储设备)。其中，存储器1132和存储介质1130可以是短暂存储或持久存储。存储在存储介质1130的程序可以包括一个或一个以上模块(图示没标出)，每个模块可以包括对服务器中的一系列指令操作。更进一步地，中央处理器1122可以设置为与存储介质1130通信，在服务器1100上执行存储介质1130中的一系列指令操作。
服务器1100还可以包括一个或一个以上电源1126，一个或一个以上有线或无线网络接口1150，一个或一个以上输入输出接口1158，和/或，一个或一个以上操作系统1141，例如Windows ServerTM，Mac OS XTM，UnixTM,LinuxTM，FreeBSDTM等等。
上述实施例中由服务器所执行的步骤可以基于该图11所示的服务器结构执行。
本申请实施例还提供一种计算机可读存储介质，所述计算机可读存储介质用于存储程序代码，所述程序代码用于执行前述各个实施例所述的方法。
本申请实施例还提供一种包括指令的计算机程序产品，当其在计算机上运行时，使得计算机执行前述各个实施例所述的方法。
以上所述，以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (16)

  1. 一种图像区域分割模型训练方法，由数据处理设备执行，所述方法包括：
    获取样本图像集合，所述样本图像集合包括多个样本图像，每个所述样本图像具有其对应的第一标注信息；
    针对所述样本图像集合中的目标样本图像，生成所述目标样本图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述目标样本图像中至少一个像素点；所述目标样本图像为所述样本图像集合中的任一个样本图像；
    通过图卷积网络模型，根据所述目标样本图像对应的图结构数据和所述目标样本图像对应的所述第一标注信息，确定所述顶点的第二标注信息；所述第二标注信息的粒度小于所述第一标注信息的粒度；所述图卷积网络模型为所述图形区域分割模型的一部分；
    根据所述第二标注信息对所述图像区域分割模型进行训练。
  2. 根据权利要求1所述的方法，所述第一标注信息包括目标区域在样本图像中的面积占比；所述通过图卷积网络模型，根据所述目标样本图像对应的图结构数据和所述目标样本图像对应的所述第一标注信息，确定所述顶点的第二标注信息，包括：
    通过所述图卷积网络模型，根据所述图结构数据得到每个顶点的预测结果；
    根据所述面积占比和所述图结构数据中顶点的总数量，确定所述目标区域对应的第一顶点数量以及背景区域对应的第二顶点数量，所述背景区域为所述样本图像中除所述目标区域外的区域；
    根据所述第一顶点数量、所述第二顶点数量和所述预测结果，确定所述顶点的第二标注信息。
  3. 根据权利要求2所述的方法，所述第一标注信息还包括估计误差值；所述根据所述面积占比和所述图结构数据中顶点的总数量，确定所述目标区域对应的第一顶点数量以及背景区域对应的第二顶点数量，包括：
    根据所述面积占比、所述估计误差值和所述图结构数据中顶点的总数量，确定所述第一顶点数量以及所述第二顶点数量。
  4. 根据权利要求2所述的方法，所述根据所述第二标注信息对所述图像区域分割模型进行训练，包括：
    根据每个顶点的所述预测结果和第二标注信息构建损失函数；
    根据所述损失函数对所述图像区域分割模型进行训练。
  5. 根据权利要求1-4任一项所述的方法，所述图像区域分割模型中包括全卷积网络，所述针对所述样本图像集合中的目标样本图像，生成所述目标样本图像对应的图结构数据，包括：
    通过所述全卷积网络对所述目标样本图像进行特征提取，得到所述目标样本图像中每个像素点对应的特征；
    根据所述像素点对应的特征对所述目标样本图像中的像素点进行聚类，得到超像素划分结果；
    根据所述超像素划分结果构建所述图结构数据，所述超像素划分结果中每个超像素点作为所述图结构数据中的一个顶点。
  6. 根据权利要求5所述的方法，所述根据所述超像素划分结果构建所述图结构数据，包括：
    根据第一顶点和第二顶点各自包括的像素点的位置信息，确定所述第一顶点与所述第二顶点之间的边；所述第一顶点和所述第二顶点为任意两个顶点；
    根据所述第一顶点和所述第二顶点各自对应的特征之间的欧氏距离，确定所述边的权重。
  7. 根据权利要6所述的方法，针对所述图结构数据中每个顶点，所述顶点对应的特征为所述顶点包括的像素点的特征的平均值。
  8. 一种图像区域分割模型训练装置，所述装置包括获取单元、生成单元、确定单元和训练单元：
    所述获取单元，用于获取样本图像集合，所述样本图像集合包括至少一个样本图像，每个所述样本图像具有其对应的第一标注信息；
    所述生成单元，用于所述针对所述样本图像集合中的目标样本图像，生成所述目标样本图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述目标样本图像中至少一个像素点；所述目标样本图像为所述样本图像集合中的任一个样本图像；
    所述确定单元，用于通过图卷积网络模型，根据所述目标样本图像对应的图结构数据和所述目标样本图像对应的所述第一标注信息，确定所述顶点的第二标注信息；所述第二标注信息的粒度小于所述第一标注信息的粒度；所述图卷积网络模型为所述图形区域分割模型的一部分；
    所述训练单元，用于根据所述第二标注信息对所述图像区域分割模型进行训练。
  9. 根据权利要求8所述的装置，所述第一标注信息包括目标区域在样本图像中的面积占比，所述确定单元具体用于：
    通过所述图卷积网络模型，根据所述图结构数据得到每个顶点的预测结果；
    根据所述面积占比和所述图结构数据中顶点的总数量，确定所述目标区域对应的第一顶点数量以及背景区域对应的第二顶点数量，所述背景区域为所述样本图像中除所述目标区域外的区域；
    根据所述第一顶点数量、所述第二顶点数量和所述预测结果，确定所述 顶点的第二标注信息。
  10. 一种图像区域分割方法，由数据处理设备执行，所述方法包括：
    获取待分割图像；
    通过图像区域分割模型生成所述待分割图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述待分割图像中至少一个像素点；所述图像区域分割模型是根据第二标注信息训练得到的，所述第二标注信息是根据样本图像对应的图结构数据和所述样本图样对应的第一标注信息确定的；所述第二标注信息的粒度小于所述第一标注信息的粒度；
    通过所述图像区域分割模型，根据所述待分割图像对应的图结构数据分割得到所述待分割图像中的目标区域。
  11. 根据权利要求10所述的方法，所述待分割图像为病理图像，所述目标区域为病灶。
  12. 一种图像区域分割装置，所述装置包括获取单元、生成单元和分割单元：
    所述获取单元，用于获取待分割图像；
    所述生成单元，用于通过图像区域分割模型生成所述待分割图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述待分割图像中至少一个像素点；所述图像区域分割模型是根据第二标注信息训练得到的，所述第二标注信息是根据样本图像对应的图结构数据和所述样本图样对应的第一标注信息确定的；所述第二标注信息的粒度小于所述第一标注信息的粒度；
    所述分割单元，用于通过所述图像区域分割模型，根据所述待分割图像对应的图结构数据分割得到所述待分割图像中的目标区域。
  13. 一种医疗器械，所述医疗器械包括图像采集模组、图像处理模组和图像显示模组：
    所述图像采集模组，用于获取待分割图像；所述待分割图像为包括生物组织的病理图像；
    所述图像处理模组，用于通过图像区域分割模型生成所述待分割图像对应的图结构数据；所述图结构数据中包括多个顶点，每个所述顶点包括所述待分割图像中至少一个像素点；所述图像区域分割模型是根据第二标注信息训练得到的，所述第二标注信息是根据样本图像对应的图结构数据和所述样本图样对应的第一标注信息确定的；所述第二标注信息的粒度小于所述第一标注信息的粒度；通过所述图像区域分割模型，根据所述图结构数据分割得到所述待分割图像中的病灶；
    所述图像显示模组，用于显示所述病灶。
  14. 一种设备，所述设备包括处理器以及存储器：
    所述存储器用于存储程序代码，并将所述程序代码传输给所述处理器；
    所述处理器用于根据所述程序代码中的指令执行权利要求1-7或10-11任一项所述的方法。
  15. 一种计算机可读存储介质，所述计算机可读存储介质用于存储程序代码，所述程序代码用于执行权利要求1-7或10-11任一项所述的方法。
  16. 一种计算机程序产品，包括指令，当其在计算机上运行时，使得计算机执行权利要求1-7或10-11任一项所述的方法。
PCT/CN2021/087128 2020-05-18 2021-04-14 一种图像区域分割模型训练方法、分割方法和装置 WO2021233021A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/707,045 US20220222932A1 (en) 2020-05-18 2022-03-29 Training method and apparatus for image region segmentation model, and image region segmentation method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010419791.7 2020-05-18
CN202010419791.7A CN111598900B (zh) 2020-05-18 2020-05-18 一种图像区域分割模型训练方法、分割方法和装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/707,045 Continuation US20220222932A1 (en) 2020-05-18 2022-03-29 Training method and apparatus for image region segmentation model, and image region segmentation method and apparatus

Publications (1)

Publication Number Publication Date
WO2021233021A1 true WO2021233021A1 (zh) 2021-11-25

Family

ID=72187287

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/087128 WO2021233021A1 (zh) 2020-05-18 2021-04-14 一种图像区域分割模型训练方法、分割方法和装置

Country Status (3)

Country Link
US (1) US20220222932A1 (zh)
CN (1) CN111598900B (zh)
WO (1) WO2021233021A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114464316A (zh) * 2022-04-11 2022-05-10 武汉大学 胃部异常风险等级预测方法、装置、终端及可读存储介质

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111598900B (zh) * 2020-05-18 2022-08-09 腾讯医疗健康(深圳)有限公司 一种图像区域分割模型训练方法、分割方法和装置
CN112102929A (zh) * 2020-09-11 2020-12-18 沈阳东软智能医疗科技研究院有限公司 医学图像标注方法、装置、存储介质及电子设备
CN111932547B (zh) * 2020-09-24 2021-06-11 平安科技（深圳）有限公司 图像中目标物的分割方法、装置、电子设备及存储介质
CN114511443A (zh) * 2020-10-29 2022-05-17 北京中祥英科技有限公司 图像处理、图像识别网络训练和图像识别的方法及装置
CN112381811B (zh) * 2020-11-20 2024-05-28 沈阳东软智能医疗科技研究院有限公司 一种实现医学影像数据标注的方法、装置及设备
CN112365513A (zh) * 2020-11-23 2021-02-12 北京三快在线科技有限公司 一种模型训练的方法及装置
CN112541928A (zh) * 2020-12-18 2021-03-23 上海商汤智能科技有限公司 网络训练方法及装置、图像分割方法及装置和电子设备
CN115477211B (zh) * 2021-06-15 2023-10-27 中移(成都)信息通信科技有限公司 一种电梯停靠方法、装置、设备及存储介质
TWI805290B (zh) * 2022-03-28 2023-06-11 臺北醫學大學 用於預測肺腺癌是否具有表皮生長因子受體突變的方法
CN114782379A (zh) * 2022-04-27 2022-07-22 北京航空航天大学 一种基于结构信息原理的图像层谱分割方法
CN115439846B (zh) * 2022-08-09 2023-04-25 北京邮电大学 图像的分割方法、装置、电子设备及介质
CN115937516B (zh) * 2022-11-21 2023-10-03 北京邮电大学 一种图像语义分割方法、装置、存储介质及终端
CN116205928B (zh) * 2023-05-06 2023-07-18 南方医科大学珠江医院 腹腔镜手术视频的图像分割处理方法、装置和设备、介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150016668A1 (en) * 2013-07-12 2015-01-15 Ut-Battelle, Llc Settlement mapping systems
CN106651886A (zh) * 2017-01-03 2017-05-10 北京工业大学 一种基于超像素聚类优化cnn的云图分割方法
CN107424159A (zh) * 2017-07-28 2017-12-01 西安电子科技大学 基于超像素边缘和全卷积网络的图像语义分割方法
CN110659692A (zh) * 2019-09-26 2020-01-07 重庆大学 基于强化学习和深度神经网络的病理图像自动标注方法
CN111598900A (zh) * 2020-05-18 2020-08-28 腾讯科技（深圳）有限公司 一种图像区域分割模型训练方法、分割方法和装置

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9886758B2 (en) * 2016-03-31 2018-02-06 International Business Machines Corporation Annotation of skin image using learned feature representation
US10366490B2 (en) * 2017-03-27 2019-07-30 Siemens Healthcare Gmbh Highly integrated annotation and segmentation system for medical imaging
CN110796661B (zh) * 2018-08-01 2022-05-31 华中科技大学 基于卷积神经网络的真菌显微图像分割检测方法及系统
CN110807754B (zh) * 2018-08-01 2022-05-31 华中科技大学 基于深度语义分割的真菌显微图像分割检测方法及系统
CN109447998B (zh) * 2018-09-29 2020-12-18 华中科技大学 基于PCANet深度学习模型下的自动分割方法
CN109671068B (zh) * 2018-12-14 2022-09-13 复旦大学附属中山医院 一种基于深度学习的腹部肌肉标注方法及装置
US11282208B2 (en) * 2018-12-24 2022-03-22 Adobe Inc. Identifying target objects using scale-diverse segmentation neural networks
JP7568628B2 (ja) * 2019-01-07 2024-10-16 エクシーニ ディアグノスティクス アーべー プラットホーム非依存の全身画像セグメント化のためのシステムおよび方法
CN109961444B (zh) * 2019-03-01 2022-12-20 腾讯科技（深圳）有限公司 图像处理方法、装置及电子设备
CN109903314A (zh) * 2019-03-13 2019-06-18 腾讯科技（深圳）有限公司 一种图像区域定位的方法、模型训练的方法及相关装置
CN110070531B (zh) * 2019-04-19 2021-05-07 京东方科技集团股份有限公司 用于检测眼底图片的模型训练方法、眼底图片的检测方法及装置
CN110210487A (zh) * 2019-05-30 2019-09-06 上海商汤智能科技有限公司 一种图像分割方法及装置、电子设备和存储介质
CN110232696B (zh) * 2019-06-20 2024-03-08 腾讯科技（深圳）有限公司 一种图像区域分割的方法、模型训练的方法及装置
CN110503654B (zh) * 2019-08-01 2022-04-26 中国科学院深圳先进技术研究院 一种基于生成对抗网络的医学图像分割方法、系统及电子设备
CN110853009B (zh) * 2019-11-11 2023-04-28 北京端点医药研究开发有限公司 基于机器学习的视网膜病理图像分析系统
US20220343623A1 (en) * 2019-11-28 2022-10-27 Beijing Xiaoying Technology Co., Ltd. Blood smear full-view intelligent analysis method, and blood cell segmentation model and recognition model construction method
JP7237870B2 (ja) * 2020-02-06 2023-03-13 株式会社日立ソリューションズ 画像処理装置、画像処理方法及び画像処理システム
CN111640120B (zh) * 2020-04-09 2023-08-29 之江实验室 一种基于显著性密集连接扩张卷积网络的胰腺ct自动分割方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150016668A1 (en) * 2013-07-12 2015-01-15 Ut-Battelle, Llc Settlement mapping systems
CN106651886A (zh) * 2017-01-03 2017-05-10 北京工业大学 一种基于超像素聚类优化cnn的云图分割方法
CN107424159A (zh) * 2017-07-28 2017-12-01 西安电子科技大学 基于超像素边缘和全卷积网络的图像语义分割方法
CN110659692A (zh) * 2019-09-26 2020-01-07 重庆大学 基于强化学习和深度神经网络的病理图像自动标注方法
CN111598900A (zh) * 2020-05-18 2020-08-28 腾讯科技（深圳）有限公司 一种图像区域分割模型训练方法、分割方法和装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114464316A (zh) * 2022-04-11 2022-05-10 武汉大学 胃部异常风险等级预测方法、装置、终端及可读存储介质
CN114464316B (zh) * 2022-04-11 2022-07-19 武汉大学 胃部异常风险等级预测方法、装置、终端及可读存储介质

Also Published As

Publication number Publication date
CN111598900B (zh) 2022-08-09
CN111598900A (zh) 2020-08-28
US20220222932A1 (en) 2022-07-14

Similar Documents

Publication Publication Date Title
WO2021233021A1 (zh) 一种图像区域分割模型训练方法、分割方法和装置
CN110321920B (zh) 图像分类方法、装置、计算机可读存储介质和计算机设备
CN110390351B (zh) 一种基于深度学习的致痫灶三维自动定位系统
CN110475505B (zh) 利用全卷积网络的自动分割
CN110689038B (zh) 神经网络模型的训练方法、装置和医学图像处理系统
CN111640120B (zh) 一种基于显著性密集连接扩张卷积网络的胰腺ct自动分割方法
Guo et al. Dual attention enhancement feature fusion network for segmentation and quantitative analysis of paediatric echocardiography
CN110414631A (zh) 基于医学图像的病灶检测方法、模型训练的方法及装置
TW202125415A (zh) 三維目標檢測及模型的訓練方法、設備、儲存媒體
CN112396605B (zh) 网络训练方法及装置、图像识别方法和电子设备
WO2019184851A1 (zh) 图像处理方法和装置及神经网络模型的训练方法
WO2019037654A1 (zh) 3d图像检测方法、装置、电子设备及计算机可读介质
Duan et al. A novel GA-based optimized approach for regional multimodal medical image fusion with superpixel segmentation
Mahapatra et al. Weakly supervised semantic segmentation of Crohn's disease tissues from abdominal MRI
CN113177592B (zh) 一种图像分割方法、装置、计算机设备及存储介质
CN111583385A (zh) 一种可变形数字人解剖学模型的个性化变形方法及系统
Ji et al. Continual segment: Towards a single, unified and non-forgetting continual segmentation model of 143 whole-body organs in ct scans
Kitrungrotsakul et al. Interactive deep refinement network for medical image segmentation
CN115131290A (zh) 图像处理方法
Shao et al. Application of an improved u2-net model in ultrasound median neural image segmentation
CN112686932B (zh) 用于医学影像的图像配准方法及图像处理方法、介质
CN111209946B (zh) 三维图像处理方法、图像处理模型训练方法及介质
WO2023104464A1 (en) Selecting training data for annotation
CN115937163A (zh) 一种spect肺灌注成像的目标区域提取方法与系统
CN113327221B (zh) 融合roi区域的图像合成方法、装置、电子设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21808005

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 13.04.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21808005

Country of ref document: EP

Kind code of ref document: A1