TWI777092B - Image processing method, electronic device, and storage medium - Google Patents

Abstract

The embodiment of that present application disclose an image processing method, an electronic device and a storage medium, the methods include: subject that first image to processing, obtaining a prediction result of a plurality of pixel points in the first image, the prediction result includes a semantic prediction result indicating that the pixel point is located in an instance region or a background region and a center relative position prediction result indicating a relative position between the pixel point and the instance center; determining the instance segmentation result of the first image based on the semantic prediction result and the center relative position prediction result of each pixel of the plurality of pixels can make the instance segmentation in the image processing have the advantages of high speed and high precision.

Description

An image processing method, electronic device and storage medium

The present application relates to the field of computer vision technology, and in particular, to an image processing method, an electronic device and a storage medium.

Image processing, also known as image processing, is a technology that uses a computer to analyze images to achieve the desired results. Image processing generally refers to bit image processing. Digital image refers to a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of the array are called pixels, and their values are called gray. degree value. Image processing plays a very important role in many fields, especially the processing of medical images.

Embodiments of the present application provide an image processing method, an electronic device, and a storage medium.

A first aspect of the embodiments of the present application provides an image processing method, including: processing a first image to obtain prediction results of multiple pixels in the first image, where the prediction results include semantic prediction results and A center relative position prediction result, wherein the semantic prediction result indicates that the pixel is located in an instance area or a background area, and the center relative position prediction result indicates a relative position between the pixel point and the instance center; based on the multiple The semantic prediction result and the center relative position prediction result of each pixel point in the pixel points determine the instance segmentation result of the first image.

Optionally, the processing the first image to obtain semantic prediction results of multiple pixels in the first image includes: processing the first image to obtain the first image in the first image. The instance area prediction probability of a plurality of pixel points, the instance area prediction probability indicates the probability that the pixel point is located in the instance area; based on the second threshold, the instance area prediction probability of the above-mentioned multiple pixel points is binarized to obtain the Semantic prediction results for each pixel in multiple pixels.

Optionally, the instance central area includes: an area within the instance area and smaller than the instance area, and the geometric center of the instance central area overlaps the geometric center of the instance area.

In an optional implementation manner, before processing the first image, the method further includes: preprocessing the second image to obtain the first image, so that the first image The preset contrast and/or the preset gray value are satisfied.

In an optional implementation manner, before processing the first image, the method further includes: preprocessing the second image to obtain the first image, so that the The first image satisfies the preset image size.

In an optional implementation manner, the determining the instance segmentation result of the first image based on the semantic prediction result and the center relative position prediction result of each pixel point in the plurality of pixel points includes: the semantic prediction result of each pixel point in the plurality of pixel points, and at least one first pixel point located in the instance area is determined from the plurality of pixel points; based on the prediction result of the relative position of the center of the first pixel point, determine The instance to which each first pixel belongs.

The instance is a segmented object in the first image, and may specifically be a closed structure in the first image.

Examples in the embodiments of the present application include cell nuclei, that is, the embodiments of the present application can be applied to cell nucleus segmentation.

In an optional embodiment, the prediction result further includes: a center area prediction result, where the center area prediction result indicates whether the pixel point is located in the center area of the instance. In this case, the method further includes: determining at least one instance center region of the first image based on the prediction result of the center region of each pixel point in the plurality of pixel points; The prediction result of the relative center position of the pixel point, and determining the instance to which the first pixel point belongs includes: based on the prediction result of the relative center position of the first pixel point, determining the each of the center regions of the at least one instance. The central area of the instance corresponding to the first pixel.

In an optional implementation manner, the determining at least one instance central region of the first image based on the prediction result of the central region of each pixel point in the plurality of pixel points includes: based on the plurality of pixel points Based on the prediction result of the central area of each pixel in the pixel points, a connected domain search process is performed on the first image to obtain at least one instance central area.

In an optional implementation manner, performing a connected domain search process on the first image based on the prediction result of the center area of each pixel point in the plurality of pixel points, and obtaining at least one instance center area includes: Based on the prediction result of the center area of each pixel point in the plurality of pixel points, a random walk algorithm is used to perform a connected domain search process on the first image to obtain at least one instance center area.

In an optional implementation manner, the determining an instance central area corresponding to the first pixel point from the at least one instance central area based on the prediction result of the relative center position of the first pixel point includes: based on The position information of the first pixel point and the prediction result of the relative position of the center of the first pixel point, determine the center prediction position of the first pixel point; based on the center prediction position of the first pixel point and the at least The location information of an instance central area, and the instance central area corresponding to the first pixel point is determined from the at least one instance central area.

In an optional implementation manner, the first pixel point is determined from the at least one instance center area based on the predicted center position of the first pixel point and the position information of the at least one instance center area The corresponding instance central area includes: in response to the predicted center position of the first pixel belonging to the first instance central area in the at least one instance central area, determining the first instance central area as the first instance central area The central area of the instance corresponding to the pixel point.

In an optional implementation manner, the first pixel point is determined from the at least one instance center area based on the predicted center position of the first pixel point and the position information of the at least one instance center area The corresponding instance central area includes: in response to the predicted center position of the first pixel point not belonging to any instance central area in the at least one instance central area, comparing the at least one instance central area with the first instance central area. The instance central area with the closest distance from the center prediction position of the pixel point is determined as the instance central area corresponding to the first pixel point.

In an optional implementation manner, the processing the first image to obtain the prediction results of multiple pixels in the first image includes: processing the first image to obtain the The prediction probability of the central area of the plurality of pixels in the first image; based on the first threshold, the central area prediction probability of the plurality of pixels is binarized to obtain the prediction probability of each pixel in the plurality of pixels. Center area prediction results.

In an optional implementation manner, the processing of the first image to obtain prediction results of multiple pixels in the first image includes: inputting the first image into a neural network for processing, The prediction results of the plurality of pixels in the first image are output.

A second aspect of an embodiment of the present application provides an electronic device, including a prediction module and a segmentation module, wherein: the prediction module is configured to process a first image to obtain predictions of multiple pixels in the first image As a result, the prediction result includes a semantic prediction result and a center relative position prediction result, wherein the semantic prediction result indicates that the pixel point is located in the instance area or the background area, and the center relative position prediction result indicates that the pixel point is located in the instance area or the background area. The relative position between the centers; the segmentation module is configured to determine the instance segmentation result of the first image based on the semantic prediction result of each pixel point in the plurality of pixel points and the prediction result of the relative position of the center.

Optionally, the prediction module is specifically configured to: process the first image to obtain an instance area prediction probability of a plurality of pixels in the first image, and the instance area prediction probability indicates the pixel point. Probability of being located in the instance area; binarizing the instance area prediction probability of the plurality of pixel points based on the second threshold to obtain a semantic prediction result of each pixel point in the plurality of pixel points.

In an optional implementation manner, the electronic device further includes a preprocessing module, configured to preprocess the second image to obtain the first image, so that the first image satisfies a preset contrast ratio and/or preset grayscale values.

In an optional implementation manner, the preprocessing module is further configured to preprocess the second image to obtain the first image, so that the first image satisfies a preset image size.

In an optional implementation manner, the segmentation module includes a first unit and a second unit, wherein: the first unit is configured to, based on the semantic prediction result of each pixel point in the plurality of pixel points, from Determining at least one first pixel point located in the instance area among the plurality of pixel points; the second unit is configured to determine each first pixel point based on the prediction result of the relative position of the center of the first pixel point the instance to which it belongs.

In an optional embodiment, the prediction result further includes: a center area prediction result, where the center area prediction result indicates whether the pixel point is located in the center area of the instance. In this case, the segmentation module further includes a third unit, configured to determine at least one instance center area of the first image based on the center area prediction result of each pixel point in the plurality of pixel points; the The second unit is specifically configured to, based on the prediction result of the relative position of the center of the first pixel point, determine the instance center area corresponding to each first pixel point from the at least one instance center area.

In an optional implementation manner, the third unit is specifically configured to, based on the prediction result of the central area of each pixel in the plurality of pixels, perform a connected domain search process on the first image, to obtain At least one instance central area.

In an optional implementation manner, the third unit is specifically configured to, based on the prediction result of the central area of each pixel point in the plurality of pixel points, use a random walk algorithm to perform a connected domain analysis on the first image The search process yields at least one instance center region.

In an optional implementation manner, the second unit is specifically configured to: determine the position of the first pixel based on the position information of the first pixel and the prediction result of the relative position of the center of the first pixel. Center prediction position; based on the center prediction position of the first pixel point and the position information of the at least one instance center area, determine the instance center area corresponding to the first pixel point from the at least one instance center area.

In an optional implementation manner, the second unit is specifically configured to: in response to that the predicted center position of the first pixel belongs to a first instance center area in the at least one instance center area, convert the first instance center area to the first instance center area. An example center area is determined as an example center area corresponding to the first pixel point.

In an optional implementation manner, the second unit is specifically configured to: in response to that the predicted center position of the first pixel does not belong to any instance center area in the at least one instance center area, change the at least one instance center area to the at least one instance center area. In an instance central area, the instance central area that is closest to the predicted center position of the first pixel point is determined as the instance central area corresponding to the first pixel point.

In an optional implementation manner, the prediction module includes a probability prediction unit and a judgment unit, wherein: the probability prediction unit is configured to process the first image to obtain a probability prediction unit in the first image. The prediction probability of the central area of the pixel points; the judgment unit is configured to perform binarization processing on the prediction probability of the central area of the plurality of pixel points based on the first threshold, to obtain each pixel point of the plurality of pixel points The prediction result of the central area.

In an optional implementation manner, the prediction module is specifically configured to input the first image into a neural network for processing, and output prediction results of multiple pixels in the first image.

A third aspect of an embodiment of the present application provides another electronic device, including a processor and a memory, where the memory is used for storing a computer program, the computer program is configured to be executed by the processor, and the processor is used for executing Part or all of the steps described in any method in the first aspect of the embodiments of the present application.

A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, where the computer-readable storage medium is used to store a computer program, wherein the computer program causes the computer to execute the method described in any of the methods in the first aspect of the embodiments of the present application. some or all of the steps described.

In the embodiment of the present application, by processing the first image, the prediction results of multiple pixels in the first image are obtained, and the prediction results include semantic prediction results and center relative position prediction results, wherein the semantic prediction results Indicate that the above-mentioned pixel points are located in the instance area or the background area, and the above-mentioned relative position prediction result of the center indicates the relative position between the above-mentioned pixel point and the center of the instance, based on the semantic prediction result of each pixel point in the above-mentioned multiple pixel points and the relative position prediction of the center As a result, by determining the instance segmentation result of the first image, the instance segmentation in image processing can have the advantages of high speed and high precision.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

The terms "first", "second" and the like in the description and the scope of the patent application of the present application and the above drawings are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally also includes For other steps or units inherent to these processes, methods, products or devices.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

The electronic device involved in the embodiments of the present application may allow access by multiple other terminal devices. The aforementioned electronic devices include terminal devices, which in specific implementations include, but are not limited to, other portable devices such as mobile phones, laptop computers, or tablet computers with touch-sensitive surfaces (eg, touch screen displays and/or touch pads). equipment. It should also be understood that, in some embodiments, the terminal device is not a portable communication device, but a desktop computer with a touch-sensitive surface (eg, a touch screen display and/or a touch pad).

The concept of deep learning in the embodiments of the present application originates from the research of artificial neural network. A multilayer perceptron with multiple hidden layers is a deep learning structure. Deep learning combines low-level features to form more abstract high-level representation attribute categories or features to discover distributed feature representations of data.

Deep learning is a method in machine learning based on representational learning of data. An observation (such as an image) can be represented in a variety of ways, such as a vector of intensity values for each pixel, or more abstractly as a series of edges, regions of a specific shape, etc. Instead, it is easier to learn tasks from examples (e.g., face recognition or facial expression recognition) using some specific representation. The benefit of deep learning is to replace handcrafted features with efficient algorithms for unsupervised or semi-supervised feature learning and hierarchical feature extraction. Deep learning is a new field in machine learning research. Its motivation is to build and simulate the neural network of the human brain for analysis and learning, so that the mechanism of the human brain can be imitated to interpret data, such as images, sounds and texts.

Like machine learning methods, deep machine learning methods also have supervised learning and unsupervised learning. The learning models established under different learning frameworks are very different. For example, a convolutional neural network (CNN) is a machine learning model under deep supervised learning, and can also be called a network structure model based on deep learning, while a deep belief network (Deep Belief Net, DBN) It is a machine learning model under unsupervised learning.

The embodiments of the present application will be described in detail below. It should be understood that the embodiments of the present disclosure may be applied to segmentation of cell nuclei or other instances having a closed structure for images, which are not limited in the embodiments of the present disclosure.

Please refer to FIG. 1 , which is a schematic flowchart of an image processing method disclosed in an embodiment of the present application. As shown in FIG. 1 , the image processing method includes the following steps.

101. Process the first image to obtain prediction results of multiple pixels in the first image, where the prediction results include semantic prediction results and center relative position prediction results, wherein the semantic prediction results indicate that the pixels are located at For instance area or background area, the above-mentioned relative center position prediction result indicates the relative position between the above-mentioned pixel point and the instance center.

Optionally, in 101, by processing the first image, a prediction result of each pixel of a plurality of pixels included in the first image is obtained, wherein the plurality of pixels may be All or part of the pixel points are not limited in this embodiment of the present disclosure. The above-mentioned first image may include pathological images obtained by various image acquisition devices (such as microscopes), such as cell nucleus images, etc. The embodiment of the present disclosure does not limit the acquisition method of the first image and the specific implementation of the example. .

In this embodiment of the present disclosure, the first image may be processed in various ways, for example, the first image may be processed by using an instance segmentation algorithm, or the above-mentioned first image may be input into a neural network for processing, Output the prediction results of the plurality of pixels in the first image, which is not limited in this embodiment of the present disclosure.

In an optional example, the prediction results of multiple pixels in the first image can be obtained through a deep learning-based neural network, such as a deep layer aggregation network (DLANet). The specific implementation of the neural network is not limited. Deep fusion network, also called deep aggregation network, expands the standard architecture through deeper aggregation to better integrate the information of each layer. Deep fusion merges the feature hierarchy in an iterative and hierarchical manner, making the network more High accuracy and fewer parameters. The tree structure is used to replace the linear structure of the previous architecture, and the logarithmic level compression of the gradient return length of the network is realized instead of linear compression, which makes the learned features more descriptive and can effectively improve the prediction accuracy of the above numerical indicators .

In some possible implementation manners, semantic segmentation processing may be performed on the first image to obtain semantic prediction results of multiple pixels in the first image, and based on the semantic prediction results of multiple pixels, Instance segmentation results. The semantic segmentation process is used for grouping/segmenting the pixels in the first image according to different semantic meanings. For example, it can be determined whether each pixel of the plurality of pixels included in the first image is an instance or a background, that is, located in an instance area or a background area.

Pixel-level semantic segmentation can divide each pixel in the image into a corresponding category, that is, to achieve pixel-level classification; and the specific object of the category is an instance. Instance segmentation not only needs to perform pixel-level classification, but also needs to distinguish different instances on the basis of specific categories. For example, there are three cell nuclei 1, 2, and 3 in the first image, and the semantic segmentation results are all cell nuclei, but the instance segmentation results are different objects.

In the embodiment of the present disclosure, for the first image, optionally, each pixel in the first image may be independently instance judged to judge the semantic segmentation category to which it belongs and the instance ID to which it belongs. For example, if there are three nuclei in an image, the semantic segmentation category of each nucleus is 1, but the IDs of different nuclei are 1, 2, and 3, respectively, and different nuclei can be distinguished by the above-mentioned nucleus ID.

In some possible implementations, the semantic prediction result of the pixel point may indicate that the above-mentioned pixel point is located in the instance area or the background area. That is, the semantic prediction result of a pixel indicates that the pixel is an instance or background.

The above instance area can be understood as the area where the instance is located, and the background area is other areas in the image other than the instance. For example, assuming that the first image is a cell image, the semantic prediction result of the pixel point may include indication information for indicating whether the pixel point is a nucleus region or a background region in the cell image. In the embodiment of the present disclosure, whether the pixel point is an instance area or a background area may be indicated in various ways. In some possible implementations, the semantic prediction result of the pixel point may be one of two preset values, wherein the two preset values correspond to the instance area and the background area, respectively. For example, the semantic prediction result of a pixel point may be 0 or a positive integer (eg, 1), where 0 represents a background area, and a positive integer (eg, 1) represents an instance area, but the embodiment of the present disclosure is not limited thereto.

Optionally, the above semantic prediction result may be a binarized result. At this time, the first image can be processed to obtain the instance area prediction probability of each pixel in the plurality of pixel points, wherein the instance area prediction probability indicates the probability that the pixel is located in the instance area, and then, based on the second threshold Binarization is performed on the instance region prediction probability of each pixel point in the plurality of pixel points to obtain a semantic prediction result of each pixel point in the plurality of pixel points.

In one example, the second threshold of the binarization process may be 0.5. In this case, pixels with a predicted probability of the instance area greater than or equal to 0.5 are determined as pixels located in the instance area, and the predicted probability of the instance area is less than 0.5. The pixels of are determined as the pixels located in the background area. Correspondingly, as an example, the semantic prediction result of the pixel point with the prediction probability of the instance region greater than or equal to 0.5 is determined to be 1, and the semantic prediction result of the pixel point of the instance region prediction probability of less than 0.5 is determined to be 0. The example is not limited to this.

In some possible implementations, the prediction result of the pixel point includes the prediction result of the relative position of the center of the pixel point, which is used to indicate the relative position between the above-mentioned pixel point and the center of the instance to which the pixel point belongs. In one example, the prediction result of the relative center position of the pixel point may include the prediction result of the center vector of the pixel point. The difference between the coordinates and the coordinates of the instance center on the horizontal and vertical axes. Optionally, the prediction result of the relative center position of the pixel point may also be achieved in other manners, which is not limited in this embodiment of the present disclosure.

Optionally, based on the prediction result of the relative position of the center of the pixel and the position information of the pixel, determine the predicted position of the instance center of the pixel, that is, the predicted position of the center of the instance to which the pixel belongs, and predict based on the instance center of the pixel. position, to determine the instance to which the pixel point belongs, but this is not limited in this embodiment of the present disclosure.

In an optional example, the position information of the center of at least one instance in the first image may be determined based on the processing of the first image, and the position information of the instance center of the pixel point and the position information of the at least one instance center may be predicted, Determines the instance to which the pixel belongs.

In another example, a small area to which the instance center belongs can be defined as the instance center area, that is, the instance center area is an area within the instance area and smaller than the instance area, and the geometric center of the instance center area is the same as the instance center area. The geometric centers of the instance areas overlap or are adjacent, for example, the center of the instance center area is the instance center. Optionally, the central area of the instance may be a circle, an ellipse, or other shapes, and the central area of the above instance may be set as required, and the embodiment of the present application does not limit the specific implementation of the central area of the instance.

At this time, optionally, at least one instance center area in the first image may be determined, and based on the positional relationship between the predicted position of the instance center of the pixel and the at least one instance center area, the at least one instance center area may be determined from the at least one instance center area. The example to which the pixel point belongs, but the specific implementation thereof is not limited in the embodiments of the present disclosure.

Optionally, the prediction result of the pixel point also includes the prediction result of the center area of the pixel point, indicating whether the pixel point is located in the center area of the instance. At least one instance central region of an image.

In one example, the first image may be processed through a neural network to obtain the prediction result of the central area of each pixel point among the plurality of pixel points included in the first image.

In some possible implementation manners, the above-mentioned neural network may be obtained by training in a supervised training manner. The sample images used in the training process can be marked with instance information, and the central area of the instance can be determined based on the instance information marked by the sample image, and the determined central area of the instance can be used as supervision to train the neural network.

Optionally, the instance center may be determined based on the instance information, and an area including a preset size or area of the instance center may be determined as the center area of the instance. Optionally, the sample image may also be corroded to obtain a corroded sample image, and the central region of the instance may be determined based on the corroded sample image.

The erosion operation of an image means that the image is probed with a certain structuring element in order to find the area inside the image where the structuring element can be placed. The image erosion processing mentioned in the embodiments of the present application may include the above-mentioned erosion operation, and the erosion operation is a process in which structural elements are translated and filled in the etched image. From the results after the erosion, the foreground area of the image is reduced, the boundary of the area becomes blurred, and some relatively small isolated foreground areas are completely eroded, achieving the effect of filtering.

For example, for each instance mask, first use a 5×5 convolution kernel to perform image erosion processing on the instance mask, and then average the coordinates of multiple pixels included in the instance to obtain the center position of the instance , and determine the maximum distance from all the pixels in the instance to the central position of the instance, and determine the pixels whose distance from the central position of the instance is less than 30% of the above-mentioned maximum distance as the pixel in the central area of the instance, That is, the central area of the instance is obtained. In this way, after the instance mask in the sample image is reduced by one circle, the image binarization process is performed to obtain the predicted binary image mask of the central area.

In addition, optionally, the relative position information of the center of the pixel point, that is, the relative position information between the above-mentioned pixel point and the center of the instance, can be obtained based on the coordinates of the pixel point included in the instance marked in the sample image and the center position of the instance. , for example, a vector from a pixel point to an instance center, and the relative position information is used as supervision to train a neural network, but the embodiment of the present disclosure is not limited to this.

In the embodiment of the present disclosure, the prediction result of the center region of each pixel point among the plurality of pixel points included in the first image can be obtained by processing the first image. In some possible implementations, the first image may be processed to obtain the prediction probability of the central area of each pixel in the plurality of pixels of the first image; The center area prediction probability of the point is binarized to obtain the center area prediction result of each pixel point in the above-mentioned multiple pixel points.

The prediction probability of the central area of the pixel point may refer to the probability that the pixel point is located in the central area of the instance.

Optionally, the pixels not located in the central area of the instance may be pixels of the background area or pixels of the instance area.

In this embodiment of the present disclosure, the binarization process may be a binarization process with a fixed threshold or a binarization process with an adaptive threshold. For example, the bimodal method, the P-parameter method, the iterative method, and the OTSU method, etc., the specific implementation of the binarization processing is not limited in the embodiments of the present disclosure.

Optionally, the first threshold or the second threshold in the binarization process may be preset or determined according to an actual situation, which is not limited in this embodiment of the present disclosure.

In some possible implementations, the prediction result of the center area of the pixel point is obtained by judging the magnitude relationship between the prediction probability of the center area of the pixel point and the above-mentioned first threshold. For example, the first threshold may be 0.5. In this case, optionally, the pixels with the prediction probability of the central region greater than or equal to 0.5 are determined as the pixels located in the central region of the instance, and the pixels with the prediction probability of the central region less than 0.5 are determined as Pixels not located in the central area of the instance, so as to obtain the prediction result of the central area of each pixel. For example, the prediction result of the central area of the pixels with the prediction probability of the central area greater than or equal to 0.5 is determined as 1, and the prediction result of the central area of the pixels with the prediction probability of the central area of less than 0.5 is determined as 0, but the embodiment of the present disclosure is not limited to this.

Step 102 may be performed after the above prediction result is obtained.

102. Determine the instance segmentation result of the first image based on the semantic prediction result and the center relative position prediction result of each pixel point in the plurality of pixel points.

In step 101, after the semantic prediction result and the center relative position prediction result are obtained, at least one pixel located in the instance area and relative position information between the at least one pixel and the instance center to which it belongs can be determined.

In some possible implementations, at least one first pixel point located in the instance area may be determined from the plurality of pixel points based on the semantic prediction result of each pixel point in the above-mentioned plurality of pixel points; The relative position prediction result of the center determines the instance to which the first pixel belongs.

At least one first pixel point located in the instance area can be determined according to the semantic prediction result of each pixel point in the plurality of pixel points. Specifically, the semantic prediction result of the plurality of pixel points indicates that the pixel point located in the instance area is determined as the first pixel.

Further, for a pixel located in the instance area (that is, the above-mentioned first pixel), the instance to which the pixel belongs can be determined according to the prediction result of the relative position of the center of the pixel, wherein the instance segmentation result of the first image includes at least The pixels included in each instance in an instance, in other words, include the instance to which each pixel located in the instance area belongs. In some possible implementations, different instances may be distinguished by different instance identifiers or labels (eg, instance IDs), wherein, optionally, the instance ID may be an integer greater than 0, for example, the instance ID of instance a is 1 , the instance ID of instance b is 2, and the instance ID corresponding to the background is 0, then the instance identifier corresponding to each pixel in the multiple pixels of the first image can be obtained, or at least one first image of the first image can be obtained. The instance identifier of each first pixel point in the pixel points, that is, the pixel point located in the background area does not have a corresponding instance identifier, which is not limited in this embodiment of the present disclosure.

For a pixel in cell instance segmentation, if its semantic prediction result is a cell and the center vector representing the prediction result of the relative position of its center points to a certain central area, then this pixel is assigned to the nucleus area of the cell (nuclear semantic area) , and all the pixels are allocated according to the above steps, and the cell segmentation result can be obtained.

Nucleus segmentation in digital microscopy enables extraction of high-quality morphological features of nuclei, as well as computational pathology analysis of nuclei. This information is an important basis for judging, for example, the level of cancer and the effectiveness of drug treatment. In the past, people used Otsu algorithm (Otsu) and waterline (also known as watershed or watershed) threshold algorithm to solve the problem of nucleus instance segmentation, but due to the diversity of nucleus morphology, the above methods are not effective. Instance segmentation can rely on Convolutional Neural Network (CNN), there are two variants of the following algorithms: MaskRCNN (Mask Regions with CNN features) and Fully Convolutional Network (Fully Convolutional Network, FCN) target instance segmentation framework. However, the disadvantage of MaskRCN is that there are many hyperparameters, and it requires personnel to have a high professional cognition for specific problems to obtain better results, and the method runs slowly. FCN requires special image post-processing to separate adhered cells into multiple instances, which also requires high expertise from practitioners.

In the embodiment of the present application, a center vector representing the positional relationship of a pixel with respect to the center of the instance to which it belongs is used for modeling, so that instance segmentation in image processing has the advantages of high speed and high precision. For the cell segmentation problem, the above FCN shrinks some instances into boundary classes, and then uses a targeted post-processing algorithm to trim the prediction of the instance to which the boundary belongs. In contrast, center vector modeling can more accurately predict the boundary state of the nucleus based on the data. , and there is no need for complex professional post-processing algorithms; the above MaskRCNN first cuts out the image of each independent instance through a rectangle, and then performs the second-class prediction of cells and backgrounds, but the cells appear as multiple irregular ovals clustered together. After the rectangle is intercepted, one instance is at the center, and other instances are still partially at the edge, which is not conducive to the next two-class segmentation. In contrast, center vector modeling does not have such problems, and can obtain accurate predictions for the boundary of the cell nucleus, thereby improving the overall prediction accuracy.

The embodiments of the present application can be applied to clinical auxiliary diagnosis. After obtaining the digitally scanned image of the patient's organ tissue slice, the doctor can input the image into the process in the embodiment of the present application to obtain the pixel mask of each independent cell nucleus. The doctor can use this as a basis to calculate the organ. cell density and cell morphological characteristics, and then draw more accurate medical judgments.

In this embodiment of the present application, by processing the first image, the prediction results of multiple pixels in the first image are obtained, and the prediction results include a semantic prediction result and a center relative position prediction result, wherein the semantic prediction result indicates the above The pixel point is located in the instance area or the background area, and the above-mentioned relative position prediction result of the center indicates the relative position between the above-mentioned pixel point and the center of the instance, based on the semantic prediction result and the center relative position prediction result of each pixel point in the above-mentioned multiple pixel points, Determining the instance segmentation result of the first image above can enable instance segmentation in image processing to have the advantages of high speed and high precision.

Please refer to FIG. 2 . FIG. 2 is a schematic flowchart of another image processing method disclosed in an embodiment of the present application, and FIG. 2 is obtained by further optimization on the basis of FIG. 1 . The main body performing the steps of the embodiments of the present application may be the aforementioned electronic device. As shown in Figure 2, the image processing method includes the following steps:

201. Preprocess the second image to obtain a first image, so that the first image satisfies a preset contrast and/or a preset gray value.

The second image mentioned in the embodiment of the present application may be a multimodal pathological image obtained by various image acquisition devices (such as microscopes). In addition, the image size, color, resolution and other characteristics of the images may be different, and the presented image styles may be different, that is, the above-mentioned second image may be one or more images. In the process of making pathological slices and imaging, the pathological image data obtained are usually very different due to differences in tissue types, acquisition methods, imaging equipment and other factors. For example, the resolution of pathological images collected under different microscopes will vary greatly. Color images (lower resolution) of pathological tissue can be acquired with light microscopy, whereas electron microscopes typically only acquire grayscale images (but higher resolution). However, for a clinically available pathology system, it is often necessary to analyze different types of pathological tissues acquired by different imaging devices.

In the data set including the above-mentioned second image, pictures of different patients, different organs, and different staining methods are complex and diverse, so the diversity of the second image can be reduced first through step 201 .

The main body performing the steps of the embodiments of the present application may be the aforementioned electronic device. The above-mentioned preset contrast and/or the above-mentioned preset gray value may be stored in the electronic device, and after the above-mentioned second image can be converted into a first image satisfying the above-mentioned preset contrast and/or the above-mentioned preset gray value, Step 202 is executed again.

The contrast mentioned in the embodiment of this application refers to the measurement of different brightness levels between the brightest white and the darkest black in the light and dark areas of an image. The larger the difference range is, the greater the contrast is, and the smaller the difference range is The smaller the contrast.

Due to the different color and brightness of each point of the scene, each point on the black-and-white photo taken or the black-and-white image reproduced by the TV receiver appears to be gray in different degrees. The relationship between white and black is divided into several levels according to the logarithmic relationship, which is called "gray level". The gray scale generally ranges from 0 to 255, white is 255, and black is 0, so black and white pictures are also called grayscale images, which are widely used in the fields of medicine and image recognition.

Optionally, the preprocessing may further unify image parameters such as image size, image resolution, and image format of the second image. For example, the second image may be cropped to obtain a first image with a preset image size, for example, a first image with a uniform size of 256*256. The electronic device may also store a preset image size and/or a preset image format, and may convert a first image that satisfies the preset image size and/or preset image format during preprocessing.

The electronic device can unify the multimodal pathological images obtained by different pathological tissues and different imaging devices with the help of technologies such as Image Super Resolution (Image Super Resolution) and image conversion, so that they can be used as the images in the embodiments of the present application. Input to the image processing pipeline. This step can also be referred to as the normalization process of the image. Converting to an image of a unified style is more convenient for subsequent unified processing of the image.

Image super-resolution technology refers to the use of image processing methods to convert existing low-resolution (LR) images into high-resolution (HR) images by means of software algorithms (emphasis on unchanged imaging hardware). Image technology can be divided into super-resolution restoration and super-resolution image reconstruction (Super resolution image reconstruction, SRIR). Currently, image super-resolution research can be divided into three main categories: interpolation-based, reconstruction-based, and learning-based methods. The core idea of super-resolution reconstruction is to exchange temporal bandwidth (obtaining multiple frame image sequences of the same scene) for spatial resolution, so as to realize the conversion from temporal resolution to spatial resolution. Through the above preprocessing, a high-resolution first image can be obtained, which is very helpful for doctors to make a correct diagnosis. If high-resolution images can be provided, the performance of pattern recognition in computer vision will be greatly improved. .

202. Process the first image to obtain prediction results of multiple pixels in the first image, where the prediction results include semantic prediction results, center relative position prediction results, and center area prediction results, wherein the semantic prediction results The result indicates that the pixel point is located in the instance area or the background area, the center relative position prediction result indicates the relative position between the pixel point and the instance center, and the center area prediction result indicates whether the pixel point is located in the instance center area.

The above step 202 may refer to the specific description in step 101 of the embodiment shown in FIG. 1 , and details are not repeated here.

203. Based on the semantic prediction result of each pixel point in the plurality of pixel points, determine at least one first pixel point located in the instance region from the plurality of pixel points.

Based on the semantic prediction result of each pixel in the plurality of pixels, it can be determined whether each pixel is located in the instance area or the background area, so that at least one first pixel located in the instance area can be determined from the plurality of pixels point.

The above instance area can be understood as the area where the instance is located, and the background area is other areas in the image other than the instance. For example, assuming that the first image is a cell image, the semantic prediction result of the pixel point may include indication information for indicating whether the pixel point is a nucleus region or a background region in the cell image. In the embodiment of the present disclosure, whether the pixel point is an instance area or a background area may be indicated in various ways. In some possible implementations, the semantic prediction result of the pixel point may be one of two preset values, wherein the two preset values correspond to the instance area and the background area, respectively. For example, the semantic prediction result of a pixel point may be 0 or a positive integer (eg, 1), where 0 represents a background area, and a positive integer (eg, 1) represents an instance area, but the embodiment of the present disclosure is not limited thereto.

204. Determine at least one instance center region of the first image based on the prediction result of the center region of each pixel point in the plurality of pixel points.

Specifically, a small area to which the instance center belongs can be defined as the instance center area, that is, the instance center area is an area within the instance area and smaller than the instance area, and the geometric center of the instance center area is the same as the instance area. The geometric centers overlap or are adjacent, for example, the center of the instance center area is the instance center. Optionally, the central area of the instance may be a circle, an ellipse, or other shapes, and the central area of the above instance may be set as required, and the embodiment of the present application does not limit the specific implementation of the central area of the instance.

The above-mentioned relative position prediction result of the center may indicate the relative position between the above-mentioned pixel point and the center of the instance. In an example, the prediction result of the relative center position of the pixel point may include the prediction result of the center vector of the pixel point. For example, the prediction result of the relative center position of the pixel point is (x, y), which respectively represent the coordinates of the pixel point and the center of the instance. The difference between the coordinates on the horizontal and vertical axes. Optionally, the prediction result of the relative center position of the pixel point may also be achieved in other manners, which is not limited in this embodiment of the present disclosure.

In this embodiment of the present application, the above-mentioned central area prediction result may indicate whether the above-mentioned pixel points are located in the central area of the instance. Therefore, the pixels located in the central area of the instance may be determined by referring to the central area prediction result, and these pixels may form the central area of the instance. Thereby, the above-mentioned at least one example central region can be determined.

Optionally, based on the prediction result of the center area of each pixel point in the plurality of pixel points, a connected domain search process may be performed on the first image to obtain at least one instance center area.

Among them, the connected region (Connected Component) generally refers to the image region (Region, Blob) composed of foreground pixels with the same pixel value and adjacent positions in the image. The above connected domain search can be understood as connected region analysis (Connected Component Analysis, Connected Component Labeling), which is used to find and label each connected region in the image.

Connected region analysis is a commonly used and basic method in many application fields of the International Conference on Computer Vision and Pattern Recognition (CVPR) and image analysis and processing. For example: character segmentation and extraction in Optical Character Recognition (OCR) recognition (license plate recognition, text recognition, subtitle recognition, etc.), segmentation and extraction of moving foreground targets in visual tracking (pedestrian intrusion detection, legacy object detection, based on Visual vehicle detection and tracking, etc.), medical image processing (extraction of target regions of interest), etc. That is to say, the connected area analysis method can be used in application scenarios where foreground objects need to be extracted for subsequent processing. Usually, the object of connected area analysis and processing is a binarized image (binary image ).

The condition for the existence of a path for the set S is that a certain arrangement of the pixel points of the path makes the adjacent pixel points satisfy a certain adjacency relationship. For example, suppose there are A1, A2, A3...An pixels between point p and point q, and the adjacent pixels all satisfy a certain adjacency. Then there is a path between p and q. If the passages are connected end to end, it is called a closed passage. A point p in the set of S has only one path, it is called a connected component, and if S has only one connected component, it is called a connected set.

For R is an image subset, if R is connected, then R is called a region. For all K regions that are not connected, their union Rk constitutes the foreground of the image, and the complement of Rk is called the background.

Based on the prediction result of the center area of each pixel point, the connected domain search process is performed on the first image, and at least one instance center area can be obtained, and then step 205 is performed.

Specifically, for the first image after binarization, a connected domain with a central region of 1 can be found to determine the central region of the instance, and an independent ID is assigned to each connected domain.

For cell segmentation, it can be determined whether the pointing position of the above-mentioned center vector is in the above-mentioned central area based on the coordinates of the pixel in the nucleus and the center vector representing the positional relationship of the pixel with respect to the center of the instance to which it belongs. If the pointing position is in the central area, the ID of the cell nucleus is assigned to the pixel point, otherwise, it means that the pixel point does not belong to any cell nucleus, and it can be assigned nearby.

Optionally, a random walk algorithm may be used to perform a connected domain search process on the first image to obtain at least one instance central region.

Random walk, also known as random walk, random walk, etc., refers to the inability to predict future development steps and directions based on past performance. The core concept of random walk is that the conserved quantity carried by any random walker corresponds to a law of diffusion and transportation, which is close to Brownian motion and is the ideal mathematical state of Brownian motion. The basic idea of the random walk for image processing in this embodiment of the present application is to regard the image as a connected weighted undirected graph composed of fixed vertices and edges, and randomly walk from unmarked vertices. The probability of reaching various marked vertices represents the possibility that the unmarked point belongs to the marked class, and the label of the class with the greatest probability is assigned to the unmarked vertex to complete the segmentation. The above-mentioned random walk algorithm can realize the assignment of pixels that do not belong to any central area, so as to obtain the above-mentioned at least one instance central area.

Optionally, a pixel point connection graph can be output through the above-mentioned deep-level fusion network model, and an instance segmentation result can be obtained after the above-mentioned connected domain search processing. Optionally, random colors can be assigned to each instance region in the above instance segmentation result for easy visualization.

Wherein, the above-mentioned steps 203 and 204 may also be performed in no particular order; after the above-mentioned at least one instance central region is determined, step 205 may be performed.

205 . Based on the prediction result of the relative center position of each first pixel in the at least one first pixel, determine an example center region corresponding to each first pixel from the at least one example center region.

Specifically, the predicted position of the center of the first pixel can be determined based on the position information of the first pixel and the prediction result of the relative position of the center of the first pixel.

In step 202, the position information of the pixel point can be obtained, which can be the coordinates of the pixel point, and the first pixel point is the pixel point located in the example area, and the prediction result of the relative position of the center of the first pixel point indicates the first pixel point. For the relative position between the pixel point and the center of the instance, it can be seen that the predicted center position of the first pixel point can be determined according to the coordinates of the first pixel point and the prediction result of the relative position of the center of the first pixel point. The above-mentioned central predicted position may indicate the central position of the central region of the instance to which the above-mentioned predicted first pixel point belongs.

Further, based on the predicted center position of the first pixel point and the position information of the at least one instance center area, the instance center area corresponding to the first pixel point may be determined from the at least one instance center area.

In step 204, when obtaining the central area of the above instance, the position information of the central area of the above instance can be obtained, which can also be represented by coordinates, and then can be determined based on the predicted position of the center of the first pixel and the position information of the central area of at least one instance. Whether the predicted center position of the first pixel belongs to the at least one instance center area, the instance center area corresponding to the first pixel is determined from the at least one instance center area.

Specifically, the first instance central area may be determined as the instance central area corresponding to the first pixel in response to that the predicted center position of the first pixel belongs to the first instance central area in the at least one instance central area.

If the predicted center position of the above-mentioned first pixel belongs to the first instance central area in the above-mentioned at least one instance central area, the above-mentioned first instance central area is determined as the instance central area corresponding to the above-mentioned first pixel point, and the pixel point can be Assigned to the central region of this instance.

Optionally, in response to that the predicted center position of the first pixel does not belong to any instance central area in the at least one instance central area, the at least one instance central area is compared with the center of the first pixel. The instance central area with the closest predicted position distance is determined as the instance central area corresponding to the first pixel point.

If the predicted center position of the above-mentioned first pixel does not belong to the central area of the first instance in the above-mentioned at least one instance central area, the pixel is not allocated to the above-mentioned first instance central area, but is allocated nearby, and the at least one instance The instance central area in the central area that is closest to the predicted center position of the first pixel is determined as the instance central area corresponding to the first pixel.

The output of the above-mentioned step 202 in this embodiment of the present application may have three branches. The first is the semantic judgment branch, including two channels, to output that each pixel is located in the instance area or the background area; the second is the center area branch, including 2 channels to output each pixel in the central area or non-central area; the third is the center vector branch, including 2 channels, to output the relative position between each pixel and the center of the instance, which can include pixels The vertical and horizontal components of the vector that point to the geometric center of the instance to which it belongs.

In the embodiment of the present application, the instance segmentation object may be a cell nucleus. Since the above-mentioned central area is the central area of a cell nucleus, after the above-mentioned central area is determined, the position of the cell nucleus is actually preliminarily determined, and a numerical number can be assigned to each cell nucleus, that is, the above-mentioned Instance ID.

Specifically, assuming that the input second picture is a 3-channel picture of [height, width, 3], in this embodiment of the present application, in step 202, three arrays of [height, width, 2] may be obtained, which are in turn of each pixel. Semantic prediction probability, center region prediction probability, and center relative position prediction results. Then, the prediction probability of the central region can be binarized with a threshold value of 0.5, and then the central region of each cell nucleus can be obtained through the above-mentioned connected domain search processing, and given an independent digital number, the above-mentioned digital number assigned to each cell That is, the aforementioned instance ID, in order to distinguish different nuclei.

For example, in step 203, it has been determined that the semantic prediction result of a pixel a is the nucleus instead of the background (it is determined that it belongs to the semantic region of the nucleus), and the center vector of the pixel a has been obtained in step 202. The center vector points to the first center area in the center area of at least one instance obtained in step 204, which means that the pixel point a has a corresponding relationship with the first center area, and the specific performance is that the pixel point a belongs to the first center The cell nucleus A where the region is located, and the first central region is the central region of the cell nucleus A.

Taking cell segmentation as an example, through the above steps, the nucleus and the image background can be segmented. Specifically, all the pixels belonging to the nucleus can be allocated, and the nucleus to which each pixel belongs, the nucleus center area to which it belongs, or the nucleus can be determined. to achieve more accurate segmentation of cells and obtain accurate instance segmentation results.

In the embodiment of the present application, the center vector is used for modeling. FCN shrinks some instances into boundary classes, and then uses targeted post-processing algorithms to trim the predictions of the instances to which the boundaries belong. Each pixel has three types of output: background, nucleus interior, and nucleus boundary. It cannot handle the boundary prediction in detail. In this way, the center vector can more accurately predict the boundary state of the nucleus based on the data, and does not require complex professional post-processing algorithms. MaskRCNN first cuts out the image of each independent instance through a rectangle, and then performs the second-class prediction of cells and backgrounds, but the cells appear as multiple irregular ovals that are clustered together. It is still partly on the edge, and it is inevitable that cells containing other instances will be included in the captured image, which is not conducive to the next two-class segmentation. In contrast, the above center vector modeling method will not have such problems. The vectors can lead to accurate predictions for the nucleus boundaries, thereby improving the overall prediction accuracy.

Using the center vector method in the embodiment of the present application, not only the running speed is fast, the processing capacity of 3 images per second can be achieved, but also certain annotation data can be obtained in any instance segmentation problem without the need for high domain knowledge of practitioners. get better results.

The embodiments of the present application can be applied to clinical auxiliary diagnosis. After obtaining the digitally scanned image of the patient's organ tissue slice, the doctor can input the image into the process in the embodiment of the present application to obtain the pixel mask of each independent cell nucleus. The doctor can use this as a basis to calculate the organ. cell density and cell morphological characteristics, and then draw more accurate medical judgments.

In this embodiment of the present application, the first image is obtained by preprocessing the second image, so that the first image satisfies the preset contrast and/or the preset gray value, and the first image is processed to obtain the first image. The prediction results of a plurality of pixel points in the above-mentioned first image, the above-mentioned prediction results include semantic prediction results, center relative position prediction results and central area prediction results, wherein the semantic prediction results indicate that the pixel points are located in the instance area or the background area, The above-mentioned relative position prediction result of the center indicates the relative position between the above-mentioned pixel and the center of the instance, the above-mentioned central area prediction result indicates whether the above-mentioned pixel is located in the central area of the instance, and then based on the semantic prediction result of each pixel in the above-mentioned multiple pixels , determine at least one first pixel point located in the example area from the above-mentioned multiple pixel points, and determine at least one example center area of the above-mentioned first image based on the prediction result of the center area of each pixel point in the above-mentioned multiple pixel points , based on the prediction result of the relative position of the center of each first pixel point in the above at least one first pixel point, and from the above at least one instance center area, determine the instance center area corresponding to each first pixel point above, so as to realize the accuracy of the instance Segmentation can make instance segmentation in image processing have the advantages of high speed and high precision.

Please refer to FIG. 3 . FIG. 3 is a schematic diagram of a cell instance segmentation result disclosed in the embodiment of the present application. As shown in the figure, taking the cell instance segmentation as an example, the method in the embodiment of the present application is used for processing, which has high speed and precision at the same time. high characteristic. In conjunction with FIG. 3 , it may facilitate a clearer understanding of the methods in the embodiments described in FIGS. 1 and 2 . More accurate prediction indicators can be obtained through the deep-level fusion network model, and the existing data sets are used to mark them. The semantic prediction results, the center area prediction results and the center relative position prediction results in the foregoing embodiment are shown in Figure 3 It includes semantic annotation, center annotation and center vector annotation of pixel point A, pixel point B, pixel point C and pixel point D, respectively. As shown in the figure, a cell nucleus can include the nucleus semantic area and the nucleus center area. For the pixel in the figure, if the semantic label of the pixel is 1, it means that the pixel belongs to the nucleus, and if it is 0, it is the image background; If the center is marked as 1, it means that the pixel is the center of the nucleus area. At this time, the center vector of the pixel is marked as (0, 0), which can be used as a reference for other pixels (such as pixel A and pixel D in the figure). , the determination of pixel A can also represent the determination of a cell nucleus). Each pixel corresponds to a coordinate, and the center vector label is the coordinate of the pixel relative to the pixel at the center of the nucleus. For example, the center vector of pixel B relative to pixel A is labeled (-5, -5), and The center vector label of the center pixel is (0, 0), such as pixel A and pixel D. In the embodiment of the present application, it can be determined that the pixel point B belongs to the central area of the cell nucleus to which the pixel point A belongs, that is, the pixel point B is allocated to the nucleus area to which the pixel point A belongs, but not in the central area but in the nucleus semantics of the above-mentioned cell. In the region, the entire segmentation process is similarly completed to obtain accurate cell instance segmentation results.

The foregoing mainly introduces the solutions of the embodiments of the present application from the perspective of the method-side execution process. It can be understood that, in order to realize the above-mentioned functions, the electronic device includes corresponding hardware structures and/or software modules for executing each function. Those skilled in the art should easily realize that the present application can be implemented in the form of hardware or a combination of hardware and computer software, in conjunction with the units and algorithm steps of each example described in the embodiments disclosed herein. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for a particular application, but such implementations should not be considered beyond the scope of this application.

In this embodiment of the present application, the electronic device may be divided into functional units according to the foregoing method examples. For example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software functional units. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical function division, and other division methods may be used in actual implementation.

Please refer to FIG. 4 , which is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. As shown in FIG. 4 , the electronic device 400 includes: a prediction module 410 and a segmentation module 420, wherein: the prediction module 410 is configured to process the first image to obtain a plurality of pixel points in the first image The prediction result includes a semantic prediction result and a center relative position prediction result, wherein the semantic prediction result indicates that the pixel point is located in the instance area or the background area, and the center relative position prediction result indicates that the pixel point is located The relative position to the center of the instance.

The segmentation module 420 is configured to determine an instance segmentation result of the first image based on the semantic prediction result and the center relative position prediction result of each pixel point in the plurality of pixel points.

Optionally, the electronic device 400 further includes a preprocessing module 430 for preprocessing the second image to obtain the first image, so that the first image satisfies a preset contrast and/or Default grayscale value.

Optionally, the segmentation module 420 includes a first unit 421 and a second unit 422, wherein: the first unit 421 is configured to, based on the semantic prediction result of each pixel point in the plurality of pixel points, Determining at least one first pixel point located in the instance area among the plurality of pixel points; the second unit 422 is configured to predict the result based on the relative position of the center of each first pixel point in the at least one first pixel point, An instance to which each of the first pixel points belongs is determined.

Optionally, the prediction result further includes: a center area prediction result, where the center area prediction result indicates whether the pixel is located in the instance center area, and the segmentation module 420 further includes a third unit 423, for The center area prediction result of each pixel point in the plurality of pixel points, to determine at least one instance center area of the first image; the second unit 422 is specifically configured to, based on each of the at least one first pixel point, The relative position prediction result of the centers of the first pixel points, and the instance center area corresponding to each first pixel point is determined from the at least one instance center area.

Optionally, the third unit 423 is specifically configured to perform a connected domain search process on the first image based on the prediction result of the center area of each pixel in the plurality of pixels to obtain at least one instance center. area.

Optionally, the second unit 422 is specifically configured to: determine the center prediction position of the first pixel point based on the position information of the first pixel point and the prediction result of the relative center position of the first pixel point; An example center region corresponding to the first pixel point is determined from the at least one example center region based on the predicted center position of the first pixel point and the position information of the at least one example center region.

Optionally, the second unit 422 is specifically configured to: in response to that the predicted center position of the first pixel belongs to the first instance center area in the at least one instance center area, to assign the first instance center area to the first instance center area. It is determined as the instance central area corresponding to the first pixel point.

Optionally, the second unit 422 is specifically configured to: in response to that the predicted center position of the first pixel point does not belong to any instance center area in the at least one instance center area, place the at least one instance center area into the at least one instance center area. The instance central region in the 100 that is closest to the predicted center position of the first pixel is determined as the instance central region corresponding to the first pixel.

Optionally, the prediction module 410 includes a probability prediction unit 411 and a determination unit 412, wherein: the probability prediction unit 411 is configured to process the first image to obtain multiple The prediction probability of the central area of the pixel point; the judgment unit 412 is configured to perform binarization processing on the prediction probability of the central area of the plurality of pixel points based on the first threshold, to obtain each pixel point in the plurality of pixel points The prediction result of the central area.

Optionally, the prediction module 410 is specifically configured to input the first image into a neural network for processing, and output prediction results of multiple pixels in the first image.

In the embodiment of the present application, the center vector is used for modeling. FCN shrinks some instances into boundary classes, and then uses targeted post-processing algorithms to trim the predictions of the instances to which the boundaries belong. Each pixel has three types of output: background, nucleus interior, and nucleus boundary. It cannot handle the boundary prediction in detail. In this way, the center vector can more accurately predict the boundary state of the nucleus based on the data, and does not require complex professional post-processing algorithms. MaskRCNN first cuts out the image of each independent instance through a rectangle, and then performs the second-class prediction of cells and backgrounds, but the cells appear as multiple irregular ovals that are clustered together. It is still partially on the edge, and it is unavoidable that cells of other instances will be included in the captured image, which is not conducive to the next two-class segmentation. In contrast, executing the above center vector modeling method through the electronic device 400 will not have this effect. For class problems, the center vector can make accurate predictions for the nucleus boundaries, thereby improving the overall prediction accuracy.

Using the electronic device 400 in the embodiment of the present application, the image processing method in the above-mentioned embodiments of FIG. 1 and FIG. 2 can be implemented, and instance segmentation is performed by the center vector method, which not only runs fast, but also can achieve a processing capacity of 3 images per second. , and without the need of high domain knowledge of practitioners, it is possible to obtain certain labeled data in any instance segmentation problem and post-processing to achieve better results.

Implementing the electronic device 400 shown in FIG. 4 , the electronic device 400 can process the first image to obtain prediction results of multiple pixels in the first image, where the prediction results include semantic prediction results and center relative position prediction. As a result, the semantic prediction result indicates that the pixel is located in the instance area or the background area, and the center relative position prediction result indicates the relative position between the pixel and the instance center, based on the The semantic prediction result and the center relative position prediction result determine the instance segmentation result of the first image, so that instance segmentation in image processing has the advantages of high speed and high precision.

Please refer to FIG. 5 , which is a schematic structural diagram of another electronic device disclosed in an embodiment of the present application. As shown in FIG. 5 , the electronic device 500 includes a processor 501 and a memory 502 , wherein the electronic device 500 may further include a bus bar 503 , the processor 501 and the memory 502 can be connected to each other through the bus bar 503 , and the bus bar 503 can It is a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus bar or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus bar and the like. The bus 503 can be classified into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in FIG. 5, but it does not mean that there is only one busbar or one type of busbar. The electronic device 500 may further include an input/output device 504, and the input/output device 504 may include a display screen, such as a liquid crystal display screen. The memory 502 is used to store the computer program; the processor 501 is used to call the computer program stored in the memory 502 to execute some or all of the method steps mentioned in the above embodiments of FIG. 1 and FIG. 2 .

Implementing the electronic device 500 shown in FIG. 5 , the electronic device 500 can process the first image to obtain prediction results of multiple pixels in the first image, where the prediction results include semantic prediction results and center relative position prediction. As a result, the semantic prediction result indicates that the pixel is located in the instance area or the background area, and the center relative position prediction result indicates the relative position between the pixel and the instance center, based on the The semantic prediction result and the center relative position prediction result determine the instance segmentation result of the first image, so that instance segmentation in image processing has the advantages of high speed and high precision.

Embodiments of the present application further provide a computer storage medium, wherein the computer storage medium is used to store a computer program, and the computer program enables the computer to execute part or all of the steps of any image processing method described in the above method embodiments.

It should be noted that, for the sake of simple description, the foregoing method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence. Because in accordance with the present application, certain steps may be performed in other orders or concurrently. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated. to another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical or other forms.

The units (modules) described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple networks. on the road unit. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art, or all or part of the technical solution, and the computer software product is stored in a memory, Several instructions are included to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), mobile hard disk, magnetic disk or CD, etc. medium.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: Pen drive, read-only memory, random access device, magnetic disk or CD, etc.

The embodiments of the present application have been introduced in detail above, and the principles and implementations of the present application are described in this paper by using specific examples. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application; at the same time, for Persons of ordinary skill in the art, based on the idea of the present application, will have changes in the specific implementation manner and application scope. In summary, the contents of this specification should not be construed as limitations on the present application.

101, 102, 201, 202, 203, 204, 205: Steps 400, 500: Electronic equipment 410: Prediction Module 411: Probabilistic prediction unit 412: Judgment unit 420: Split Module 421: Unit 1 422: Unit Two 423: Unit 3 430: preprocessing module 501: Processor 502: memory 503: Busbar 504: I/O device

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the accompanying drawings required in the embodiments will be briefly introduced below. FIG. 1 is a schematic flowchart of an image processing method disclosed in an embodiment of the present application. FIG. 2 is a schematic flowchart of another image processing method disclosed in an embodiment of the present application. FIG. 3 is a schematic diagram of a cell instance segmentation result disclosed in an embodiment of the present application. FIG. 4 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. FIG. 5 is a schematic structural diagram of another electronic device disclosed in an embodiment of the present application.

101, 102: Steps

Claims (9)

An image processing method, comprising: inputting a first image into a deep learning neural network for processing, and outputting respective prediction results of multiple pixels in the first image, wherein the prediction results include semantic prediction results, The center relative position prediction result and the center area prediction result, wherein the semantic prediction result indicates that the pixel point is located in the instance area or the background area, and the center relative position prediction result indicates the relative position between the pixel point and the instance center , the central area prediction result indicates whether the pixel point is located in the instance central area; based on the semantic prediction result of each pixel point in the plurality of pixel points, at least one pixel located in the instance area is determined from the plurality of pixel points A first pixel point; for each of the first pixel points, determine the instance to which each first pixel point belongs based on the prediction result of the relative position of the center of the first pixel point and the prediction result of the center area. The image processing method according to item 1 of the scope of the application, before processing the first image, further comprising: preprocessing the second image to obtain the first image, so that the first image is An image satisfies a preset contrast ratio and/or a preset grayscale value. According to the image processing method described in item 1 of the scope of the application, determining the instance to which the first pixel belongs based on the prediction result of the relative position of the center of the first pixel and the prediction result of the center area, comprising: based on the The prediction result of the central area of each pixel in multiple pixels, determining at least one instance central area of the first image; and determining an instance central area corresponding to the first pixel point from the at least one instance central area based on the prediction result of the relative center position of the first pixel point. According to the image processing method according to item 3 of the scope of the application, determining at least one instance center region of the first image based on the prediction result of the center region of each pixel point in the plurality of pixel points, comprising: based on For the prediction result of the central area of each pixel point in the plurality of pixel points, a connected domain search process is performed on the first image to obtain the at least one instance central area. According to the image processing method according to claim 3 or 4, based on the prediction result of the relative position of the center of the first pixel point, it is determined from the at least one instance center area that each first pixel point corresponds to The example center area of the The center position of the instance central area to which the first pixel point belongs; based on the predicted center position of the first pixel point and the position information of the at least one instance central area, determine from the at least one instance central area The example center area corresponding to the first pixel point. The image processing method according to item 5 of the scope of the patent application, based on the predicted position of the center of the first pixel point and the position of the center region of the at least one instance and determining the instance central area corresponding to the first pixel point from the at least one instance central area, including: responding to the center prediction position of the first pixel point belonging to the first pixel in the at least one instance central area An instance central area, the first instance central area is determined as the instance central area corresponding to the first pixel point; or in response to the center prediction position of the first pixel point not belonging to the at least one instance central area In any instance central area of the at least one instance central area, the instance central area that is closest to the predicted center position of the first pixel point in the at least one instance central area is determined as the instance central area corresponding to the first pixel point. An electronic device, comprising: a prediction module, which inputs a first image into a deep learning neural network for processing, and outputs prediction results of a plurality of pixels in the first image, where the prediction results include semantic prediction results, The center relative position prediction result and the center area prediction result, wherein the semantic prediction result indicates that the pixel point is located in the instance area or the background area, and the center relative position prediction result indicates the relative position between the pixel point and the instance center , the central area prediction result indicates whether the pixel is located in the central area of the instance; the segmentation module is configured to determine, from the plurality of pixel points, based on the semantic prediction result of each pixel point in the plurality of pixel points At least one first pixel point of the instance area; for each of the first pixel points, based on the prediction result of the relative position of the center of the first pixel point and the prediction result of the center area, determine which each first pixel point belongs to. instance. An electronic device, comprising a processor and a memory, the memory is used for storing a computer program, the computer program is configured to be executed by the processor, and the processor is used for executing the invention as described in items 1 to 6 of the patent application scope The image processing method described in any one of. A computer-readable storage medium is used for storing a computer program, wherein the computer program enables a computer to execute the image processing method described in any one of items 1 to 6 of the patent application scope.

Images