TWI786330B - Image processing method, electronic device, and storage medium - Google Patents

Abstract

The embodiment of the present application discloses an image processing method, an electronic device, and a storage medium, wherein the method includes: acquiring N sets of segmentation output data, wherein the N sets of segmentation output data are respectively performed by N instance segmentation models Processing the obtained instance segmentation output result, and the N sets of instance segmentation output data have different data structures, the N being an integer greater than 1; and segmenting the output data based on the N sets of instances to obtain integrated semantic data of the image And integrating central region data, wherein the integrated semantic data indicates pixel points in the image region of the image, the integrated central region data indicating pixel points in the image center region of the image; integration based on the image Semantic data and integration of central region data, to obtain the example segmentation results of the image, can achieve the complementary advantages of each instance segmentation model, and achieve higher precision in the instance segmentation problem.

Description

An image processing method, electronic device and storage medium

The present application relates to the technical field of computer vision, in particular to an image processing method, electronic equipment and a storage medium.

Image processing, also known as image processing, is a technology that uses computers to analyze images to achieve the desired results. Image processing generally refers to digital image processing. Digital image refers to a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of this array are called pixels, and their values are called gray degree value. Image processing plays a very important role in many fields.

Embodiments of the present application provide an image processing method, electronic equipment, and a storage medium.

The first aspect of the embodiments of the present application provides an image processing method, including: acquiring N sets of instance segmentation output data, wherein the N sets of instance segmentation output data are the instance segmentation obtained by processing the image with N instance segmentation models Output the result, and the N sets of instance segmentation output data have different data structures, and the N is an integer greater than 1; based on the N sets of instance segmentation output data, the integrated semantic data and the integrated central area of the image are obtained data, wherein the integrated semantic data indicates the pixel points located in the instance area in the image, and the integrated central area data indicates the pixel points located in the instance central area in the image; based on the integrated semantics of the image Data and integration center region data to obtain the instance segmentation result of the image.

In an optional implementation manner, said segmenting the output data based on the N groups of instances to obtain the integrated semantic data and integrated central area data of the image includes: segmenting each instance in the model for the N instances Segmentation model, based on the instance segmentation output data of the instance segmentation model, the semantic data and central area data of the instance segmentation model are obtained; based on the semantic data and central area data of each instance segmentation model in the N instance segmentation models , to obtain the integrated semantic data and integrated central area data of the image.

In an optional implementation manner, the instance segmentation output data based on the instance segmentation model, obtaining the semantic data and central region data of the instance segmentation model includes: the instance segmentation output data based on the instance segmentation model , determine the instance identification information corresponding to each pixel in the plurality of pixels of the image in the instance segmentation model; based on the instance corresponding to each pixel in the plurality of pixels in the instance segmentation model identification information to obtain the semantic prediction value of each pixel in the instance segmentation model, wherein the semantic data of the instance segmentation model includes the semantic prediction of each pixel in the plurality of pixels of the image value.

In an optional implementation manner, the instance segmentation output data based on the instance segmentation model is used to obtain the semantic data and central region data of the instance segmentation model, and further includes: the instance segmentation output based on the instance segmentation model data, determining in the instance segmentation model, at least two pixels located in the instance region in the image; based on the position information of at least two pixels located in the instance region in the instance segmentation model, determining the instance The instance center position of the segmentation model; based on the instance center position of the instance segmentation model and the position information of the at least two pixel points, determine the instance center area of the instance segmentation model.

In an optional implementation manner, after determining, based on the instance segmentation output data of the instance segmentation model, that in the instance segmentation model, before at least two pixel points located in the instance region in the image, further includes: Erosion processing is performed on the instance segmentation output data of the instance segmentation model to obtain erosion data of the instance segmentation model. In this case, the determining, in the instance segmentation model based on the instance segmentation output data of the instance segmentation model, at least two pixels located in the instance region in the image, includes: based on the instance segmentation Corrosion data of the model, determining at least two pixel points located in the instance region in the image in the instance segmentation model.

In an optional implementation manner, the determining the instance center position of the instance segmentation model based on the position information of at least two pixel points located in the instance area in the instance segmentation model includes: The average value of the positions of at least two pixel points is used as the instance center position of the instance segmentation model.

In an optional implementation manner, the determining the instance center area of the instance segmentation model based on the instance center position of the instance segmentation model and the position information of the at least two pixels includes: based on the instance Segmenting the instance center position of the model and the position information of the at least two pixel points, determining the maximum distance between the at least two pixel points and the instance center position; based on the maximum distance, determining a first threshold; Among the at least two pixel points, a pixel point whose distance from the central position of the instance is less than or equal to the first threshold is determined as a pixel point in the central region of the instance.

In an optional implementation manner, the integrated semantic data and integrated central area data of the image are obtained based on the semantic data and central area data of each instance segmentation model in the N instance segmentation models, including: Based on the semantic data of each instance segmentation model in the N instance segmentation models, determine the semantic voting value of each pixel in the plurality of pixels of the image; for each pixel in the plurality of pixels The semantic voting value of the image is binarized to obtain the integrated semantic value of each pixel in the image, wherein the integrated semantic data of the image includes the integrated semantic value of each pixel in the plurality of pixels value.

In an optional implementation manner, performing binarization processing on the semantic voting value of each pixel in the plurality of pixels to obtain an integrated semantic value of each pixel in the image includes: Based on the number N of the plurality of instance segmentation models, determine a second threshold; based on the second threshold, perform binarization processing on the semantic voting value of each pixel in the plurality of pixels, to obtain the The integrated semantic value of each pixel in the image.

In an optional implementation manner, the second threshold is a rounded-up result of N/2.

In an optional implementation manner, the obtaining the instance segmentation result of the image based on the integrated semantic data and the integrated central area data of the image includes: obtaining the instance segmentation result based on the integrated central area data of the image At least one instance central area of the image; based on the at least one instance central area and the integrated semantic data of the image, determine the instance to which each pixel of the multiple pixels of the image belongs.

In an optional implementation manner, the determining the instance to which each pixel of the plurality of pixels of the image belongs based on the integrated semantic data of the at least one instance central area and the image includes: Based on the integrated semantic value of each pixel in the multiple pixels of the image and the central area of the at least one instance, random walk is performed to obtain the instance to which each pixel belongs.

The second aspect of the embodiment of the present application provides an electronic device, including an acquisition module, a conversion module, and a segmentation module, wherein: the acquisition module is used to acquire N sets of instance segmentation output data, wherein the N sets of instance segmentation output data are respectively Instance segmentation output results obtained by processing images for N instance segmentation models, and the N sets of instance segmentation output data have different data structures, and the N is an integer greater than 1; the conversion module is used for based on The N sets of instance segmentation output data to obtain the integrated semantic data and integrated central area data of the image, wherein the integrated semantic data indicates the pixels located in the instance area in the image, and the integrated central area data Indicating the pixels located in the central area of the instance in the image; the segmentation module is configured to obtain an instance segmentation result of the image based on the integrated semantic data and the integrated central area data of the image.

In an optional implementation manner, the conversion module includes a first conversion unit and a second conversion unit, wherein: the first conversion unit is configured to segment a model for each of the N instance segmentation models, Based on the instance segmentation output data of the instance segmentation model, the semantic data and central region data of the instance segmentation model are obtained; the second conversion unit is configured to obtain the instance segmentation model based on each of the N instance segmentation models Semantic data and central area data, the integrated semantic data and integrated central area data of the image are obtained.

In an optional implementation manner, the first conversion unit is specifically configured to: determine, based on the instance segmentation output data of the instance segmentation model, that each of the multiple pixel points of the image in the instance segmentation model Instance identification information corresponding to pixels; based on the instance identification information corresponding to each pixel in the plurality of pixels in the instance segmentation model, the semantic prediction of each pixel in the instance segmentation model is obtained value, wherein the semantic data of the instance segmentation model includes the semantic prediction value of each pixel in the plurality of pixels of the image.

In an optional implementation manner, the first conversion unit is specifically further configured to: determine, in the instance segmentation model, based on the instance segmentation output data of the instance segmentation model, the At least two pixel points; based on the position information of at least two pixel points located in the instance area in the instance segmentation model, determine the instance center position of the instance segmentation model; based on the instance center position of the instance segmentation model and the The position information of at least two pixel points determines the instance central region of the instance segmentation model.

In an optional implementation manner, the conversion module further includes an erosion processing unit, configured to perform erosion processing on the instance segmentation output data of the instance segmentation model to obtain erosion data of the instance segmentation model; the first conversion unit Specifically, based on the erosion data of the instance segmentation model, determine at least two pixel points located in the instance region in the image in the instance segmentation model.

In an optional implementation manner, the first conversion unit is specifically configured to use the average value of the positions of at least two pixel points located in the instance region as the instance center position of the instance segmentation model.

In an optional implementation manner, the first conversion unit is further configured to: determine the at least two pixel points based on the instance center position of the instance segmentation model and the position information of the at least two pixel points The maximum distance from the central position of the instance; based on the maximum distance, determine a first threshold; divide the at least two pixel points from the pixels whose distance from the central position of the instance is less than or equal to the first threshold The point is determined as the pixel point in the central area of the instance.

In an optional implementation manner, the conversion module is specifically configured to: determine, based on the semantic data of the instance segmentation model, the semantic voting value of each pixel in the multiple pixels of the image; The semantic voting value of each pixel in the plurality of pixels is binarized to obtain the integrated semantic value of each pixel in the image, wherein the integrated semantic data of the image includes the multiple The integrated semantic value of each pixel in the pixel.

In an optional implementation manner, the conversion module is further configured to: determine a second threshold based on the number N of the plurality of instance segmentation models; The semantic voting value of each pixel in the point is binarized to obtain the integrated semantic value of each pixel in the image.

In an optional implementation manner, the second threshold is a rounded-up result of N/2.

The third aspect of the embodiment of the present application provides another electronic device, including a processor and a memory, the memory is used to store a computer program, the computer program is configured to be executed by the processor, and the processor is used to execute Part or all of the steps described in any method of the first aspect of the embodiment of the present application.

The fourth aspect of the embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium is used to store a computer program, wherein the computer program enables the computer to execute any method described in the third aspect of the embodiment of the present application. Some or all of the steps described.

The embodiment of the present application obtains N sets of instance segmentation output data, wherein the above N sets of instance segmentation output data are the instance segmentation output results obtained by processing images with N instance segmentation models, and the above N sets of instance segmentation output data have different The above N is an integer greater than 1, and then the output data is segmented based on the above N groups of instances to obtain the integrated semantic data and integrated central area data of the above image, wherein the above integrated semantic data indicates that the image is located in the instance area The pixel points of the above-mentioned integrated center area data indicate the pixels located in the center area of the instance in the above-mentioned image, and then based on the integrated semantic data and integrated central area data of the above-mentioned image, the instance segmentation result of the above-mentioned image is obtained, which can be obtained in the image In the instance segmentation problem to be dealt with, the complementary advantages of each instance segmentation model are realized, and the data output of each model is no longer required to have the same structure or meaning, and higher accuracy is achieved in the instance segmentation problem.

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

The terms "first", "second" and the like in the specification and patent scope of the present application and the above drawings are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "include" and "have", as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes unlisted steps or units, or optionally further includes For other steps or units inherent in these processes, methods, products or devices.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The occurrences of this phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is understood explicitly and implicitly by those skilled in the art that the embodiments described herein can be combined with other embodiments.

The electronic device involved in the embodiment of the present application may allow multiple other terminal devices to access. Such electronic devices include terminal devices, and in particular implementations, such terminal devices include, but are not limited to, other portable equipment. It should also be understood that, in some embodiments, the terminal device is not a portable communication device, but a desktop computer with a touch-sensitive surface (eg, a touch screen display and/or a touchpad).

Deep learning is a method based on representation learning of data in machine learning. Observations (such as an image) can be represented in a variety of ways, such as a vector of intensity values for each pixel, or more abstractly represented as a series of edges, regions of a specific shape, etc. Instead, it is easier to learn tasks from examples (e.g., face recognition or facial expression recognition) using certain representations. The advantage of deep learning is to use unsupervised or semi-supervised feature learning and hierarchical feature extraction efficient algorithms to replace manual feature acquisition. Deep learning is a new field in machine learning research. Its motivation is to establish and simulate the neural network of human brain for analysis and learning, so that it can imitate the mechanism of human brain to explain data, such as images, sounds and texts.

Like machine learning methods, deep machine learning methods can also be divided into supervised learning and unsupervised learning. The learning models established under different learning frameworks are very different. For example, Convolutional neural network (CNN) is a machine learning model under deep supervised learning, which can also be called a network structure model based on deep learning, while Deep Belief Net (DBN) It is a machine learning model under unsupervised learning.

The following describes the embodiments of the present application in detail. It should be understood that the embodiments of the present disclosure may be applied to performing cell nucleus segmentation or other types of instance segmentation on an image, for example, any instance segmentation with a closed structure, which is not limited in the embodiments of the present disclosure.

Please refer to FIG. 1. FIG. 1 is a schematic flow diagram of an image processing method disclosed in the embodiment of the present application. The method can be executed by any electronic device, such as a terminal device, server or processing platform, etc., which is not covered in the embodiment of the present disclosure. limited. As shown in Figure 1, the image processing includes the following steps.

101. Obtain N sets of instance segmentation output data, wherein the above N sets of instance segmentation output data are the instance segmentation output results obtained by processing images with N instance segmentation models, and the above N sets of instance segmentation output data have different data structure, the above N is an integer greater than 1.

First of all, the instance segmentation problem in image processing is defined as: For an input image, it is necessary to independently judge each pixel, and judge its semantic category and instance ID. For example, there are three nuclei 1, 2, and 3 in the image, and their semantic categories are all nuclei, but the instance segmentation results are different objects.

In some possible implementations, instance segmentation can rely on Convolutional Neural Network (CNN), which mainly has the following two variants of the algorithm: called MaskRCNN (Mask Regions with CNN features) and simple combing of the full volume A target instance segmentation framework for Fully Convolutional Network (FCN). The disadvantage of MaskRCN is that there are many parameters. For specific problems, practitioners are required to have high professional knowledge to get better results, and the method runs slowly; FCN needs special image post-processing to separate the glued same-semantic multi-objects into Multiple instances, which also require a high level of expertise from practitioners.

In another possible implementation manner, the instance segmentation can also be implemented by means of an instance segmentation algorithm, such as a machine learning model such as an instance segmentation algorithm based on a support vector machine. Embodiments of the present disclosure do not limit the specific implementation of the instance segmentation model.

Different instance segmentation models have their own advantages and disadvantages, and embodiments of the present disclosure integrate advantages of different single models by integrating multiple instance segmentation models.

In the embodiment of the present application, optionally, before step 101 is performed, different instance segmentation models may be used to process the images respectively, for example, MaskRCNN and FCN are used to process the images respectively to obtain an output result of instance segmentation. Assuming that there are N instance segmentation models, the instance segmentation results of the instance segmentation models (hereinafter referred to as instance segmentation output data) can be obtained, that is, N sets of instance segmentation output data can be obtained. Alternatively, the N sets of instance segmentation output data may be acquired from other devices, and the embodiment of the present disclosure does not limit the manner of acquiring the N sets of instance segmentation output data.

Optionally, before processing the image using the instance segmentation model, it is also possible to preprocess the image, such as contrast and/or grayscale adjustment, or cropping, horizontal and vertical flipping, rotation, scaling, noise removal, etc. or any number of operations, so that the preprocessed image meets the requirements of the instance segmentation model for the input image, which is not limited in this embodiment of the present disclosure.

In the embodiments of the present disclosure, the instance segmentation output data output by the N instance segmentation models may have different data structures or meanings. For example, for an image input whose dimension is [height, width, 3], the instance segmentation output data includes the data of [height, width], where the instance ID is 0 to represent the background, and different numbers greater than 0 represent different instance of . Suppose there are 3 instance segmentation models, and different instance segmentation models correspond to different algorithms or neural network structures. Among them, the instance segmentation output data of the first instance segmentation model is a three-class probability map of [boundary, target, background]; The instance segmentation output data of the second instance segmentation model is the binary classification probability map of [boundary, background] and the binary classification map with the dimension of [target, background]; the instance segmentation output data of the third instance segmentation model is [central area , target overall, background] three-category probability map, and so on. Different instance segmentation models have different meaningful data outputs. At this point, it is not possible to use arbitrary weighted averaging algorithms to integrate the outputs of individual instance segmentation models to achieve more stable and high-precision results. The method in the embodiment of the present application can perform cross-instance segmentation model integration on the basis of the N sets of instance segmentation output data with different data structures.

Step 102 may be executed after the above N groups of instance segmentation output data are obtained.

102. Based on the above-mentioned N groups of instance segmentation output data, the integrated semantic data and integrated central area data of the above-mentioned image are obtained, wherein the above-mentioned integrated semantic data indicates the pixels located in the instance area in the above-mentioned image, and the above-mentioned integrated central area data indicates the above-mentioned Pixels in the center of the instance in the image.

Specifically, the electronic device may convert the above N groups of instance segmentation output data to obtain integrated semantic data and integrated central area data of the image.

The semantic segmentation mentioned in the embodiment of this application is a basic task in computer vision. In semantic segmentation, we need to divide the visual input into different semantically interpretable categories, that is, the classification categories are meaningful in the real world. An image is composed of many pixels (Pixel), and semantic segmentation, as the name implies, is to group (Grouping)/segment (Segmentation) pixels according to the different semantic meanings expressed in the image. For example, we might need to distinguish all pixels in an image that belong to cars and color those pixels blue.

Pixel-level semantic segmentation can divide each pixel in the image into a corresponding category, that is, realize pixel-level classification; and the specific object of the class is an instance, so in fact, instance segmentation not only needs to perform pixel-level Classification also needs to distinguish different instances on the basis of specific categories. For example, there are three people A, B, and C in the image, and the semantic segmentation results are all people, but the instance segmentation results are different objects.

The above-mentioned instance area can be understood as the area where the instance in the image is located, that is, the area except the background area, and the above-mentioned integrated semantic data can indicate the pixels located in the instance area in the above-mentioned image, such as for the processing of cell nucleus segmentation, the above-mentioned integrated The semantic data may include judgment results of pixels located in the nucleus area.

The above-mentioned integrated central area data may indicate the pixel points located in the central area of the instance in the above-mentioned image.

A small area to which the instance center belongs can be defined as the instance center area, that is, the instance center area is an area within the instance area and smaller than the instance area, and the geometric center of the instance center area overlaps with the geometric center of the instance area or nearby, for example, the center of the instance center region is the instance center. Optionally, the central area of the example may be a circle, an ellipse or other shapes, and the central area of the above example may be set as required, and the embodiment of the present application does not limit the specific realization of the central area of the example.

Specifically, based on the instance segmentation output data of each instance segmentation model in the above N instance segmentation models, the semantic data and central area data of each of the above instance segmentation models can be obtained, that is, a total of N sets of semantic data and N sets of central areas data, and then perform integrated processing based on the semantic data and central area data of each instance segmentation model in the above N instance segmentation models, to obtain the integrated semantic data and integrated central area data of the above image.

For the instance segmentation output data of each instance segmentation model in each instance segmentation model of N instance segmentation models, the instance identification information (instance ID) corresponding to each pixel in the instance segmentation model can be determined, and then based on the above The instance identification information corresponding to each of the plurality of pixels in the instance segmentation model is used to obtain the semantic prediction value of each pixel in the above instance segmentation model. Wherein, the semantic data of the above-mentioned instance segmentation model includes a semantic prediction value of each pixel point among the multiple pixel points of the above-mentioned image.

The binarization (Thresholding) mentioned in the embodiment of this application is a simple method of image segmentation. Binarization can convert a grayscale image into a binary image. The grayscale of pixels greater than a certain critical grayscale value can be set as the maximum grayscale value, and the grayscale of pixels smaller than this value can be set as grayscale. minimum value to achieve binarization.

In the embodiment of the present disclosure, the binarization process may be a binarization process with a fixed threshold or a binarization process with an adaptive threshold. For example, the bimodal method, the P parameter method, the iterative method, and the OTSU method, etc., the embodiment of the present disclosure does not limit the specific implementation of the binarization processing.

In the embodiment of the present disclosure, the semantic prediction result of each pixel in the plurality of pixels included in the first image may be obtained by processing the first image. In some possible implementation manners, the semantic prediction result of the pixel is obtained by judging the magnitude relationship between the semantic prediction value of the pixel and the above-mentioned first threshold. Optionally, the first threshold of the foregoing binarization processing may be preset or determined according to actual conditions, which is not limited in this embodiment of the present disclosure.

After the integrated semantic data and integrated central area data of the above image are obtained, step 103 may be performed.

103. Based on the integrated semantic data and the integrated central region data of the above image, obtain an instance segmentation result of the above image.

In some possible implementation manners, at least one instance central area of the above-mentioned image may be obtained based on the integrated central area data of the above-mentioned image, and the above-mentioned image may be determined based on the above-mentioned at least one instance central area and the integrated semantic data of the above-mentioned image The instance to which each pixel in multiple pixels of .

The above-mentioned integrated semantic data indicates at least one pixel located in the instance area in the image. For example, the integrated semantic data may include an integrated semantic value of each pixel in multiple pixels of the image, and the integrated semantic value is used to indicate whether the pixel In the instance area, or used to indicate that the pixel is located in the instance area or the background area. The above-mentioned integrated central area data indicates at least one pixel point located in the central area of the instance in the above-mentioned image. The value is used to indicate whether the pixel is located in the central area of the instance.

Optionally, at least one pixel contained in the instance region of the image may be determined through the integrated semantic data, and at least one pixel contained in the instance central region of the image may be determined through the integrated central region data. Based on the integrated central area data and integrated semantic data of the above image, the instance to which each pixel of the multiple pixels of the above image belongs can be determined, and an instance segmentation result of the image can be obtained.

The instance segmentation result obtained by the above method integrates the instance segmentation output results of N instance segmentation models, integrates the advantages of different instance segmentation models, no longer requires different instance segmentation models to have the same meaning of data output, and improves the instance segmentation accuracy .

In this embodiment of the present application, N sets of instance segmentation output data are acquired, wherein the above N sets of instance segmentation output data are the instance segmentation output results obtained by processing images with N instance segmentation models, and the above N sets of instance segmentation output data have Different data structures, the above-mentioned N is an integer greater than 1, and then the output data is segmented based on the above-mentioned N groups of instances to obtain the integrated semantic data and integrated central area data of the above-mentioned image, wherein the above-mentioned integrated semantic data indicates that the instance located in the above-mentioned image is The pixel points of the region, the above-mentioned integration center area data indicate the pixels located in the center area of the instance in the above-mentioned image, and then based on the integrated semantic data and the integration center area data of the above-mentioned image, the instance segmentation result of the above-mentioned image is obtained, which can be shown in Fig. In the instance segmentation problem of image processing, the complementary advantages of each instance segmentation model are realized, and the data output of each model is no longer required to have the same structure or meaning, and higher accuracy is achieved in the instance segmentation problem.

Please refer to FIG. 2 . FIG. 2 is a schematic flowchart of another image processing method disclosed in the embodiment of the present application. FIG. 2 is further optimized on the basis of FIG. 1 . The method may be executed by any electronic device, such as a terminal device, a server, or a processing platform, which is not limited in this embodiment of the present disclosure. As shown in Figure 2, the image processing method includes the following steps.

201. Obtain N sets of instance segmentation output data, wherein the above N sets of instance segmentation output data are respectively instance segmentation output results obtained by processing images with N instance segmentation models, and the above N sets of instance segmentation output data have different data structure, the above N is an integer greater than 1.

Wherein, for the above step 201, reference may be made to the specific description in step 101 of the embodiment shown in FIG. 1 , which will not be repeated here.

202. Based on the instance segmentation output data of the instance segmentation model, determine at least two pixel points located in the instance region in the image in the instance segmentation model.

A small area to which the instance center belongs can be defined as the instance center area, that is, the instance center area is an area within the instance area and smaller than the instance area, and the geometric center of the instance center area overlaps with the geometric center of the instance area or nearby, for example, the center of the instance center region is the instance center. Optionally, the central area of the example may be a circle, an ellipse or other shapes, and the central area of the above example may be set as required, and the embodiment of the present application does not limit the specific realization of the central area of the example. Optionally, the instance segmentation output data may include instance identification information corresponding to each pixel in at least two pixels located in the instance area in the image, for example, the instance ID is an integer greater than 0 such as 1, 2, or 3, or It can also be other values, and the instance identification information corresponding to the pixel in the background area can be a preset value, or the pixel in the background area does not correspond to any instance identification information. In this way, at least two pixel points located in the instance region in the image can be determined based on the instance identification information corresponding to each of the plurality of pixel points in the instance segmentation output data.

Optionally, the instance segmentation output data may not include the instance identification information corresponding to each pixel. At this time, at least two pixel points located in the instance area in the image can be obtained by processing the instance segmentation output data. The disclosed embodiments do not limit this.

After determining at least two pixel points located in the instance area in the above image, step 203 may be performed.

203. Based on the location information of at least two pixel points located in the instance area in the instance segmentation model, determine the instance center position of the instance segmentation model.

After determining at least two pixel points located in the instance region in the above-mentioned instance segmentation model, the position information of the above-mentioned at least two pixel points can be obtained, wherein, optionally, the position information can include the coordinates of the pixel points in the image , but the embodiments of the present disclosure are not limited thereto.

The instance center position of the above instance segmentation model can be determined according to the position information of the at least two pixel points above. The above instance center position is not limited to the geometric center position of the instance, but is the center of the instance region predicted in the instance region The location is used to further determine the central area of the instance, which can be understood as any position in the central area of the above instance.

Optionally, the average value of the positions of at least two pixel points located in the instance region may be used as the instance center position of the instance segmentation model.

Specifically, the coordinates of the at least two pixel points located in the instance area may be averaged as the coordinates of the instance center position of the instance segmentation model, so as to determine the instance center position.

204. Determine an instance center region of the instance segmentation model based on the instance center position of the instance segmentation model and the position information of the at least two pixel points.

Specifically, based on the instance center position of the above instance segmentation model and the position information of the at least two pixel points, determine the maximum distance between the above at least two pixel points and the above instance center position, and then determine the first threshold based on the above maximum distance , and then, among the at least two pixel points, the pixel points whose distance from the central position of the instance is less than or equal to the first threshold may be determined as the pixel points in the central region of the instance.

For example, the distance (pixel distance) from each pixel to the center of the instance can be calculated based on the instance center position of the above instance segmentation model and the position information of the at least two pixels above, and the electronic device can pre-set the above first A threshold algorithm, for example, the above-mentioned first threshold can be set to 30% of the maximum distance among the above-mentioned pixel point distances, after determining the maximum distance among the above-mentioned pixel point distances, the above-mentioned first threshold value can be obtained by calculating, based on this, the pixel If the pixel points whose point distance is smaller than the above-mentioned first threshold are determined as the pixel points of the central area of the above-mentioned instance, that is, the central area of the above-mentioned instance is determined.

Optionally, the sample image may also be etched to obtain an etched sample image, and the central region of the instance may be determined based on the etched sample image.

The erosion operation of the image means to detect the image with a certain structural element in order to find out the area where the structural element can be placed inside the image. The image erosion processing mentioned in the embodiment of the present application may include the above erosion operation, and the erosion operation is a process in which structural elements are translated and filled in the etched image. Judging from the results after erosion, the foreground area of the image shrinks, and the boundary of the area becomes blurred. At the same time, some relatively small isolated foreground areas are completely corroded, achieving the effect of filtering.

For example, for each instance mask, first use a 5×5 convolution kernel to perform image erosion processing on the instance mask (mask), and then average the coordinates of multiple pixels included in the instance to obtain the center position of the instance , and determine the maximum distance from all the pixels in the instance to the center of the instance, and determine the pixels whose distance from the center of the instance is less than 30% of the above-mentioned maximum distance as the pixels in the center area of the instance, That is, the central region of the instance is obtained. In this way, after the example mask in the sample image is shrunk a circle, image binarization is performed to obtain the binary image mask predicted by the central region.

In addition, optionally, based on the coordinates of the pixel points contained in the instance marked in the sample image and the center position of the instance, the relative position information of the center of the pixel point can be obtained, that is, the relative position information between the above-mentioned pixel point and the center of the instance , such as a vector from a pixel point to the center of an instance, and use the relative position information as a supervision for neural network training, but the embodiments of the present disclosure are not limited thereto.

205. Based on the semantic data of each instance segmentation model in the above N instance segmentation models, determine the semantic voting value of each pixel in the multiple pixels of the image.

The electronic device may perform semantic voting on each of the plurality of pixels based on the semantic data of each of the above N instance segmentation models, and determine the semantics of each of the plurality of pixels of the above image For the voting value, for example, the semantic data of the above-mentioned instance segmentation model is processed by using the voting based on the sliding window, and the semantic voting value of each pixel is determined, and then step 206 can be executed.

206. Perform binarization processing on the semantic voting value of each pixel in the above-mentioned plurality of pixels to obtain the integrated semantic value of each pixel in the above-mentioned image, wherein the integrated semantic data of the above-mentioned image includes the above-mentioned multiple The integrated semantic value of each pixel in the pixel.

The semantic voting value determined above comes from the above N instance segmentation models. Further, the semantic voting value of each pixel can be binarized to obtain the integrated semantic value of each pixel in the above image, which can be understood as The semantic masks obtained by different instance segmentation models are summed to obtain an integrated semantic mask.

Specifically, the second threshold can be determined based on the number N of the above multiple instance segmentation models; based on the above second threshold, the semantic voting value of each pixel in the above multiple pixels is binarized to obtain the above The integrated semantic value of each pixel in the image.

The value of each pixel in the above-mentioned plurality of pixel points has the value possibility of the number of instance segmentation models, and the second threshold can be determined based on the number N of the above-mentioned multiple instance segmentation models. For example, the above-mentioned second threshold can be It is the rounded up result of N/2.

The second threshold can be used as the judgment basis of the binarization process in this step to obtain the integrated semantic value of each pixel in the above image. The calculation method of the above-mentioned second threshold can be stored in the electronic device, for example, it is stipulated that the above-mentioned preset pixel point threshold is N/2, and if N/2 is not an integer, it is rounded up. For example, if there are 4 sets of instance segmentation output data obtained by 4 instance segmentation models, then N=4, 4/2=2, the second threshold at this time is 2, compare the above semantic voting value with the above second threshold, If the semantic voting value is greater than or equal to 2, it is truncated to 1, and if it is less than 2, it is truncated to 0, thus obtaining the integrated semantic value of each pixel in the above image, and the output data at this time can specifically be an integrated semantic binary image. The above-mentioned integrated semantic value can be understood as the semantic segmentation result of each pixel above, and based on this, the instance to which the pixel belongs can be determined to realize instance segmentation.

207. Based on the integrated semantic value of each pixel in the plurality of pixels of the above image and the center area of at least one instance, perform random walk to obtain the instance to which each pixel above belongs.

Random walk (random walk), also known as random walk, random walk, etc., means that based on past performance, it is impossible to predict future development steps and directions. The core concept of random walk is that the conserved quantity carried by any random walker corresponds to a diffusion transport law, which is close to Brownian motion and is the ideal mathematical state of Brownian motion. The basic idea of the random walk for image processing in the embodiment of the present application is to regard the image as a connected weighted undirected graph composed of fixed vertices and edges, start a random walk from unmarked vertices, and reach all kinds of markers for the first time The probability of the vertex represents the possibility that the unmarked point belongs to the marked class, and the label of the class with the highest probability is assigned to the unmarked vertex to complete the segmentation.

Based on the integrated semantic value of each pixel in the multiple pixels of the above image and the central region of the above at least one instance, use the form of random walk to judge the distribution of the pixel according to the integrated semantic value of the pixel, so as to obtain the above-mentioned each The instance to which a pixel belongs, for example, the instance corresponding to the central area of the instance closest to the pixel can be determined as the instance to which the pixel belongs. In this embodiment of the present application, the final instance segmentation result can be obtained by obtaining the final integrated semantic graph and integrated central region graph, combined with a specific implementation of the above-mentioned connected region search and random walk (nearest allocation) to determine instance pixel assignment.

The instance segmentation result obtained by the above method integrates the instance segmentation output results of N instance segmentation models, integrates the advantages of these instance segmentation models, no longer requires different instance segmentation models to have the same meaning of the continuous probability map output, and improves the instance Segmentation accuracy.

The method in the embodiment of the present application is applicable to any instance segmentation problem, for example, in clinical auxiliary diagnosis. After the doctor obtains the digital scan image of the patient's organ tissue slice, the image is input into the processing steps of the embodiment of the present application, and the pixel point mask of each independent cell nucleus can be obtained, and the doctor can use this as a basis to calculate the The cell density and cell morphology characteristics of the cell, and then draw a medical judgment. Another example is around the hive, after the breeder obtains the dense bee flying images around the hive, he can use this algorithm to obtain the instance pixel mask of each individual bee, and can perform macroscopic bee counting and behavior pattern calculations, etc. It has great practical value.

In the specific application of the embodiment of the present application, for the bottom-up method, the UNet model can be preferably applied. UNet was first developed for semantic segmentation and efficiently fuses information from multiple scales. For the top-down approach, the MaskR-CNN model can be applied, which extends Faster R-CNN by adding a head for the segmentation task. Furthermore, the proposed MaskR-CNN can align tracked features to the input, avoiding any quantization of bilinear interpolation. Alignment is very important for pixel-level tasks, such as instance segmentation tasks.

The network structure of the UNet model consists of a contracting path and an expanding path. Among them, the contraction path is used to obtain context information (context), and the expansion path is used for precise positioning (localization), and the two paths are symmetrical to each other. The network can be trained end-to-end from very few images and outperforms the previous best method (sliding window convolutional network) for segmenting cellular structures such as neurons in electron microscopy. In addition to running very fast,

In a specific implementation, the UNet and Mask R-CNN models can be used to segment and predict instances, obtain the semantic mask of each instance segmentation model, and integrate them through pixel voting (Vote). The center mask of each instance segmentation model is then computed by erosion processing, and the center masks are integrated. Finally, instance segmentation results are obtained from the integrated semantic and center masks using a random walk algorithm.

For the above results, the cross-validation (Cross-validation) method can be used for evaluation. Cross-validation is primarily used in modeling applications. In the given modeling samples, take most of the samples to build the model, leave a small part of the samples to predict with the model just built, and calculate the forecast error of this small part of the samples, and record their sum of squares. In the embodiment of this application, 3-fold cross-validation can be used for evaluation, and three UNet models with AJI (5) scores of 0.605, 0.599, and 0.589 are combined with a MaskR-CNN model with an AJI (5) score of 0.565, and implemented using this application The final AJI (5) score of the result obtained by the example method is 0.616, which shows that the image processing method of the present application has obvious advantages.

In this embodiment of the present application, N sets of instance segmentation output data are acquired, wherein the above N sets of instance segmentation output data are the instance segmentation output results obtained by processing images with N instance segmentation models, and the above N sets of instance segmentation output data have Different data structures, the above-mentioned N is an integer greater than 1, and based on the instance segmentation output data of the above-mentioned instance segmentation model, it is determined that in the above-mentioned instance segmentation model, at least two pixel points located in the instance area in the above-mentioned image are based on the above-mentioned instance segmentation Based on the position information of at least two pixels in the instance area in the model, determine the instance center position of the above-mentioned instance segmentation model, and determine the above-mentioned instance segmentation model based on the instance center position of the above-mentioned instance segmentation model and the position information of the above-mentioned at least two pixel points Based on the semantic data of each instance segmentation model in the above N instance segmentation models, determine the semantic voting value of each pixel in the multiple pixels of the above image, and for each of the above multiple pixel points The semantic voting value of the pixel is binarized to obtain the integrated semantic value of each pixel in the above image, wherein the integrated semantic data of the above image includes the integrated semantic value of each pixel in the above multiple pixels , based on the integrated semantic value of each pixel in the multiple pixels of the above image and the center area of at least one instance above, random walk is performed to obtain the instance to which each pixel above belongs, which can be used in the instance segmentation problem of image processing In this method, the complementary advantages of each instance segmentation model are realized, and the data output of each model is no longer required to have the same structure or meaning, and higher accuracy is achieved in the instance segmentation problem.

Please refer to Fig. 3. Fig. 3 is a schematic diagram of the image representation of a cell instance segmentation disclosed in the embodiment of the present application. Instance segmentation results with higher accuracy. Use N kinds of instance segmentation models (only 4 are shown in the figure) to give the instance prediction masks of the input picture (different colors in the figure represent different cell instances), and convert the instance prediction masks into semantic prediction segmentation. After the semantic mask and the center region masked by the center prediction segmentation, the pixel points are voted separately, and then integrated to finally obtain the instance segmentation result. It fixed the error of bonding two cells in the middle of method 2, and also repaired the phenomenon that there are actually three cells in the lower left corner and a small cell in the middle that failed to be found by the four methods. The ensemble method can be ensembled on arbitrary instance segmentation models, combining the advantages of different methods. Through the above examples, the specific process and advantages of the foregoing embodiments can be understood more clearly.

The foregoing mainly introduces the solutions of the embodiments of the present application from the perspective of executing the process on the method side. It can be understood that, in order to realize the above-mentioned functions, the electronic device includes hardware structures and/or software modules corresponding to each function. Those skilled in the art should easily realize that the present application can be realized in the form of hardware or a combination of hardware and computer software in combination with the units and algorithm steps of each example described in the embodiments disclosed herein. Whether a certain function is executed by hardware or computer software drives the hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for specific applications, but such implementation should not be regarded as exceeding the scope of the present application.

The embodiment of the present application may divide the electronic device into functional units according to the above method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The above-mentioned integrated units can be implemented not only in the form of hardware, but also in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic, and is only a logical function division, and there may be another division manner in actual implementation.

Please refer to FIG. 4 . FIG. 4 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. As shown in FIG. 4 , the electronic device 400 includes: an acquisition module 410, a conversion module 420, and a segmentation module 430, wherein: the acquisition module 410 is used to acquire N sets of instance segmentation output data, wherein the N sets of instance segmentation The output data are the instance segmentation output results obtained by processing the image with N instance segmentation models, and the N sets of instance segmentation output data have different data structures, and the N is an integer greater than 1; the conversion module 420 , for segmenting output data based on the N groups of instances to obtain integrated semantic data and integrated central area data of the image, wherein the integrated semantic data indicates pixels located in the instance area in the image, the The integrated central area data indicates the pixels located in the central area of the instance in the image; the segmentation module 430 is configured to obtain the instance segmentation result of the image based on the integrated semantic data and integrated central area data of the image .

Optionally, the conversion module 420 includes a first conversion unit 421 and a second conversion unit 422, wherein: the first conversion unit 421 is configured to segment an instance of a model based on each instance of the N instance segmentation model Segment the output data to obtain the semantic data and central area data of each instance segmentation model; the second conversion unit 422 is used to segment the semantic data and central area of each instance segmentation model based on the N instance segmentation models data to obtain the integrated semantic data and integrated central area data of the image.

Optionally, the first conversion unit 421 is specifically configured to: determine, based on the instance segmentation output data of the instance segmentation model, that each of the multiple pixels of the image in the instance segmentation model corresponds to instance identification information; based on the instance identification information corresponding to each pixel in the plurality of pixels in the instance segmentation model, the semantic prediction value of each pixel in the instance segmentation model is obtained, wherein, The semantic data of the instance segmentation model includes a semantic prediction value of each pixel in the plurality of pixels of the image.

Optionally, the first conversion unit 421 is further configured to: determine, in the instance segmentation model, at least two pixels located in the instance region in the image based on the instance segmentation output data of the instance segmentation model point; based on the position information of at least two pixel points located in the instance region in the instance segmentation model, determine the instance center position of the instance segmentation model; based on the instance center position of the instance segmentation model and the at least two pixels Point location information to determine the instance center region of the instance segmentation model.

Optionally, the conversion module 420 further includes an erosion processing unit 423, configured to perform erosion processing on the instance segmentation output data of the instance segmentation model to obtain the erosion data of the instance segmentation model; the first conversion unit 421 specifically uses Then, based on the erosion data of the instance segmentation model, determine at least two pixel points located in the instance region in the image in the instance segmentation model.

Optionally, the first conversion unit 421 is specifically configured to use the average value of the positions of at least two pixel points located in the instance region as the instance center position of the instance segmentation model.

Optionally, the first conversion unit 421 is further configured to: determine the relationship between the at least two pixel points and the instance center position of the instance segmentation model and the position information of the at least two pixel points. The maximum distance of the center position; based on the maximum distance, determine a first threshold; determine the pixel point of the at least two pixel points whose distance from the center position of the instance is less than or equal to the first threshold value as an instance Pixels in the central area.

Optionally, the conversion module 420 is specifically configured to: determine the semantic vote of each pixel in the multiple pixels of the image based on the semantic data of each instance segmentation model in the N instance segmentation models value; the semantic voting value of each pixel in the plurality of pixels is binarized to obtain the integrated semantic value of each pixel in the image, wherein the integrated semantic data of the image includes An integrated semantic value of each pixel in the plurality of pixels.

Optionally, the conversion module 420 is specifically further configured to: determine a second threshold based on the number N of the plurality of instance segmentation models; The semantic voting value of the pixel is binarized to obtain the integrated semantic value of each pixel in the image.

Optionally, the second threshold is a rounded-up result of N/2.

Optionally, the segmentation module 430 includes a central area unit 431 and a determination unit 432, wherein: the central area unit 431 is configured to obtain at least one of the images based on the integrated central area data of the image Instance central area; the determining unit 432 is configured to determine the instance to which each pixel of the multiple pixels of the image belongs based on the at least one instance central area and the integrated semantic data of the image.

Optionally, the determining unit 432 is specifically configured to perform a random walk based on the integrated semantic value of each pixel in the plurality of pixels of the image and the central area of the at least one instance, to obtain each The instance the pixel belongs to.

Implementing the electronic device 400 shown in FIG. 4 , the electronic device 400 can obtain N sets of instance segmentation output data, wherein the above N sets of instance segmentation output data are the instance segmentation output results obtained by processing images with N instance segmentation models, And the above N groups of instance segmentation output data have different data structures, the above N is an integer greater than 1, and then based on the above N groups of instance segmentation output data, the integrated semantic data and integrated central area data of the above image are obtained, wherein the above integrated The semantic data indicates the pixels located in the instance area in the image above, and the integrated center area data indicates the pixels located in the instance center area in the image above, and then based on the integrated semantic data and the integrated center area data of the above image, the above image is obtained The instance segmentation results of the image can realize the complementary advantages of each instance segmentation model in the instance segmentation problem of image processing, no longer require each model to have the same structure or meaning of data output, and achieve higher accuracy in the instance segmentation problem .

Please refer to FIG. 5 . FIG. 5 is a schematic structural diagram of another electronic device disclosed in an embodiment of the present application. As shown in FIG. 5 , the electronic device 500 includes a processor 501 and a memory 502, wherein the electronic device 500 may further include a bus bar 503, the processor 501 and the memory 502 may be connected to each other through the bus bar 503, and the bus bar 503 may It is a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 503 can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in FIG. 5 , but it does not mean that there is only one bus bar or one type of bus bar. Wherein, the electronic device 500 may further include an input and output device 504, and the input and output device 504 may include a display screen, such as a liquid crystal display screen. The memory 502 is used to store computer programs; the processor 501 is used to invoke the computer programs stored in the memory 502 to execute some or all of the method steps mentioned in the embodiments of FIG. 1 and FIG. 2 .

Implementing the electronic device 500 shown in FIG. 5 , the electronic device 500 can acquire N sets of instance segmentation output data, wherein the above N sets of instance segmentation output data are the instance segmentation output results obtained by processing images with N instance segmentation models, And the above N groups of instance segmentation output data have different data structures, the above N is an integer greater than 1, and then based on the above N groups of instance segmentation output data, the integrated semantic data and integrated central area data of the above image are obtained, wherein the above integrated The semantic data indicates the pixels located in the instance area in the image above, and the integrated center area data indicates the pixels located in the instance center area in the image above, and then based on the integrated semantic data and the integrated center area data of the above image, the above image is obtained The instance segmentation results of the image can realize the complementary advantages of each instance segmentation model in the instance segmentation problem of image processing, no longer require each model to have the same structure or meaning of data output, and achieve higher accuracy in the instance segmentation problem .

An embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables the computer to execute any part or part of the image processing method described in the above-mentioned method embodiments. All steps.

It should be noted that for the foregoing method embodiments, for the sake of simple description, they are expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence. Depending on the application, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical or other forms.

The unit (module) described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may also be distributed to multiple networks. road unit. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented not only in the form of hardware, but also in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of software products, and the computer software products are stored in a memory. Several instructions are included to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), mobile hard disk, magnetic disk or optical disk, etc., which can store program codes. medium.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: Pen drive, ROM, RAM, floppy disk or CD, etc.

The embodiments of the present application have been introduced in detail above, and specific examples have been used in this paper to illustrate the principles and implementation methods of the present application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present application; meanwhile, for Those skilled in the art will have changes in specific implementation methods and application scopes based on the ideas of the present application. In summary, the contents of this specification should not be construed as limiting the present application.

101, 102, 103, 201, 202, 203, 204, 205, 206, 207: steps 400, 500: electronic equipment 410: Get module 420: conversion module 421: The first conversion unit 422: Second conversion unit 423: Corrosion Treatment Unit 430:Split module 431: Central area unit 432: determine the unit 501: Processor 502: memory 503: busbar 504: Input and output devices

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings that are required in the description of the embodiments or the prior art. FIG. 1 is a schematic flowchart of an image processing method disclosed in an embodiment of the present application. Fig. 2 is a schematic flowchart of another image processing method disclosed in the embodiment of the present application. FIG. 3 is a schematic diagram of an image representation of a cell instance segmentation disclosed in an embodiment of the present application. Fig. 4 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. FIG. 5 is a schematic structural diagram of another electronic device disclosed in an embodiment of the present application.

101, 102, 103: steps

Claims (10)

An image processing method, comprising: acquiring N sets of instance segmentation output data, wherein the N sets of instance segmentation output data are the instance segmentation output results obtained by processing images with N different instance segmentation models, and the N groups of instance segmentation output data have data structures representing different meanings, the N is an integer greater than 1, and the N different instance segmentation models correspond to different algorithms or neural network structures; based on the N groups of instance segmentation output data, to obtain the integrated semantic data and integrated central area data of the image, wherein the integrated semantic data indicates the pixels located in the instance area in the image, and the integrated central area data indicates the pixels located in the image in the Pixels in the central region of the instance; and based on the integrated central region data of the image, at least one central region of the instance of the image is obtained; based on the integrated semantic value of each pixel in the multiple pixels of the image Perform a random walk with the at least one instance center area to obtain the instance to which each pixel belongs, wherein the integrated semantic data includes the integrated semantic value of each pixel in the plurality of pixels of the image . According to the image processing method described in item 1 of the scope of the patent application, the output data is segmented based on the N groups of instances, and the integrated semantic data and integrated central area data of the image are obtained, including: for the N different instances For each instance segmentation model in the segmentation model, based on the instance segmentation output data of the instance segmentation model, the semantic data and central region data of the instance segmentation model are obtained; and Based on the semantic data and central area data of each of the N different instance segmentation models, the integrated semantic data and integrated central area data of the image are obtained. According to the image processing method described in item 2 of the scope of patent application, based on the instance segmentation output data of the instance segmentation model, the semantic data and central area data of the instance segmentation model are obtained, including: based on the instance segmentation model Instance segmentation output data, determining the instance identification information corresponding to each of the plurality of pixels in the image in the instance segmentation model; and based on each of the plurality of pixels in the instance segmentation model The instance identification information corresponding to the pixel, to obtain the semantic prediction value of each pixel in the instance segmentation model, wherein the semantic data of the instance segmentation model includes each of the plurality of pixels of the image The semantic prediction value of the pixel. According to the image processing method described in item 2 or 3 of the scope of patent application, based on the instance segmentation output data of the instance segmentation model, the semantic data and central area data of the instance segmentation model are obtained, further comprising: based on the instance Instance segmentation output data of the segmentation model, determining in the instance segmentation model, at least two pixel points located in the instance region in the image; based on the positions of at least two pixel points located in the instance region in the instance segmentation model information, determine the instance center position of the instance segmentation model; and determine the instance center area of the instance segmentation model based on the instance center position of the instance segmentation model and the position information of the at least two pixel points. In the image processing method described in item 4 of the scope of patent application, based on the instance segmentation output data of the instance segmentation model, it is determined that in the instance segmentation model, at least two pixels located in the instance region of the image Before the point, the image processing method further includes: performing erosion processing on the instance segmentation output data of the instance segmentation model to obtain the erosion data of the instance segmentation model; and based on the instance segmentation output data of the instance segmentation model, Determining that in the instance segmentation model, at least two pixel points located in the instance region in the image include: determining, in the instance segmentation model, that in the image is based on corrosion data of the instance segmentation model At least two pixels in the instance area. According to the image processing method described in item 4 of the patent scope of the application, the determination of the instance center position of the instance segmentation model based on the position information of at least two pixel points located in the instance area in the instance segmentation model includes: Taking the average value of the positions of the at least two pixel points located in the instance region as the instance center position of the instance segmentation model. According to the image processing method described in item 4 of the scope of patent application, the instance center area of the instance segmentation model is determined based on the instance center position of the instance segmentation model and the position information of the at least two pixels, including: Based on the instance center position of the instance segmentation model and the position information of the at least two pixel points, determine the maximum distance between the at least two pixel points and the instance center position; determine a first threshold based on the maximum distance ; and the distance between the at least two pixel points and the central position of the instance is small A pixel point equal to or greater than the first threshold is determined as a pixel point in the central region of the instance. An electronic device, comprising: an acquisition module, configured to acquire N sets of instance segmentation output data, wherein the N sets of instance segmentation output data are respectively instance segmentation output results obtained by processing images with N different instance segmentation models, And the N groups of instance segmentation output data have data structures representing different meanings, the N is an integer greater than 1, and the N different instance segmentation models correspond to different algorithms or neural network structures; the conversion module is used for Based on the N groups of instance segmentation output data, the integrated semantic data and integrated central area data of the image are obtained, wherein the integrated semantic data indicates the pixels located in the instance area in the image, and the integrated central area The data indicates the pixels located in the central area of the instance in the image; and the segmentation module is configured to obtain at least one central area of the instance of the image based on the integrated central area data of the image; based on the image The integrated semantic value of each pixel in the plurality of pixels of the image and the central area of the at least one instance are randomly walked to obtain the instance to which each pixel belongs, wherein the integrated semantic data includes the image The integrated semantic value of each pixel in multiple pixels of the image. An electronic device, including a processor and a memory, the memory is used to store a computer program, the computer program is configured to be executed by the processor, and the processor is used to execute items 1 to 7 of the scope of the patent application The image processing method described in any one. A computer-readable storage medium, the computer-readable storage medium is used to store a computer program, wherein the computer program enables the computer to execute the The image processing method described in any one of items 1 to 7.

Images