TW202013311A - Image processing method, electronic device, and storage medium - Google Patents

Abstract

The embodiment of the present application discloses an image processing method, an electronic device, and a storage medium, wherein the method includes: acquiring N sets of segmentation output data, wherein the N sets of segmentation output data are respectively performed by N instance segmentation models Processing the obtained instance segmentation output result, and the N sets of instance segmentation output data have different data structures, the N being an integer greater than 1; and segmenting the output data based on the N sets of instances to obtain integrated semantic data of the image And integrating central region data, wherein the integrated semantic data indicates pixel points in the image region of the image, the integrated central region data indicating pixel points in the image center region of the image; integration based on the image Semantic data and integration of central region data, to obtain the example segmentation results of the image, can achieve the complementary advantages of each instance segmentation model, and achieve higher precision in the instance segmentation problem.

Description

Image processing method, electronic equipment and storage medium

This application relates to the field of computer vision technology, and in particular to an image processing method, electronic equipment, and storage media.

Image processing, also known as image processing, is a technique for analyzing images with a computer to achieve the desired result. Image processing is generally exponential bit image processing. Digital image refers to a large two-dimensional array obtained by shooting with industrial cameras, cameras, scanners, etc. The elements of this array are called pixels, and the value is called gray. Degree value. Image processing plays a very important role in many fields.

The embodiments of the present application provide an image processing method, an electronic device, and a storage medium.

A first aspect of an embodiment of the present application provides an image processing method, including: acquiring N sets of instance segmentation output data, wherein the N sets of instance segmentation output data are respectively instance segments obtained by processing images by N instance segmentation models Output results, and the N sets of instance segmentation output data have different data structures, and N is an integer greater than 1; based on the N sets of instance segmentation output data, the integrated semantic data and integration center area of the image are obtained Data, wherein the integrated semantic data indicates pixels located in the instance area in the image, and the integrated center area data indicates pixels located in the center area of the instance in the image; integrated semantics based on the image The data and the central area data are integrated to obtain an instance segmentation result of the image.

In an optional implementation manner, the segmenting the output data based on the N sets of instances to obtain the integrated semantic data and the integrated central area data of the image includes: segmenting each instance in the N instances of the model Segmentation model, based on the instance segmentation output data of the instance segmentation model, to obtain the semantic data and central region data of the instance segmentation model; based on the N instance segmentation models, the semantic data and central region data of each instance segmentation model To obtain the integrated semantic data and integrated central area data of the image.

In an optional embodiment, the instance segmentation output data based on the instance segmentation model to obtain the semantic data and central region data of the instance segmentation model includes: instance segmentation output data based on the instance segmentation model , Determining instance identification information corresponding to each pixel in the plurality of pixels of the image in the instance segmentation model; based on the instance corresponding to each pixel in the plurality of pixels in the instance segmentation model Identification information to obtain the semantic prediction value of each pixel in the instance segmentation model, wherein the semantic data of the instance segmentation model includes the semantic prediction of each pixel in the plurality of pixels of the image value.

In an optional embodiment, the instance segmentation output data based on the instance segmentation model to obtain semantic data and central region data of the instance segmentation model further includes: instance segmentation output based on the instance segmentation model Data to determine at least two pixels in the instance region in the image in the instance segmentation model; determine the instance based on the position information of at least two pixels in the instance region in the instance segmentation model The instance center position of the segmentation model; based on the instance center position of the instance segmentation model and the position information of the at least two pixels, the instance center area of the instance segmentation model is determined.

In an optional embodiment, in the instance segmentation output data based on the instance segmentation model, it is determined that in the instance segmentation model, the image is located before at least two pixels of the instance region in the image, and further includes: Corrosion processing is performed on the instance segmentation output data of the instance segmentation model to obtain corrosion data of the instance segmentation model. In this case, the instance segmentation output data based on the instance segmentation model, determining that in the instance segmentation model, at least two pixels located in an instance region in the image includes: segmentation based on the instance The corrosion data of the model determines that at least two pixels located in the instance area in the image in the instance segmentation model.

In an optional embodiment, the determining the center position of the instance of the instance segmentation model based on the position information of at least two pixels located in the instance region in the instance segmentation model includes: The average value of the positions of the at least two pixels is used as the instance center position of the instance segmentation model.

In an optional embodiment, the determining an instance center area of the instance segmentation model based on the instance center position of the instance segmentation model and the position information of the at least two pixels includes: based on the instance The center position of the instance of the segmentation model and the position information of the at least two pixels, to determine the maximum distance between the at least two pixels and the center position of the instance; based on the maximum distance, determine a first threshold; Among the at least two pixels, the distance from the center position of the instance is less than or equal to the first threshold is determined as the pixel in the center area of the instance.

In an optional implementation manner, the segmenting the semantic data and central region data of each instance in the N instance segmentation models to obtain the integrated semantic data and integrated central region data of the image includes: Based on the semantic data of each instance segmentation model in the N instance segmentation models, determine the semantic voting value of each pixel in the plurality of pixels of the image; for each pixel in the plurality of pixels The semantic voting value of is binarized to obtain the integrated semantic value of each pixel in the image, wherein the integrated semantic data of the image includes the integrated semantic of each pixel in the plurality of pixels value.

In an optional implementation manner, the binarizing the semantic voting value of each pixel in the plurality of pixels to obtain the integrated semantic value of each pixel in the image includes: Based on the number N of the multiple instance segmentation models, a second threshold is determined; based on the second threshold, the semantic voting value of each pixel in the plurality of pixels is binarized to obtain the The integrated semantic value of each pixel in the image.

In an optional implementation manner, the second threshold is an upward rounding result of N/2.

In an optional implementation manner, the obtaining of an instance segmentation result of the image based on the integrated semantic data of the image and the integrated central area data includes: based on the integrated central area data of the image, obtaining At least one instance central area of the image; based on the at least one instance central area and the integrated semantic data of the image, an instance to which each pixel of the plurality of pixels of the image belongs is determined.

In an optional embodiment, the determining, based on the integrated semantic data of the central area of the at least one instance and the image, the instance to which each pixel of the plurality of pixels of the image belongs includes: Based on the integrated semantic value of each pixel in the plurality of pixels of the image and the at least one instance central area, a random walk is performed to obtain the instance to which each pixel belongs.

A second aspect of an embodiment of the present application provides an electronic device, including an acquisition module, a conversion module, and a segmentation module, wherein: the acquisition module is configured to acquire N sets of instance segmentation output data, wherein the N sets of instance segmentation output data are respectively An instance segmentation output result obtained by processing an image for N instance segmentation models, and the N sets of instance segmentation output data have different data structures, and the N is an integer greater than 1; the conversion module is used to The N sets of instance segmentation output data to obtain integrated semantic data and integrated central area data of the image, wherein the integrated semantic data indicates pixels located in the instance area in the image, and the integrated central area data Indicate the pixels located in the center area of the image in the image; the segmentation module is used to obtain an instance segmentation result of the image based on the integrated semantic data and the integrated center area data of the image.

In an optional implementation manner, the conversion module includes a first conversion unit and a second conversion unit, wherein: the first conversion unit is configured to divide the model for each of the N instance division models, Based on the instance segmentation output data of the instance segmentation model, obtain the semantic data and central region data of the instance segmentation model; the second conversion unit is used to segment the model based on each instance of the N instance segmentation models Semantic data and central area data to obtain the integrated semantic data and integrated central area data of the image.

In an optional implementation manner, the first conversion unit is specifically configured to determine, based on the instance segmentation output data of the instance segmentation model, each of the plurality of pixels of the image in the instance segmentation model Instance identification information corresponding to each pixel; based on the instance identification information corresponding to each pixel in the plurality of pixels in the instance segmentation model, the semantic prediction of each pixel in the instance segmentation model is obtained Value, wherein the semantic data of the instance segmentation model includes the semantic prediction value of each pixel in the plurality of pixels of the image.

In an optional implementation manner, the first conversion unit is further specifically configured to determine, based on the instance segmentation output data of the instance segmentation model, in the instance segmentation model, the image located in the instance region At least two pixels; based on the position information of at least two pixels located in the instance region in the instance segmentation model, determining the instance center position of the instance segmentation model; based on the instance center position of the instance segmentation model and the The position information of at least two pixels determines the instance center area of the instance segmentation model.

In an optional embodiment, the conversion module further includes a corrosion processing unit, configured to perform corrosion processing on the instance segmentation output data of the instance segmentation model to obtain corrosion data of the instance segmentation model; the first conversion unit Specifically, it is used to determine, based on the corrosion data of the instance segmentation model, at least two pixels located in the instance region in the image in the instance segmentation model.

In an optional implementation manner, the first conversion unit is specifically configured to use the average value of the positions of the at least two pixels located in the instance area as the instance center position of the instance segmentation model.

In an optional implementation manner, the first conversion unit is further specifically configured to determine the at least two pixels based on the instance center position of the instance segmentation model and the position information of the at least two pixels The maximum distance from the center position of the instance; based on the maximum distance, a first threshold is determined; the pixel between the at least two pixels and the center position of the instance is less than or equal to the first threshold The point is determined as the pixel in the central area of the instance.

In an optional embodiment, the conversion module is specifically configured to: determine the semantic voting value of each pixel of the plurality of pixels of the image based on the semantic data of the instance segmentation model; The semantic voting value of each pixel in the plurality of pixels is binarized to obtain the integrated semantic value of each pixel in the image, wherein the integrated semantic data of the image includes the multiple The integrated semantic value of each pixel in the pixel.

In an optional embodiment, the conversion module is further specifically configured to: determine a second threshold based on the number N of the multiple instance segmentation models; and based on the second threshold, determine the second threshold The semantic voting value of each pixel in the point is binarized to obtain the integrated semantic value of each pixel in the image.

In an optional implementation manner, the second threshold is an upward rounding result of N/2.

A third aspect of the embodiments of the present application provides another electronic device, including a processor and a memory, where the memory is used to store a computer program, the computer program is configured to be executed by the processor, and the processor is used to execute Part or all of the steps as described in any method of the first aspect of the embodiments of the present application.

A fourth aspect of the embodiments of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program causes the computer to perform any method as described in the third aspect of the embodiments of the present application Describe some or all of the steps.

In this embodiment of the present application, N sets of instance segmentation output data are obtained, wherein the above N sets of instance segmentation output data are respectively instance segmentation output results obtained by processing images by N instance segmentation models, and the above N group instance segmentation output data have different The data structure of N is an integer greater than 1, and then the output data is divided based on the N sets of instances to obtain integrated semantic data and integrated central area data of the image, where the integrated semantic data indicates that the image is located in the instance area Pixels, the integrated center area data indicates pixels located in the center area of the instance in the image, and then based on the integrated semantic data and integrated center area data of the image, an instance segmentation result of the image is obtained. In dealing with the instance segmentation problem, the advantages of each instance segmentation model are complemented. It is no longer required that each model has the same structure or meaning of data output, and a higher accuracy is obtained in the instance segmentation problem.

The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application.

The description and patent application scope of the present application and the terms "first" and "second" in the above drawings are used to distinguish different objects, not to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally also includes Other steps or units inherent to these processes, methods, products, or equipment.

Reference herein to "embodiments" means that specific features, structures, or characteristics described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art understand explicitly and implicitly that the embodiments described herein can be combined with other embodiments.

The electronic device involved in the embodiments of the present application may allow multiple other terminal devices to access. The above-mentioned electronic device includes a terminal device. In a specific implementation, the above-mentioned terminal device includes, but is not limited to, other portable devices such as a mobile phone, laptop computer, or tablet computer with a touch-sensitive surface (eg, touch screen display and/or touch pad) equipment. It should also be understood that, in some embodiments, the terminal device is not a portable communication device, but a desktop computer with a touch-sensitive surface (eg, touch screen display and/or touch pad).

Deep learning is a method of machine learning based on representational learning of data. Observed values (such as an image) can be expressed in a variety of ways, such as a vector of intensity values for each pixel, or more abstractly expressed as a series of edges, regions of a particular shape, etc. It is easier to learn tasks from examples (for example, face recognition or facial expression recognition) using some specific representation methods. The benefit of deep learning is to use unsupervised or semi-supervised feature learning and hierarchical feature extraction efficient algorithms to replace manual feature acquisition. Deep learning is a new field in machine learning research. Its motivation is to establish and simulate a neural network for human brain analysis and learning, so that it can mimic the mechanism of the human brain to interpret data, such as images, sounds, and text.

Like machine learning methods, deep machine learning methods also have supervised learning and unsupervised learning. The learning models established under different learning frameworks are very different. For example, Convolutional neural network (CNN) is a machine learning model under deep supervised learning, which can also be called a deep learning-based network structure model, and Deep Belief Net (DBN) It is a machine learning model under unsupervised learning.

The following describes the embodiments of the present application in detail. It should be understood that the embodiments of the present disclosure may be applied to segmentation of cell nuclei or other types of instances of images, for example, any instance segmentation with a closed structure, which is not limited in the embodiments of the present disclosure.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of an image processing method disclosed in an embodiment of the present application. The method may be executed by any electronic device, such as a terminal device, a server, or a processing platform. The embodiment of the present disclosure does not do this. limited. As shown in FIG. 1, the image processing includes the following steps.

101. Obtain N sets of instance segmentation output data, wherein the above N sets of instance segmentation output data are respectively instance segmentation output results obtained by processing images by N instance segmentation models, and the above N group instance segmentation output data have different data Structure, the above N is an integer greater than 1.

First of all, the instance segmentation problem in image processing is defined as: for an input image, each pixel should be independently judged to determine its semantic category and instance ID. For example, there are three cell nuclei 1, 2, and 3 in the image, and their semantic categories are all cell nuclei, but the results of instance segmentation are different objects.

In some possible implementations, instance segmentation can rely on Convolutional Neural Network (CNN), which mainly has the following two algorithm variants: named MaskRCNN (Mask Regions with CNN features) and simple combing full volume The target instance segmentation framework of Fully Convolutional Network (FCN). The disadvantage of MaskRCN is that there are many parameters. For specific problems, practitioners must have a high degree of professional cognition to get good results, and the method runs slowly; FCN requires special image post-processing to separate the glued multi-targets with the same semantics. Multiple instances, which also require high professional knowledge of practitioners.

In another possible implementation manner, instance segmentation may also be implemented by an instance segmentation algorithm, such as a machine learning model such as an instance segmentation algorithm based on support vector machines. The embodiments of the present disclosure do not limit the specific implementation of the instance segmentation model.

Different instance segmentation models have their own advantages and disadvantages. The embodiments of the present disclosure integrate the advantages of different single models by integrating multiple instance segmentation models.

In the embodiment of the present application, optionally, before performing step 101, different instance segmentation models may be used to separately process the image, such as using MaskRCNN and FCN to separately process the image to obtain an instance segmentation output result. Assuming that there are N instance segmentation models, the instance segmentation results of the instance segmentation model (hereinafter referred to as instance segmentation output data) can be obtained, that is, N sets of instance segmentation output data are obtained. Alternatively, the N sets of instance segmentation output data may be obtained from other devices. The embodiment of the present disclosure does not limit the manner of acquiring the N sets of instance segmentation output data.

Optionally, before processing the image using the instance segmentation model, the image can also be pre-processed, such as contrast and/or grayscale adjustment, or cropping, horizontal and vertical flip, rotation, scaling, noise removal, etc. Or any number of operations, so that the preprocessed image meets the requirements of the instance segmentation model for the input image, which is not limited in the embodiments of the present disclosure.

In the embodiment of the present disclosure, the instance segmentation output data output by the N instance segmentation models may have different data structures or meanings. For example, for the input of an image with a dimension of [height, width, 3], the instance segmentation output data includes data of [height, width], where the instance ID is 0 for the background, and different numbers greater than 0 indicate different Examples. Suppose there are 3 instance segmentation models, and different instance segmentation models correspond to different algorithms or neural network structures. Among them, the instance segmentation output data of the first instance segmentation model is a three-class probability map of [boundary, target, background]; The instance segmentation output data of the second instance segmentation model is the binary classification probability map of [boundary, background] and the dimension [target, background] binary classification graph; the instance segmentation output data of the third instance segmentation model is [central area , Target overall, background] three-class probability map, etc. Different instance segmentation models have different meanings of data output. At this time, it is impossible to use any weighted average algorithm to integrate the output of each instance segmentation model to obtain more stable and higher accuracy results. The method in the embodiment of the present application can integrate cross-instance segmentation models based on the N sets of instance segmentation output data with different data structures.

After obtaining the above N sets of instance segmentation output data, step 102 may be performed.

102. Segment the output data based on the above N sets of instances to obtain integrated semantic data and integrated central area data of the image, wherein the integrated semantic data indicates pixels located in the instance area in the image, and the integrated central area data indicates the above The pixel in the center of the instance in the image.

Specifically, the electronic device may divide the output data of the above N groups of instances for conversion processing to obtain integrated semantic data of the image and integrated central area data.

The semantic segmentation mentioned in the embodiments of the present application is a basic task in computer vision. In semantic segmentation, we need to divide the visual input into different semantically interpretable categories, that is, the classification categories are meaningful in the real world. The image is composed of many pixels, and semantic segmentation, as the name suggests, is to group the pixels according to the different semantic meanings expressed in the image (Grouping)/Segmentation. For example, we may need to distinguish all pixels that belong to a car in the image, and paint these pixels blue.

Pixel-level semantic segmentation can divide each pixel in the image into a corresponding category, that is, to achieve pixel-level classification; and the specific object of the class is an instance, then the actual instance segmentation not only needs to be pixel-level For classification, different examples need to be distinguished on the basis of specific categories. For example, there are three people A, B, and C in the image. The semantic segmentation results are all people, but the instance segmentation results are different objects.

The above-mentioned instance area can be understood as the area where the instance in the image is located, that is, the area other than the background area, and the above-mentioned integrated semantic data can indicate the pixel points in the above-mentioned image in the instance area, for example, for the processing of cell nuclear segmentation, The semantic data may include the judgment result of pixels located in the cell nucleus area.

The integrated center area data may indicate pixels located in the center area of the instance in the image.

A small area to which the instance center belongs can be defined as the instance center area, that is, the instance center area is an area within the instance area and smaller than the instance area, and the geometric center of the instance center area overlaps with the geometric center of the instance area Or adjacent, for example, the center of the instance center area is the instance center. Optionally, the central area of the instance may be a circle, an ellipse, or other shapes. The central area of the foregoing instance may be set as needed. The embodiments of the present application do not limit the specific implementation of the central area of the instance.

Specifically, based on the instance segmentation output data of each instance segmentation model in the above N instance segmentation models, the semantic data and central region data of each instance segmentation model described above can be obtained, that is, a total of N sets of semantic data and N groups of central regions The data is then integrated based on the semantic data and central region data of each instance segmentation model in the above N instance segmentation models to obtain the integrated semantic data and integrated center region data of the above image.

For the instance segmentation output data of each instance segmentation model in each of the N instance segmentation models, the instance identification information (instance ID) corresponding to each pixel in the instance segmentation model can be determined, and then based on the above The instance identification information corresponding to each pixel in the plurality of pixels in the instance segmentation model to obtain the semantic prediction value of each pixel in the above instance segmentation model. Wherein, the semantic data of the above example segmentation model includes the semantic prediction value of each pixel in the plurality of pixels of the above image.

Thresholding mentioned in the embodiments of the present application is a simple method for image segmentation. Binarization can convert a grayscale image into a binary image. You can set the grayscale of pixels greater than a critical grayscale value as the maximum grayscale value, and set the grayscale of pixels smaller than this value as grayscale The minimum value, so as to achieve binarization.

In an embodiment of the present disclosure, the binarization process may be a fixed threshold binarization process or an adaptive threshold binarization process. For example, the bimodal method, the P-parameter method, the iterative method, and the OTSU method, etc., the embodiments of the present disclosure do not limit the specific implementation of the binarization process.

In an embodiment of the present disclosure, the first image may be processed to obtain a semantic prediction result of each pixel in the plurality of pixels included in the first image. In some possible implementation manners, the semantic prediction result of the pixel point is obtained by judging the magnitude relationship between the semantic prediction value of the pixel point and the first threshold. Optionally, the first threshold of the above-mentioned binarization process may be preset or determined according to actual conditions, which is not limited in this embodiment of the present disclosure.

After obtaining the integrated semantic data and integrated center area data of the above image, step 103 may be performed.

103. Based on the integrated semantic data and integrated central area data of the above image, obtain an instance segmentation result of the above image.

In some possible implementations, at least one instance central area of the image may be obtained based on the integrated central area data of the image, and the image may be determined based on the integrated semantic data of the at least one instance central area and the image. The instance to which each pixel belongs in multiple pixels of.

The above-mentioned integrated semantic data indicates at least one pixel located in the instance area in the image. For example, the integrated semantic data may include an integrated semantic value of each pixel in a plurality of pixels of the image. The integrated semantic value is used to indicate whether the pixel is Located in the instance area, or used to indicate that the pixel is located in the instance area or the background area. The integrated center area data indicates at least one pixel located in the center area of the instance in the image, for example, the integrated center area data includes the integrated center area prediction value of each pixel in the plurality of pixels of the image, and the integrated center area prediction The value indicates whether the pixel is in the center of the instance.

Optionally, at least one pixel included in the instance area of the image can be determined through the integrated semantic data, and at least one pixel included in the center area of the image can be determined through the integrated center area data. Based on the integrated center area data and integrated semantic data of the above image, the instance to which each pixel of the plurality of pixels in the above image belongs can be determined, and the instance segmentation result of the image can be obtained.

The instance segmentation results obtained by the above method integrate the instance segmentation output results of N instance segmentation models, integrate the advantages of different instance segmentation models, and no longer require different instance segmentation models to have the same meaning of data output, and improve the instance segmentation accuracy .

In this embodiment of the present application, N sets of instance segmentation output data are obtained, wherein the above N sets of instance segmentation output data are respectively instance segmentation output results obtained by processing images by N instance segmentation models, and the above N group instance segmentation output data have Different data structures, the above N is an integer greater than 1, and then the output data is divided based on the above N sets of instances to obtain integrated semantic data and integrated central area data of the above image, wherein the integrated semantic data indicates that the image is located in the instance Pixels of the area, the integrated center area data indicates the pixels located in the center area of the instance in the image, and then based on the integrated semantic data and integrated center area data of the image, an instance segmentation result of the image is obtained. In the instance segmentation problem of processing, the advantages of implementing each instance segmentation model are complementary, and it is no longer required that each model has the same structure or meaning of data output, and a higher accuracy is obtained in the instance segmentation problem.

Please refer to FIG. 2. FIG. 2 is a schematic flowchart of another image processing method disclosed in an embodiment of the present application. FIG. 2 is obtained by further optimizing on the basis of FIG. 1. The method may be executed by any electronic device, such as a terminal device, a server, or a processing platform, etc., which is not limited in the embodiments of the present disclosure. As shown in FIG. 2, the image processing method includes the following steps.

201. Acquire N sets of instance segmentation output data, wherein the above N sets of instance segmentation output data are respectively instance segmentation output results obtained by processing images by N instance segmentation models, and the above N group instance segmentation output data have different data Structure, the above N is an integer greater than 1.

For the above step 201, reference may be made to the specific description in step 101 of the embodiment shown in FIG. 1, which will not be repeated here.

202. Based on the instance segmentation output data of the above instance segmentation model, determine that in the above instance segmentation model, at least two pixels located in the instance area in the image.

A small area to which the instance center belongs can be defined as the instance center area, that is, the instance center area is an area within the instance area and smaller than the instance area, and the geometric center of the instance center area overlaps with the geometric center of the instance area Or adjacent, for example, the center of the instance center area is the instance center. Optionally, the central area of the instance may be a circle, an ellipse, or other shapes. The central area of the foregoing instance may be set as needed. The embodiments of the present application do not limit the specific implementation of the central area of the instance. Optionally, the instance segmentation output data may include instance identification information corresponding to each pixel in at least two pixels located in the instance area in the image, for example, the instance ID is an integer greater than 0, such as 1, 2 or 3, or It may also be other numerical values. The instance identification information corresponding to the pixel located in the background area may be a preset value, or the pixel located in the background area does not correspond to any instance identification information. In this way, based on the instance identification information corresponding to each pixel in the plurality of pixels in the instance segmentation output data, at least two pixels located in the instance area in the image can be determined.

Optionally, the instance segmentation output data may not include the instance identification information corresponding to each pixel. In this case, the instance segmentation output data may be processed to obtain at least two pixels located in the instance area in the image. The disclosed embodiments do not limit this.

After determining at least two pixels located in the example area in the above image, step 203 may be performed.

203. Determine the instance center position of the above instance segmentation model based on the position information of at least two pixels located in the instance region in the instance segmentation model.

After determining at least two pixels located in the instance area in the above instance segmentation model, position information of the at least two pixels can be obtained, where, optionally, the position information can include the coordinates of the pixels in the image However, the embodiments of the present disclosure are not limited to this.

The instance center position of the instance segmentation model can be determined according to the position information of the at least two pixels. The instance center position is not limited to the geometric center position of the instance, but is the center of the instance area predicted in the instance area The location is used to further determine the center area of the instance, which can be understood as any position in the center area of the instance.

Optionally, the average value of the positions of the at least two pixels located in the instance area may be used as the instance center position of the instance segmentation model.

Specifically, the coordinates of the at least two pixels located in the instance area may be averaged as the coordinates of the instance center position of the instance segmentation model to determine the instance center position.

204. Determine the instance center area of the instance segmentation model based on the instance center position of the instance segmentation model and the position information of the at least two pixels.

Specifically, the maximum distance between the at least two pixels and the center position of the instance may be determined based on the center position of the instance of the instance segmentation model and the position information of the at least two pixels, and then the first threshold may be determined based on the maximum distance Then, a pixel point whose distance between the at least two pixel points and the center position of the instance is less than or equal to the first threshold may be determined as a pixel point in the center area of the instance.

For example, the distance between each pixel to the center of the instance (pixel distance) can be calculated based on the instance center position of the instance segmentation model and the position information of the at least two pixels. A threshold algorithm, for example, the first threshold can be set to 30% of the maximum distance in the pixel distance. After the maximum distance in the pixel distance is determined, the first threshold can be calculated and used to reserve pixels If the pixel distance is less than the first threshold, determine that these pixels are pixels in the central area of the instance, that is, determine the central area of the instance.

Optionally, the sample image may also be etched to obtain the etched sample image, and the instance center area may be determined based on the etched sample image.

The erosion operation of an image means that a certain structural element is used to detect the image, so as to find an area where the structural element can be placed inside the image. The image erosion processing mentioned in the embodiments of the present application may include the above-mentioned erosion operation. The erosion operation is a process in which structural elements are translationally filled in the eroded image. Judging from the eroded results, the foreground area of the image is reduced, and the boundary of the area becomes blurred. At the same time, some relatively small isolated foreground areas are completely eroded, and the filtering effect is achieved.

For example, for each instance mask, first use a 5×5 convolution kernel to perform image erosion on the instance mask, and then average the coordinates of multiple pixels included in the instance to obtain the center position of the instance , And determine the maximum distance of all pixels in the instance to reach the center position of the instance, and determine the pixel point whose distance from the center position of the instance is less than 30% of the maximum distance as the pixel of the center area of the instance, The central area of the instance is obtained. In this way, after the example mask in the sample image is reduced by one circle, the image binarization process is performed to obtain the binary map mask predicted by the central region.

In addition, optionally, the relative position information of the center of the pixel point, that is, the relative position information between the pixel point and the center of the instance may be obtained based on the coordinates of the pixel point included in the instance marked in the sample image and the center position of the instance For example, a vector from pixels to the center of the instance, and the relative position information is used as a supervise to train the neural network, but the embodiments of the present disclosure are not limited to this.

205. Based on the semantic data of each instance segmentation model in the above N instance segmentation models, determine the semantic voting value of each pixel in the plurality of pixels in the image.

The electronic device may, based on the semantic data of each instance segmentation model in the above N instance segmentation models, perform a semantic vote on each pixel in multiple pixels to determine the semantics of each pixel in multiple pixels in the above image The voting value, for example, using the sliding window-based voting to process the semantic data of the above instance segmentation model to determine the semantic voting value of each pixel, and then step 206 can be performed.

206. Perform a binarization process on the semantic voting value of each pixel in the plurality of pixels to obtain an integrated semantic value for each pixel in the image, wherein the integrated semantic data of the image includes the multiple The integrated semantic value of each pixel in the pixel.

The semantic voting value determined above comes from the above N instance segmentation models. Further, the semantic voting value of each pixel can be binarized to obtain the integrated semantic value of each pixel in the above image, which can be understood as The semantic masks obtained from different instance segmentation models are added to obtain an integrated semantic mask.

Specifically, the second threshold can be determined based on the number N of the multiple instance segmentation models; based on the second threshold, the semantic voting value of each pixel in the multiple pixels can be binarized to obtain the above The integrated semantic value of each pixel in the image.

The value of each pixel in the plurality of pixels has the possibility of the number of instance segmentation models. The second threshold can be determined based on the number N of the plurality of instance segmentation models. For example, the second threshold can be Rounds up the result for N/2.

The integrated semantic value of each pixel in the above image can be obtained by using the second threshold as the judgment basis for the binarization process in this step. The calculation method of the second threshold may be stored in the electronic device, for example, the predetermined pixel threshold is specified as N/2, and if N/2 is not an integer, it is rounded up. For example, for example, 4 sets of instance segmentation output data obtained by 4 instance segmentation models, then N=4, 4/2=2, the second threshold at this time is 2, compare the above semantic voting value with the above second threshold, The truncation of the semantic voting value greater than or equal to 2 is 1, and the truncation of less than 2 is 0. Thus, the integrated semantic value of each pixel in the above image is obtained, and the output data at this time may specifically be an integrated semantic binary map. The above-mentioned integrated semantic value can be understood as the result of the semantic segmentation of each pixel, and the instance to which the pixel belongs can be determined on the basis of the segmentation.

207. Perform a random walk based on the integrated semantic value of each pixel in the plurality of pixels of the image and the at least one instance center area to obtain the instance to which each pixel belongs.

Random walk (random walk), also known as random walk, random walk, etc., refers to the past performance and cannot predict the future development steps and directions. The core concept of random walk is that any conserved quantity carried by any random walker corresponds to a diffusion transport law, which is close to Brownian motion and is the ideal mathematical state of Brownian motion. The basic idea of random walk for image processing in the embodiments of the present application is to treat the image as a connected weighted undirected graph composed of fixed vertices and edges, to start a random walk from unmarked vertices, and to reach various marks for the first time The probability of the vertex represents the possibility that the unlabeled point belongs to the labeled class, and the label of the class with the highest probability is assigned to the unlabeled vertex to complete the segmentation.

Based on the integrated semantic value of each pixel in the multiple pixels of the above image and the at least one instance central area, a random walk is used to determine the distribution of pixels according to the integrated semantic value of the pixel, thereby obtaining the above The instance to which each pixel belongs, for example, the instance corresponding to the center area of the instance closest to the pixel can be determined as the instance to which the pixel belongs. In the embodiments of the present application, the final integrated semantic graph and the integrated central area graph can be obtained, and the above-mentioned connected area search and random walk (a local distribution) can be used to determine the pixel distribution of the instance to obtain the final instance segmentation result.

The instance segmentation results obtained by the above method integrate the instance segmentation output results of N instance segmentation models, and integrate the advantages of these instance segmentation models. It is no longer required that different instance segmentation models have a continuous probability map output with the same meaning, and the instance is improved. Segmentation accuracy.

The method in the embodiment of the present application is applicable to any instance segmentation problem, for example, in clinical auxiliary diagnosis. After the doctor obtains a digitally scanned image of the patient's organ tissue slice, and enters the image into the processing steps of the embodiment of the present application, a pixel mask for each independent cell nucleus can be obtained, and the doctor can use this as a basis to calculate the organ The cell density, cell morphology characteristics, and then draw medical judgment. Another example is that around the hive, the breeder can obtain the image pixel mask of each individual bee after obtaining the dense image of the bees flying around the hive, which can be used for macro bee counting and behavior mode calculation. Has great practical value.

In the specific application of the embodiments of the present application, for the bottom-up method, the UNet model may be preferably applied. UNet was first developed for semantic segmentation and effectively fuse information from multiple scales. For the top-down approach, the MaskR-CNN model can be applied. MaskR-CNN extends the faster R-CNN by adding a head to the segmentation task. In addition, the proposed MaskR-CNN can align the tracking features with the input, avoiding any quantization of bilinear interpolation. Alignment is very important for pixel-level tasks, such as instance segmentation tasks.

The network structure of the UNet model is composed of a contracting path and an expanding path. Among them, the contraction path is used to obtain context information (context), the expansion path is used for precise localization (localization), and the two paths are symmetrical to each other. The network can be trained end-to-end from very few images, and it performs better than the best method (sliding window convolution network) for segmenting cell structures such as neurons in an electron microscope. In addition, it runs very fast,

In a specific embodiment, the UNet and Mask R-CNN models can be used to perform segmentation prediction on the instances, to obtain a semantic mask for each instance segmentation model, and to integrate through pixel voting (Vote). Then, the center mask of each instance segmentation model is calculated through corrosion treatment, and the center mask is integrated. Finally, the random walk algorithm is used to obtain the instance segmentation results from the integrated semantic mask and central mask.

Cross-validation (Cross-validation) method can be used to evaluate the above results. Cross-validation is mainly used in modeling applications. In a given modeling sample, most of the samples are taken to build a model, and a small part of the sample is used for forecasting with the newly established model, and the forecast error of this small part of the sample is calculated, and the sum of their squares is recorded. In this embodiment of the present application, three-fold cross-validation can be used for evaluation. Three UJI models with AJI (5) scores of 0.605, 0.599, and 0.589 are combined with a MaskR-CNN model with an AJI (5) score of 0.565. The result obtained by the example method finally has an AJI (5) score of 0.616, which shows that the image processing method of this application has obvious advantages.

In this embodiment of the present application, N sets of instance segmentation output data are obtained, wherein the above N sets of instance segmentation output data are respectively instance segmentation output results obtained by processing images by N instance segmentation models, and the above N group instance segmentation output data have Different data structures, the above N is an integer greater than 1, based on the instance segmentation output data of the instance segmentation model, it is determined that in the instance segmentation model, at least two pixels located in the instance area in the image are segmented based on the instance segmentation Position information of at least two pixels located in the instance area of the model to determine the instance center position of the above instance segmentation model, and based on the instance center position of the instance segmentation model and the position information of the at least two pixels, the instance segmentation model is determined The central area of the instance, based on the semantic data of each instance segmentation model in the above N instance segmentation models, to determine the semantic voting value of each pixel in the multiple pixels of the above image, for each of the multiple pixels The semantic voting value of the pixels is binarized to obtain the integrated semantic value of each pixel in the above image, wherein the integrated semantic data of the above image includes the integrated semantic value of each pixel in the plurality of pixels , Based on the integrated semantic value of each pixel in the multiple pixels of the above image and the at least one instance center area, perform a random walk to obtain the instance to which each pixel belongs, which can be segmented in the instance of image processing In order to achieve the complementary advantages of each instance segmentation model, it is no longer required that each model has the same structure or meaning of data output, and a higher accuracy is obtained in the instance segmentation problem.

Please refer to FIG. 3, which is a schematic diagram of an image representation form of cell instance segmentation disclosed in an embodiment of the present application. As shown in the figure, taking cell instance segmentation as an example, the method in the embodiment of the present application can be used to process Instance segmentation results with higher accuracy. Use N types of instance segmentation models (only 4 are shown in the figure) to give example prediction masks for input pictures (different colors in the figure represent different cell instances), and convert the example prediction masks to those using semantic prediction segmentation After the semantic masking and the center region masking using the center prediction segmentation, pixel voting and integration are carried out separately, and finally the instance segmentation results are obtained. It can be seen that the right three cell missed detection method 2 in method 1 is repaired in this process This error fixes the error that the two cells in the middle of the method 2 are bonded, and also fixes the phenomenon that the bottom left corner that was not found by the four methods is actually three cells, and there is a small cell in the middle. This integration method can be integrated on any instance segmentation model, integrating the advantages of different methods. Through the above examples, the specific process and advantages of the foregoing embodiments can be more clearly understood.

The above mainly introduces the solutions of the embodiments of the present application from the perspective of the execution process on the method side. It can be understood that, in order to realize the above functions, the electronic device includes a hardware structure and/or a software module corresponding to each function. Those skilled in the art should easily realize that, in conjunction with the exemplary units and algorithm steps described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed by hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for specific applications to implement the described functions, but such implementation should not be considered beyond the scope of this application.

The embodiments of the present application may divide the functional unit of the electronic device according to the above method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or software function unit. It should be noted that the division of the units in the embodiments of the present application is schematic, and is only a division of logical functions. In actual implementation, there may be another division manner.

Please refer to FIG. 4, which is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. As shown in FIG. 4, the electronic device 400 includes: an acquisition module 410, a conversion module 420, and a segmentation module 430, wherein: the acquisition module 410 is used to acquire N sets of instance segmentation output data, wherein the N sets of instance segmentation The output data are respectively instance segmentation output results obtained by processing images by N instance segmentation models, and the N sets of instance segmentation output data have different data structures, and the N is an integer greater than 1; the conversion module 420 For segmenting output data based on the N sets of instances to obtain integrated semantic data and integrated central area data of the image, where the integrated semantic data indicates pixels located in the instance area in the image, the The integrated center area data indicates pixels located in the center area of the instance in the image; the segmentation module 430 is used to obtain an instance segmentation result of the image based on the integrated semantic data and the integrated center area data of the image .

Optionally, the conversion module 420 includes a first conversion unit 421 and a second conversion unit 422, wherein: the first conversion unit 421 is used to divide an instance of the model based on each of the N instance division models Segment the output data to obtain the semantic data and central region data of each instance segmentation model; the second conversion unit 422 is used to segment the semantic data and central region of each instance segment model based on the N instance segmentation models Data to obtain integrated semantic data of the image and integrated central area data.

Optionally, the first conversion unit 421 is specifically configured to determine, based on the instance segmentation output data of the instance segmentation model, that each pixel in the plurality of pixels of the image in the instance segmentation model corresponds to The instance identification information of; based on the instance identification information corresponding to each pixel in the plurality of pixels in the instance segmentation model, the semantic prediction value of each pixel in the instance segmentation model is obtained, wherein, The semantic data of the example segmentation model includes the semantic prediction value of each pixel in the pixels of the image.

Optionally, the first conversion unit 421 is further specifically configured to determine, based on the instance segmentation output data of the instance segmentation model, at least two pixels located in the instance region in the image in the instance segmentation model Points; based on the position information of at least two pixels located in the instance area of the instance segmentation model, determine the instance center position of the instance segmentation model; based on the instance center position of the instance segmentation model and the at least two pixels The position information of the point determines the instance center area of the instance segmentation model.

Optionally, the conversion module 420 further includes a corrosion processing unit 423 for performing corrosion processing on the instance segmentation output data of the instance segmentation model to obtain corrosion data of the instance segmentation model; the first conversion unit 421 is specifically used Therefore, based on the corrosion data of the instance segmentation model, it is determined that at least two pixels located in the instance region in the image in the instance segmentation model.

Optionally, the first conversion unit 421 is specifically configured to use the average value of the positions of the at least two pixels located in the instance area as the instance center position of the instance segmentation model.

Optionally, the first conversion unit 421 is further specifically configured to determine the at least two pixels and the instance based on the instance center position of the instance segmentation model and the position information of the at least two pixels The maximum distance of the center position; based on the maximum distance, a first threshold is determined; the pixel point of which the distance between the at least two pixels and the center position of the instance is less than or equal to the first threshold is determined as an instance Pixels in the central area.

Optionally, the conversion module 420 is specifically configured to: based on the semantic data of each instance segmentation model in the N instance segmentation models, determine the semantic vote of each pixel in the plurality of pixels of the image Value; binarize the semantic voting value of each pixel in the plurality of pixels to obtain the integrated semantic value of each pixel in the image, wherein the integrated semantic data of the image includes The integrated semantic value of each pixel in the plurality of pixels.

Optionally, the conversion module 420 is further specifically configured to: determine a second threshold based on the number N of the multiple instance segmentation models; and based on the second threshold, for each of the multiple pixels The semantic voting values of the pixels are binarized to obtain the integrated semantic value of each pixel in the image.

Optionally, the second threshold is an N/2 upward rounding result.

Optionally, the segmentation module 430 includes a central area unit 431 and a determination unit 432, wherein: the central area unit 431 is used to obtain at least one of the images based on the integrated central area data of the image An instance central area; the determining unit 432 is configured to determine an instance to which each pixel of the plurality of pixels of the image belongs based on the at least one instance central area and the integrated semantic data of the image.

Optionally, the determining unit 432 is specifically configured to perform a random walk based on the integrated semantic value of each pixel in the plurality of pixels of the image and the at least one instance central area to obtain each The instance to which the pixel belongs.

Implementing the electronic device 400 shown in FIG. 4, the electronic device 400 can obtain N sets of instance segmentation output data, wherein the above N sets of instance segmentation output data are respectively instance segmentation output results obtained by processing images by N instance segmentation models, And the above N sets of instance segmentation output data have different data structures, the above N is an integer greater than 1, and then based on the above N sets of instance segmentation output data, the integrated semantic data and integration center area data of the above image are obtained, wherein the above integration Semantic data indicates pixels located in the instance area in the above image, and the integrated center area data indicates pixels located in the center area of the instance in the image, and then the above image is obtained based on the integrated semantic data and integrated center area data of the image Image instance segmentation results, in the instance segmentation problem of image processing, the advantages of each instance segmentation model can be complemented, no longer requires each model to have the same structure or meaning of data output, and achieve higher accuracy in the instance segmentation problem .

Please refer to FIG. 5, which is a schematic structural diagram of another electronic device disclosed in an embodiment of the present application. As shown in FIG. 5, the electronic device 500 includes a processor 501 and a memory 502, wherein the electronic device 500 may further include a bus 503, the processor 501 and the memory 502 may be connected to each other through the bus 503, and the bus 503 may It is a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 503 may be divided into an address bus, a data bus, and a control bus. For ease of representation, only a thick line is used in FIG. 5, but it does not mean that there is only one bus bar or one type of bus bar. The electronic device 500 may further include an input and output device 504, and the input and output device 504 may include a display screen, such as a liquid crystal display screen. The memory 502 is used to store computer programs; the processor 501 is used to call the computer programs stored in the memory 502 to perform some or all of the method steps mentioned in the embodiments of FIG. 1 and FIG. 2 above.

Implementing the electronic device 500 shown in FIG. 5, the electronic device 500 can obtain N sets of instance segmentation output data, wherein the above N sets of instance segmentation output data are respectively instance segmentation output results obtained by processing images by N instance segmentation models, And the above N sets of instance segmentation output data have different data structures, the above N is an integer greater than 1, and then based on the above N sets of instance segmentation output data, the integrated semantic data and integration center area data of the above image are obtained, wherein the above integration Semantic data indicates pixels located in the instance area in the above image, and the integrated center area data indicates pixels located in the center area of the instance in the image. Based on the integrated semantic data and integrated center area data in the image, the above image is obtained The result of image segmentation can achieve the complementary advantages of each instance segmentation model in the instance segmentation problem of image processing. It no longer requires each model to have the same structure or meaning of data output, and achieve higher accuracy in the instance segmentation problem. .

Embodiments of the present application also provide a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program causes the computer to execute any part of the image processing method described in the above method embodiment or All steps.

It should be noted that, for the sake of simple description, the foregoing method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited by the described action sequence, Because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

In the above embodiments, the description of each embodiment has its own emphasis. For a part that is not detailed in an embodiment, you can refer to related descriptions in other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed device may be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or integrated To another system, or some features can be ignored, or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical or other forms.

The unit (module) described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple networks On the road unit. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software function unit.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product in essence or part of the contribution to the existing technology or all or part of the technical solution, and the computer software product is stored in a memory, Several instructions are included to enable a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (Random Access Memory, RAM), mobile hard disk, magnetic disk or optical disk, etc. that can store program code medium.

Persons of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing relevant hardware through a program. The program may be stored in a computer-readable memory, and the memory may include: Pen drive, read-only memory, random access device, magnetic disk or optical disc, etc.

The embodiments of the present application are described in detail above, and specific examples are used to explain the principles and implementation of the present application. The descriptions of the above embodiments are only used to help understand the method and core idea of the present application; Those of ordinary skill in the art, according to the ideas of the present application, may have changes in specific implementations and application scopes. In summary, the content of this specification should not be construed as limiting the present application.

101, 102, 103, 201, 202, 203, 204, 205, 206, 207: steps 400, 500: electronic equipment 410: Get module 420: Conversion module 421: First conversion unit 422: Second conversion unit 423: Corrosion treatment unit 430: Segmentation module 431: Central area unit 432: determination unit 501: processor 502: Memory 503: busbar 504: input and output devices

In order to more clearly explain the technical solutions in the embodiments or the prior art of the present application, the drawings required in the description of the embodiments or the prior art will be briefly introduced below. FIG. 1 is a schematic flowchart of an image processing method disclosed in an embodiment of the present application. 2 is a schematic flowchart of another image processing method disclosed in an embodiment of the present application. FIG. 3 is a schematic diagram of an image representation form of a cell segmentation example disclosed in the embodiments of the present application. 4 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application. 5 is a schematic structural diagram of another electronic device disclosed in an embodiment of the present application.

101, 102, 103: steps

Claims (10)

An image processing method, including: Obtaining N sets of instance segmentation output data, wherein the N sets of instance segmentation output data are respectively instance segmentation output results obtained by processing images by N instance segmentation models, and the N groups of instance segmentation output data have different data Structure, the N is an integer greater than 1; The output data is segmented based on the N sets of instances to obtain integrated semantic data and integrated central area data of the image, wherein the integrated semantic data indicates pixels located in the instance area in the image, and the integrated central area The data indicates pixels in the central area of the instance in the image; and Based on the integrated semantic data of the image and the integrated central area data, an instance segmentation result of the image is obtained. According to the image processing method described in item 1 of the patent application scope, the output data is segmented based on the N sets of instances to obtain integrated semantic data and integrated central area data of the image, including: For each instance segmentation model in the N instance segmentation models, based on the instance segmentation output data of the instance segmentation model, obtain the semantic data and central region data of the instance segmentation model; and Based on the semantic data and central region data of each instance segmentation model in the N instance segmentation models, the integrated semantic data and integrated center region data of the image are obtained. According to the image processing method described in item 2 of the patent application scope, based on the instance segmentation output data of the instance segmentation model, the semantic data and central region data of the instance segmentation model are obtained, including: Based on the instance segmentation output data of the instance segmentation model, determining instance identification information corresponding to each pixel of the plurality of pixels of the image in the instance segmentation model; and Obtaining the semantic prediction value of each pixel in the instance segmentation model based on the instance identification information corresponding to each pixel in the plurality of pixels in the instance segmentation model, wherein the instance segmentation model The semantic data of includes the semantic prediction value of each pixel in the pixels of the image. According to the image processing method described in item 2 or 3 of the patent application scope, based on the instance segmentation output data of the instance segmentation model, the semantic data and the central area data of the instance segmentation model are obtained, which also includes: Based on the instance segmentation output data of the instance segmentation model, determining that in the instance segmentation model, at least two pixels located in the instance region in the image; Determine the instance center position of the instance segmentation model based on the position information of at least two pixels located in the instance region in the instance segmentation model; and Based on the instance center position of the instance segmentation model and the position information of the at least two pixels, the instance center region of the instance segmentation model is determined. The image processing method as described in item 4 of the patent application scope, Before the instance segmentation output data based on the instance segmentation model determines that in the instance segmentation model, the image is located before at least two pixels in the instance region, the image processing method further includes: The instance segmentation output data of the instance segmentation model is subjected to corrosion processing to obtain corrosion data of the instance segmentation model; and Based on the instance segmentation output data of the instance segmentation model, determining that in the instance segmentation model, at least two pixels located in the instance region in the image include: based on the corrosion data of the instance segmentation model, determining In the instance segmentation model, at least two pixels located in the instance area in the image. According to the image processing method described in item 4 or 5 of the patent application, based on the position information of at least two pixels located in the instance area in the instance segmentation model, determining the instance center position of the instance segmentation model, include: The average value of the positions of the at least two pixels located in the instance area is taken as the instance center position of the instance segmentation model. The image processing method according to any one of items 4 to 6 of the patent application scope, based on the instance center position of the instance segmentation model and the position information of the at least two pixels, the determination of the instance segmentation model Examples of central areas, including: Determine the maximum distance between the at least two pixels and the instance center position based on the instance center position of the instance segmentation model and the position information of the at least two pixels; Determine the first threshold based on the maximum distance; and A pixel point in which the distance between the at least two pixel points and the center position of the instance is less than or equal to the first threshold is determined as a pixel point in the center area of the instance. An electronic device, including: An acquisition module for acquiring N sets of instance segmentation output data, wherein the N sets of instance segmentation output data are respectively instance segmentation output results obtained by processing images by N instance segmentation models, and the N group instance segmentation output The data has different data structures, and the N is an integer greater than 1; A conversion module, configured to segment output data based on the N sets of instances to obtain integrated semantic data and integrated central area data of the image, wherein the integrated semantic data indicates pixels located in the instance area in the image, The integrated central area data indicates pixels in the central area of the instance in the image; and A segmentation module is used to obtain an instance segmentation result of the image based on the integrated semantic data of the image and integrated central area data. An electronic device, including a processor and a memory, the memory is used to store a computer program, the computer program is configured to be executed by the processor, the processor is used to execute the patent application scope of items 1 to 7 The image processing method according to any one of the above. A computer-readable storage medium for storing a computer program, wherein the computer program causes the computer to execute the image processing method according to any one of items 1 to 7 of the patent application scope.

Images