CN114648762A - Semantic segmentation method and device, electronic equipment and computer-readable storage medium - Google Patents

Abstract

The embodiments of the present application disclose a semantic segmentation method, an apparatus, an electronic device, and a computer-readable storage medium; in the embodiments of the present application, an image to be segmented is acquired, and the to-be-segmented image is divided to obtain the to-be-segmented image. the initial area image corresponding to the image; determine the target pixel context information in the initial area image, and determine the target area image corresponding to the initial area image according to the target pixel context information and the initial area image; determine the target The target area context information between the area images, and according to the target area image and the target area context information, determine the target pixel feature of the to-be-segmented image; Split to get the split result. The embodiments of the present application can reduce computational complexity. The embodiments of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and automatic driving.

Description

Semantic segmentation method, apparatus, electronic device, and computer-readable storage medium

technical field

The present application relates to the technical field of image processing, and in particular, to a semantic segmentation method, an apparatus, an electronic device, and a computer-readable storage medium.

Background technique

Semantic segmentation is to classify each pixel in the image, and the pixels belonging to the same class are divided into the same area, so as to realize the segmentation of the image.

In order to improve the image segmentation accuracy, an attention mechanism is added to the semantic segmentation to calculate the image context information, and the computational complexity of this method is high.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a semantic segmentation method, apparatus, electronic device, and computer-readable storage medium, which can solve the technical problem of high computational complexity in semantic segmentation.

A semantic segmentation method including:

Obtaining an image to be divided, and dividing the above-mentioned image to be divided, to obtain an initial area image corresponding to the above-mentioned to-be-divided image;

Determine the target pixel context information in the above-mentioned initial area image, and determine the target area image corresponding to the above-mentioned initial area image according to the above-mentioned target pixel context information and the above-mentioned initial area image;

Determine the target area context information between the above-mentioned target area images, and determine the target pixel feature of the above-mentioned to-be-segmented image according to the above-mentioned target area image and the above-mentioned target area context information;

According to the feature of the target pixel, the image to be segmented is segmented to obtain a segmentation result.

Correspondingly, an embodiment of the present application provides a semantic segmentation device, including:

an image acquisition module, configured to acquire an image to be segmented, and to divide the image to be segmented to obtain an initial region image corresponding to the image to be segmented;

a first determining module, configured to determine the target pixel context information in the above-mentioned initial area image, and determine the target area image corresponding to the above-mentioned initial area image according to the above-mentioned target pixel context information and the above-mentioned initial area image;

a second determination module, configured to determine the target area context information between the above-mentioned target area images, and determine the target pixel feature of the above-mentioned to-be-segmented image according to the above-mentioned target area image and the above-mentioned target area context information;

The image segmentation module is configured to segment the image to be segmented according to the feature of the target pixel to obtain a segmentation result.

Optionally, the image acquisition module is specifically configured to execute:

Perform feature extraction on the above-mentioned to-be-segmented image to obtain a feature image;

Acquiring a preset grid, and performing displacement prediction on the vertices of the preset grid according to the feature image and the preset grid to obtain a target deformation grid;

The feature image is divided according to the target deformation grid to obtain the initial region image corresponding to the image to be divided.

Optionally, the image acquisition module is specifically configured to execute:

Determine the vertex coordinates in the above preset grid;

According to the above-mentioned vertex coordinates, perform a vertex feature search in the above-mentioned feature image, and obtain the initial feature corresponding to the above-mentioned vertex coordinates;

According to the above-mentioned initial feature and the context information of the above-mentioned vertex coordinate, determine the target characteristic corresponding to the above-mentioned vertex coordinate;

The target displacement of the vertex coordinates is predicted according to the target feature, and the vertex coordinates are moved according to the target displacement to obtain a target deformation mesh.

Optionally, the second determining module is specifically configured to execute:

Determine the initial area feature corresponding to the target area image according to the average value of the pixels of the target area image;

According to the above-mentioned initial area feature, determine the target area context information between the above-mentioned target area images;

Determine the target area feature according to the above-mentioned initial area characteristic and the above-mentioned target area context information;

The above-mentioned target area feature is mapped to the above-mentioned target deformation grid to obtain the above-mentioned target pixel characteristic of the to-be-segmented image.

Optionally, the above-mentioned semantic segmentation device further includes:

Training module for:

Obtain a training sample set, and determine the target weight corresponding to the category of the training pixel according to the number of training pixels in the training image of the training sample set;

Divide the above training images to obtain training area images;

Determine the initial pixel context information in the above-mentioned training area image through the pixel context layer in the neural network model to be trained, and determine the first area image corresponding to the above-mentioned training image according to the above-mentioned initial pixel context information and the above-mentioned training area image;

Determine the initial area context information between the above-mentioned first area images through the area context layer in the above-mentioned neural network model to be trained, and determine the initial pixel features of the above-mentioned training images according to the above-mentioned first area images and the above-mentioned initial area context information;

Determine the target category of the training pixel corresponding to the above-mentioned initial pixel feature through the segmentation layer in the neural network model to be trained;

According to the above target category, the label of the above training pixel and the above target weight, determine the target loss value;

Obtain the training times of the above-mentioned neural network model to be trained;

Based on the above-mentioned target loss value and the above-mentioned training times, the above-mentioned neural network model to be trained is trained to obtain the above-mentioned trained neural network model.

Optionally, the training module is specifically used to execute:

Through the feature extraction layer in the neural network model to be trained, feature extraction is performed on the training image to obtain a training feature image;

Through the deformation network layer in the neural network model to be trained, the displacement prediction is performed on the vertices of the preset grid according to the training feature image and the preset grid to obtain an initial deformation grid;

Through the above-mentioned pixel context layer in the above-mentioned neural network model to be trained, the above-mentioned training feature image is divided according to the above-mentioned initial deformation grid to obtain a training area image;

According to the above-mentioned target category, the label of the above-mentioned training pixel and the above-mentioned target weight, determine the first target loss value;

Determine the second target loss value according to the above-mentioned initial pixel features and the mean value of the above-mentioned initial pixel features;

Determine the target loss value according to the first target loss value and the second target loss value;

If the above-mentioned target loss value does not meet the preset conditions and/or the training times of the above-mentioned neural network model to be trained is less than the preset threshold, then the above-mentioned training times are increased by 1, and the above-mentioned neural network to be trained is updated according to the above-mentioned first target loss value. In the model, the network parameters of the above-mentioned feature extraction layer, the network parameters of the above-mentioned pixel context layer, the network parameters of the above-mentioned area context layer, and the network parameters of the above-mentioned segmentation layer are updated according to the above-mentioned second target loss value. Deform the network parameters of the network layer, and return to execute the feature extraction layer in the neural network model to be trained, and perform feature extraction on the training image to obtain a training feature image;

If the target loss value satisfies the preset condition and/or the training times of the neural network model to be trained is equal to the preset threshold, the training is stopped, and the trained neural network model is obtained.

Optionally, the training module is specifically used to execute:

Determine the sub-area of each grid in the above-mentioned initial deformation grid and the total area of the above-mentioned to-be-segmented image;

Determine the third target loss value according to the above-mentioned sub-areas and the above-mentioned total area;

Determine the target loss value according to the first target loss value, the second target loss value, and the third target loss value;

If the above-mentioned target loss value does not meet the preset conditions and/or the training times of the above-mentioned neural network model to be trained is less than the preset threshold, then the above-mentioned training times are increased by 1, and the above-mentioned neural network to be trained is updated according to the above-mentioned first target loss value. In the model, the network parameters of the feature extraction layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer are updated according to the second target loss value and the third target loss value. The network parameters of the above-mentioned deformation network layer in the trained neural network model are returned to execute the feature extraction layer in the above-mentioned neural network model to be trained, and feature extraction is performed on the above-mentioned training image to obtain a training characteristic image.

In addition, an embodiment of the present application further provides an electronic device, including a processor and a memory, the memory stores a computer program, and the processor is configured to run the computer program in the memory to implement the semantic segmentation method provided by the embodiment of the present application.

In addition, the embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and the computer program is adapted to be loaded by a processor to execute any semantics provided by the embodiments of the present application segmentation method.

In addition, the embodiments of the present application further provide a computer program product, including a computer program, which implements any one of the semantic segmentation methods provided by the embodiments of the present application when the computer program is executed by a processor.

In the embodiment of the present application, the image to be segmented is obtained first, and the image to be segmented is divided to obtain an initial region image corresponding to the image to be segmented. Then, the target pixel context information in the initial area image is determined, and the target area image corresponding to the initial area image is determined according to the target pixel context information and the initial area image. Next, the target area context information between the target area images is determined, and the target pixel features of the to-be-segmented image are determined according to the target area images and the target area context information. Finally, according to the target pixel features, the image to be segmented is identified and segmented.

That is, in the embodiment of the present application, the target pixel context information in the initial area image is first determined, and the target area image corresponding to the initial area image is determined according to the target pixel context information and the initial area image, and then the target area between the target area images is determined. Context information, and according to the target area image and the target area context information, determine the target pixel characteristics of the image to be segmented, that is, by determining the target area context information between the two target area images, it is unnecessary to calculate the relationship between each pixel in the target area image and the target area image. The context information between all the pixels in the other target area images makes it unnecessary to calculate the context information between each pixel in the to-be-segmented image and all other pixels in the to-be-segmented image, thereby reducing the computational complexity.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

Fig. 1 is a scene schematic diagram of a semantic segmentation process provided by an embodiment of the present application;

2 is a schematic flowchart of a semantic segmentation method provided by an embodiment of the present application;

3 is a schematic diagram of an initial area image provided by an embodiment of the present application;

4 is a schematic diagram of a preset grid and a target deformation grid provided by an embodiment of the present application;

5 is a schematic flowchart of a training method for a neural network model to be trained provided by an embodiment of the present application;

6 is a schematic structural diagram of a neural network model to be trained provided by an embodiment of the present application;

7 is a schematic flowchart of a method for applying a trained neural network model provided by an embodiment of the present application;

8 is a schematic diagram of a segmentation result of a trained neural network model provided by an embodiment of the present application;

9 is a schematic structural diagram of a semantic segmentation device provided by an embodiment of the present application;

FIG. 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

Embodiments of the present application provide a semantic segmentation method, apparatus, electronic device, and computer-readable storage medium. Wherein, the semantic segmentation device may be integrated in an electronic device, and the electronic device may be a server or a terminal or other device.

The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, and cloud communications. , middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.

Moreover, a plurality of servers can be formed into a blockchain, and the servers are nodes on the blockchain.

The terminal may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected through wired or wireless communication, which is not limited in this application.

For example, as shown in FIG. 1, the terminal can obtain the image to be divided, and divide the image to be divided to obtain the initial area image corresponding to the image to be divided; The initial area image determines the target area image corresponding to the initial area image; determines the target area context information between the target area images, and determines the target pixel feature of the image to be segmented according to the target area image and the target area context information; According to the target pixel feature, Segment the image to be segmented to obtain the segmentation result.

In addition, the "plurality" in the embodiments of the present application refers to two or more. In the embodiments of the present application, "first" and "second" etc. are used to distinguish descriptions, and should not be understood as implying relative importance.

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning.

Computer Vision Technology (Computer Vision, CV) Computer vision is a science that studies how to make machines "see". Further, it refers to the use of cameras and computers instead of human eyes to recognize and measure objects and other machine vision, and further. Do graphics processing to make computer processing become images more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, trying to build artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping It also includes common biometric identification technologies such as face recognition and fingerprint recognition.

Machine Learning (ML) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or realize human learning behaviors to acquire new knowledge or skills, and to reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications are in all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, teaching learning and other techniques.

The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, automatic driving, and remote sensing. Autonomous driving usually includes technologies such as high-precision maps, environmental perception, behavioral decision-making, path planning, and motion control.

Each of them will be described in detail below. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

Please refer to FIG. 2 , which is a schematic flowchart of a semantic segmentation method provided by an embodiment of the present application. The semantic segmentation method may include:

S201. Acquire an image to be segmented, and divide the image to be segmented to obtain an initial region image corresponding to the image to be segmented.

An image to be segmented refers to an image that includes an area of interest for the user. For example, image A includes vehicles, trees, and the sky. If the user wants to obtain an image of the vehicle in image A, then image A is an image to be segmented.

The image to be segmented may be an image of an automatic driving scene or a remote sensing image, and the type of the image to be segmented is not limited in this embodiment.

The terminal may acquire the image to be segmented by shooting through its own camera when receiving the acquiring instruction. Alternatively, the terminal may acquire the image to be segmented locally when receiving the acquiring instruction.

Alternatively, the terminal may also forward the acquisition instruction to other terminals when receiving the acquisition instruction, and the other terminal acquires the to-be-segmented image locally based on the acquisition instruction, or shoots based on the acquisition instruction, thereby acquiring the to-be-segmented image. Then the other terminal sends the image to be divided to the terminal, and the terminal obtains the image to be divided.

As for the manner in which the terminal acquires the image to be segmented, the user may select according to the actual situation, which is not limited in this embodiment.

After acquiring the to-be-segmented image, the terminal may divide the to-be-segmented image based on a preset grid to obtain an initial region image corresponding to the to-be-segmented image.

Divide refers to dividing the image into various regional images, but there is still a connection relationship between the regional images. For example, if the image to be segmented is shown in Figure 3, the initial area image may be as shown in Figure 3, and the target refers to the thing in the image to be segmented, for example, the target may be a vehicle, a person, or a tree.

Since the position of the target in each image to be divided is different, if the image to be divided is directly divided based on the preset grid, the obtained initial area image is not accurate, that is, at this time , the edge of the preset grid does not fit the boundary of the target in the image to be segmented. Therefore, in order to improve the accuracy of the initial area image, in some embodiments, the to-be-segmented image is divided to obtain the initial area image corresponding to the to-be-segmented image, including:

Perform feature extraction on the image to be segmented to obtain a feature image;

Obtain a preset mesh, and perform displacement prediction on the vertices of the preset mesh according to the feature image and the preset mesh to obtain the target deformation mesh;

The feature image is divided according to the target deformation grid, and the initial region image corresponding to the image to be divided is obtained.

In this embodiment, the grid may be initialized to obtain a preset grid. Among them, the grid can be initialized according to the following three aspects:

In the first aspect, the mesh topology remains unchanged before and after deformation, that is, the topology of the preset mesh is the same as that of the target deformed mesh.

In the second aspect, the edges of the preset mesh are flexible and diverse, so that the target deformation mesh can be obtained after the preset mesh has as little displacement deviation as possible.

In the third aspect, the number of grids of the preset grid is constant, which facilitates subsequent structured batch processing of images.

The grid is initialized according to the above three aspects, and the obtained preset grid can be as shown in FIG. 4 .

After obtaining the preset grid, the terminal can displace the vertices of the preset grid according to the feature image to obtain the target deformable grid, so that the edges of the target deformable grid fit the boundary of the target in the feature image, that is, the feature image The same target is in the same grid in the target deformation grid, that is, the target color homogeneity (same color) or semantic homogeneity (belonging to the same category) in the same grid in the target deformation grid.

For example, after displacing the vertices of the preset mesh in FIG. 4 , the obtained target deformation mesh may be as shown in FIG. 4 .

After obtaining the target deformation grid, the terminal divides the feature image according to the target deformation grid to obtain an initial region image corresponding to the image to be divided.

In this embodiment, after feature extraction is performed on the image to be segmented to obtain the feature image, displacement prediction is performed on the vertices of the preset grid according to the feature image and the preset grid to obtain the target deformation grid, and then according to the target deformation grid The feature image is divided to obtain the initial area image corresponding to the image to be divided, so that the grid edge of the target deformation grid matches the boundary of the target in the image.

In some embodiments, displacement prediction is performed on the vertices of the preset mesh according to the feature image and the preset mesh to obtain the target deformation mesh, including:

Determine the vertex coordinates in the preset mesh;

According to the vertex coordinates, perform vertex feature search in the feature image to obtain the initial features corresponding to the vertex coordinates;

Determine the target feature corresponding to the vertex coordinate according to the initial feature and the context information of the vertex coordinate;

The target displacement of the vertex coordinates is predicted according to the target features, and the vertex coordinates are moved according to the target displacement to obtain the target deformation mesh.

In the object detection of images, the offset of the position of the center point, the upper left corner vertex or the lower right corner vertex of the detection frame is predicted by the fully connected layer or the convolution layer. However, the vertices or center points of different detection boxes on the same image do not interfere with each other in object detection. Semantic segmentation is a dense prediction task, that is, the vertices of each grid are not isolated from each other, but are connected to each other through the edges of the grid, which are a set of points with context dependencies. Therefore, in this embodiment, after the initial feature corresponding to the vertex is obtained, the target feature corresponding to the vertex coordinate is determined according to the initial feature and the context information of the vertex, so that the target displacement can be obtained more accurately, and thus the target displacement can be obtained more accurately The target deformable mesh.

Among them, according to the initial features and the context information of the vertices, the target features corresponding to the vertex coordinates are determined, including:

Perform point-by-point feature transformation on the initial features to obtain intermediate features, and then determine the target features according to the context information of the intermediate features and vertices.

It should be noted that the context information of the vertices can be obtained through the self-attention mechanism layer of the deformation network layer in the trained neural network, and the point-by-point feature transformation of the initial features can be obtained through the deformation network layer in the trained neural network. The point convolution layer is implemented, and then the target feature corresponding to the vertex coordinate is determined according to the initial feature and the context information of the vertex through the prediction convolution layer of the deformation network layer in the trained neural network.

Moreover, the deformation network layer can include multiple point-by-point convolution layers and self-attention mechanism layers, so as to realize multiple feature transformations on the initial features, and obtain the context information of the vertices multiple times. For example, the deformation network layer can include 6 point-wise convolutional layers and self-attention mechanism layers.

In order to obtain the intermediate features more accurately, after the initial features are obtained, the vertex coordinates can also be fused with the initial features (for example, the vertex coordinates are concatenated with the initial features) to obtain candidate features, and then the candidate features can be transformed point by point. , to get the intermediate features.

Among them, the vertex coordinates and the initial features can be fused through the CoordConv layer of the deformation network layer in the trained neural network.

In other embodiments, performing feature extraction on the image to be segmented to obtain the feature image includes: performing feature extraction on the image to be segmented through the feature extraction layer in the trained neural network model to obtain the feature image.

The feature extraction layer in the trained neural network model may be a residual neural network (ResNet) or a convolutional network. Optionally, when the feature extraction layer is a residual neural network, in order to reduce the loss of resolution and expand the receptive field, part of the pooling layer in the residual neural network can be replaced with an extended convolution layer, that is, at this time, the feature The extraction layer is a residual neural network containing dilated convolutional layers.

S202. Determine the target pixel context information in the initial area image, and determine the target area image corresponding to the initial area image according to the target pixel context information and the initial area image.

After obtaining the initial area image, the terminal obtains the target pixel context information between the pixels in the initial area image, and then determines the target area image corresponding to the initial area image according to the target pixel context information and the initial area image.

For example, if there are pixel 1, pixel 2, and pixel 3 in the initial area image a, calculate the target pixel context information between pixel 1 and pixel 2, calculate the target pixel context information between pixel 1 and pixel 3, and calculate the target pixel context information between pixel 2 and pixel 2. Target pixel context information between pixels 3.

Among them, the target pixel context information in the initial area image can be determined through the pixel context layer in the trained neural network model, and the target area image corresponding to the initial area image can be determined according to the target pixel context information and the initial area image, and the pixel context layer can be Self-attention mechanism layer.

Alternatively, the Euclidean distance algorithm or the cosine similarity algorithm may be used to determine the target pixel context information in the initial area image, and the target area image corresponding to the initial area image may be determined according to the target pixel context information and the initial area image.

Context information refers to the interaction information between different objects. Pixel context information refers to the interaction information between different pixels in the same image.

S203. Determine the target area context information between the target area images, and determine the target pixel feature of the to-be-segmented image according to the target area image and the target area context information.

In the related art, through the self-attention mechanism layer, not only the target pixel context information between the pixels in the initial area image is calculated, but also the context information between the pixels in different initial area images needs to be calculated, resulting in high computational complexity.

For example, if there are pixel 1 and pixel 2 in the initial area image a, and pixel 4 and pixel 5 in the initial area image b, then through the self-attention mechanism layer, not only the target pixel context information between pixel 1 and pixel 2 needs to be calculated, Calculate the target pixel context information between pixel 4 and pixel 5, and calculate the target pixel context information between pixel 1 and pixel 4, calculate the target pixel context information between pixel 1 and pixel 5, calculate pixel 2 and pixel 4 The target pixel context information between and calculate the target pixel context information between pixel 2 and pixel 5.

In this embodiment, after obtaining the target pixel context between the pixels in the initial area image, the target area context information between the target area images is calculated, and then according to the target area image and the target area context information, the image to be segmented is determined. The target pixel feature does not need to calculate the pixel context information between the pixels of different target area images, thereby reducing the computational complexity, so as to achieve higher semantic segmentation on the basis of lower computational complexity and computational redundancy precision.

For example, pixel 1 and pixel 2 exist in initial area image a, pixel 4 and pixel 5 exist in initial area image b, pixel 6 and pixel 7 exist in target area image a1, and pixel 8 and pixel 9 exist in target area image b1, Then in this embodiment, the area context information between the target area image a1 and the target area image b1 is calculated, so that it is not necessary to calculate the target pixel context information between the pixel 1 and the pixel 4, and the Target pixel context information, no need to calculate the target pixel context information between pixel 2 and pixel 4, and no need to calculate the target pixel context information between pixel 2 and pixel 5.

The target area context information between the target area images can be determined through the area context layer in the trained neural network model, and the target pixel feature of the image to be segmented can be determined according to the target area image and the target area context information.

Alternatively, the Euclidean distance algorithm or the cosine similarity algorithm can be used to determine the target area context information between the target area images, and the target pixel features of the to-be-segmented image can be determined according to the target area images and the target area context information.

In some embodiments, the target area context information between the target area images is determined, and according to the target area images and the target area context information, the target pixel feature of the image to be segmented is determined, including:

According to the mean value of the pixels of the target area image, determine the initial area feature corresponding to the target area image;

Determine the target area context information between the target area images according to the initial area features;

Determine the target area feature according to the initial area feature and the target area context information;

The target area features are mapped to the target deformation grid to obtain the target pixel features of the image to be segmented.

In this embodiment, the average value of the pixels of the target area image is used as the initial area feature corresponding to the target area image, and then the target area context information between the target area images is calculated according to the initial area feature corresponding to the target area image, so that the two The target area context information between each target area image only needs to be calculated once, and does not need to be calculated multiple times, thereby greatly reducing the computational complexity.

When the target area context information between the target area images is determined through the area context layer in the trained neural network model, and the target pixel feature of the image to be segmented is determined according to the target area image and the target area context information, the area context layer can be used to determine the target pixel features of the image to be segmented. The middle region pooling layer determines the initial region feature corresponding to the target region image according to the average value of the pixels of the target region image. Through the region-level self-attention mechanism layer in the region context layer, the target region context information between the target region images can be determined according to the initial region feature, and the target region feature can be determined according to the initial region feature and the target region context information. The target region features can be mapped to the target deformation grid through the region de-pooling layer in the region context layer to obtain the target pixel features of the image to be segmented.

S204 , segment the image to be segmented according to the target pixel feature to obtain a segmentation result.

After obtaining the feature of the target pixel, the terminal further divides the image to be segmented according to the feature of the target pixel to obtain a segmentation result.

In some embodiments, segmenting the image to be segmented according to the feature of the target pixel to obtain the segmentation result includes: segmenting the image to be segmented according to the feature of the target pixel through the segmentation layer in the trained neural network model to obtain the segmentation result.

Optionally, since the target area image contains detailed information, and the feature image includes all information, in order to make the segmentation more accurate, you can use the segmentation layer in the trained neural network model, according to the feature image, the target area image and the target pixel features, Segment the image to be segmented to obtain the segmentation result.

In other embodiments, when the target pixel context information and the target area image are determined by the pixel context layer in the trained neural network model, the target area context information and the target pixel feature are determined by the area context layer in the trained neural network model, When the segmentation result is determined by the segmentation layer in the trained neural network model, the method further includes:

Obtain the training sample set, and determine the target weight corresponding to the category of the training pixel according to the number of training pixels in the training image of the training sample set;

Divide the training image to obtain the training area image;

Determine the initial pixel context information in the training area image through the pixel context layer in the neural network model to be trained, and determine the first area image corresponding to the training image according to the initial pixel context information and the training area image;

Determine the initial area context information between the first area images through the area context layer in the neural network model to be trained, and determine the initial pixel feature of the training image according to the first area image and the initial area context information;

Determine the target category of the training pixels corresponding to the initial pixel features through the segmentation layer in the neural network model to be trained;

Determine the target loss value according to the target category, the label of the training pixel and the target weight;

Obtain the training times of the neural network model to be trained;

Based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain the trained neural network model.

Among them, the number of training pixels in the training images of the training sample set can be substituted into the following formula to obtain the target weights corresponding to the categories of the training pixels:

Among them, _i represents the ith category of training pixels, ni represents the number of pixels of the ith category, z represents the number of categories, q _i represents the frequency corresponding to the pixel of the ith category, and q _m represents the median of the frequency , w _i represents the target weight corresponding to the pixel of the i-th category.

Substitute the target category, the label of the training pixel, and the target weight into the following formula to get the target loss value:

where _yi represents the label of the training pixel, _pi represents the target class, and _Lc represents the target loss value.

Since the number of pixels of each category in the training sample set is quite different, the weighted cross-entropy loss function is used to calculate the target loss value, and the target weight is determined according to the number of training pixels, so as to avoid overfitting and make the obtained The trained neural network model is more accurate for semantic segmentation.

Optionally, determine the initial area context information between the first area images through the area context layer in the neural network model to be trained, and determine the initial pixel features of the training image according to the first area image and the initial area context information, including: :

Determine the first area feature corresponding to the first area image according to the average value of the pixels of the first area image;

According to the first region feature, determine the initial region context information between the first region images, and obtain the second region feature according to the initial region context information and the first region feature;

The second region feature is mapped to the initial deformation grid to obtain the initial pixel feature of the training image.

Optionally, based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain a trained neural network model, including:

If the number of training is less than the preset threshold, increase the number of training by 1, update the network parameters of the neural network model to be trained according to the target loss value, and then return to execute the division of the training image to obtain the training area image. If the number of training times is equal to the preset threshold, the neural network model to be trained is used as the trained neural network model.

Or, based on the target loss value and the number of training times, train the neural network model to be trained to obtain the trained neural network model, including:

If the target loss value does not meet the preset conditions, update the network parameters of the neural network model to be trained according to the target loss value, and then return to execute the division of the training image to obtain the training area image. If the target loss value satisfies the preset condition, the neural network model to be trained is used as the trained neural network model.

Or, based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain the trained neural network model, including:

If the target loss value does not meet the preset conditions and the training times is less than the preset threshold, increase the training times by 1, update the network parameters of the neural network model to be trained according to the target loss value, and then return to execute to divide the training images to obtain training area image;

If the target loss value satisfies the preset condition and the number of training times is equal to the preset threshold, the neural network model to be trained is used as the trained neural network model.

When the target deformation mesh is obtained through the deformation network layer in the trained neural network model, the training image is divided to obtain the training area image, including:

Through the feature extraction layer in the neural network model to be trained, feature extraction is performed on the training image to obtain the training feature image;

Through the deformation network layer in the neural network model to be trained, the displacement prediction is performed on the vertices of the preset mesh according to the training feature image and the preset mesh, and the initial deformation mesh is obtained;

Through the pixel context layer in the neural network model to be trained, the training feature image is divided according to the initial deformation grid, and the training area image is obtained;

According to the target category, the label of the training pixel and the target weight, determine the target loss value, including:

Determine the first target loss value according to the target category, the label of the training pixel and the target weight;

Determine the second target loss value according to the initial pixel feature and the mean value of the initial pixel feature;

Determine the target loss value according to the first target loss value and the second target loss value;

Based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain the trained neural network model, including:

If the target loss value does not meet the preset conditions and/or the training times of the neural network model to be trained is less than the preset threshold, the training times are increased by 1, and the feature extraction layer in the neural network model to be trained is updated according to the first target loss value The network parameters of the pixel context layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer, update the network parameters of the deformation network layer in the neural network model to be trained according to the second target loss value, and return to execute The feature extraction layer in the trained neural network model performs feature extraction on the training image to obtain the training feature image;

If the target loss value satisfies the preset condition and/or the training times of the neural network model to be trained is equal to the preset threshold, the training is stopped, and the trained neural network model is obtained.

Because the positions of more boundary points or mutation points in the image may be the positions of the vertices in the preset grid after displacement, that is, more boundary points or mutation points in the image may be the same as the vertices in the preset grid after displacement. Coincidence, so the accuracy of obtaining the target displacement of the vertices in the preset mesh is low. Moreover, optimizing the positions of the vertices in the preset mesh alone ignores the topological relationship between the vertices in the preset mesh and the overall distribution of the preset mesh.

Therefore, in this embodiment, a second target loss value is set, and then when the target loss value does not meet the preset conditions and/or the number of training times of the neural network model to be trained is less than the preset threshold value, the second target loss value is updated according to the The deformation network layer in the neural network model to be trained, so that when the target displacement of the vertices in the preset grid is predicted through the deformation network layer, the predicted target displacement is more accurate.

The second target loss value can be obtained by substituting the initial pixel feature and the mean value of the initial pixel feature into the following formula for calculation:

L _var represents the second target loss value, N represents the number of first region images, j represents the jth first region image, p _m represents the coordinates of the mth initial pixel feature in the jth first region image, _fm represents the m-th initial pixel feature in the j-th first region image, and f _j represents the mean value of the initial pixel features of the j-th first region image. |||| ₂ means 2-norm operation.

Through the deformation network layer in the neural network model to be trained, the displacement of the vertices of the preset mesh is predicted according to the training feature image and the preset mesh, and the initial deformation mesh is obtained, including:

Determine the vertex coordinates in the preset mesh;

According to the vertex coordinates, perform vertex feature search in the training feature image to obtain the first feature corresponding to the vertex coordinates;

According to the first feature and the context information of the vertex coordinates, determine the second feature corresponding to the vertex coordinates;

The initial displacement of the vertex coordinates is predicted according to the second feature, and the vertex coordinates are moved according to the initial displacement to obtain an initial deformation mesh.

In other embodiments, determining the target loss value according to the first target loss value and the second target loss value includes:

Determine the sub-areas of each grid in the initial deformed grid and the total area of the image to be segmented;

Determine the third target loss value according to the sub area and the total area;

Determine the target loss value according to the first target loss value, the second target loss value and the third target loss value;

If the target loss value does not meet the preset conditions and/or the training times of the neural network model to be trained is less than the preset threshold, the training times are increased by 1, and the features in the neural network model to be trained are updated according to the first target loss value The network parameters of the extraction layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer are updated according to the second target loss value. The network parameters of the deformation network layer in the neural network model to be trained are returned to execute Through the feature extraction layer in the neural network model to be trained, feature extraction is performed on the training image to obtain the training feature image, including:

If the target loss value does not meet the preset conditions and/or the training times of the neural network model to be trained is less than the preset threshold, the training times are increased by 1, and the feature extraction layer in the neural network model to be trained is updated according to the first target loss value The network parameters of the pixel context layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer are updated according to the second target loss value and the third target loss value. The network parameters of the deformation network layer in the neural network model to be trained , and return to execute the feature extraction layer in the neural network model to be trained to perform feature extraction on the training image to obtain the training feature image.

In order to avoid the problem of intersection between adjacent grids in the initial deformed grid, that is, the grids in the initial deformed grid overlap, causing the sum of the sub-areas of all grids in the initial deformed grid to exceed the total area of the image to be segmented, so , this embodiment also updates the deformation network layer according to the third target loss value, thereby avoiding the phenomenon of grid intersection in the initial deformation grid, and further, when predicting the target displacement of the vertices in the preset grid through the deformation network layer, predicting The target displacement is more accurate, that is, the target deformation meshes predicted by the deformation network layer will not cross the phenomenon.

Among them, the sub-area and total area can be substituted into the following formula to obtain the third target loss value:

L _area represents the third target loss value, area _j represents the sub-area of the j-th first area image, that is, the sub-area of the j-th grid, and area _img represents the total area of the image to be segmented.

Substitute the first target loss value, the second target loss value and the third target loss value into the following formula to obtain the target loss value L:

L=L _c +αL _var +βL _area

α represents the weight of the first target loss value, and β represents the weight of the second target loss value.

It should be noted that, when the number of training times is used as a condition for judging whether the neural network model to be trained terminates the training, determining the target loss value according to the first target loss value and the second target loss value may also refer to combining the first target loss value and the second target loss value. The second target loss value is used as the target loss value, that is, the target loss value includes the first target loss value and the second target loss value.

Similarly, when the number of training times is used as a condition for judging whether the neural network model to be trained terminates the training, determining the target loss value according to the first target loss value, the second target loss value and the third target loss value may also refer to: The first target loss value, the second target loss value and the third target loss value are used as the target loss value, that is, the target loss value includes the first target loss value, the second target loss value and the third target loss value.

It can be seen from the above that, in the embodiments of the present application, the images to be segmented are obtained first, and the images to be segmented are divided to obtain the initial region images corresponding to the images to be segmented. Then, the target pixel context information in the initial area image is determined, and the target area image corresponding to the initial area image is determined according to the target pixel context information and the initial area image. Next, the target area context information between the target area images is determined, and the target pixel features of the to-be-segmented image are determined according to the target area images and the target area context information. Finally, according to the target pixel features, the image to be segmented is identified and segmented.

That is, in the embodiment of the present application, the target pixel context information in the initial area image is first determined, and the target area image corresponding to the initial area image is determined according to the target pixel context information and the initial area image, and then the target area between the target area images is determined. Context information, and according to the target area image and the target area context information, determine the target pixel characteristics of the image to be segmented, that is, by determining the target area context information between the two target area images, it is unnecessary to calculate the relationship between each pixel in the target area image and the target area image. The context information between all the pixels in the other target area images makes it unnecessary to calculate the context information between each pixel in the to-be-segmented image and all other pixels in the to-be-segmented image, thereby reducing the computational complexity.

According to the methods described in the above embodiments, the following examples will be used for further detailed description.

In this embodiment, the semantic segmentation device is integrated in the terminal as an example. The semantic segmentation method may include an application method of the trained neural network model and a training method of the trained neural network model. Specifically, FIG. 5 is a schematic flowchart of a training method of a trained neural network model, and FIG. 6 is a schematic flowchart of an application method of the trained neural network model.

Referring to Figure 5, the training methods of the trained neural network model include:

S501. The terminal acquires a training sample set, and determines the target weight corresponding to the category of the training pixels according to the number of training pixels in the training images of the training sample set.

S502: The terminal performs feature extraction on the training image through the feature extraction layer in the neural network model to be trained to obtain a training feature image.

In this embodiment, the structure of the neural network model to be trained may be as shown in FIG. 6 . The neural network model to be trained includes a feature extraction layer, a pixel context layer, a region context layer, a deformation network layer, and a segmentation layer. The extraction layer can be a residual neural network, and then the training feature image F is obtained through the feature extraction layer.

It should be understood that after obtaining the training feature image, the training feature image can also be dimensionally reduced to obtain a dimensionality-reduced training feature image X, and then the dimensionality-reduced training feature image is input to the deformation network layer and the pixel context layer.

S503. The terminal determines the vertex coordinates in the preset grid by deforming the network layer in the neural network model to be trained, and performs vertex feature search in the training feature image according to the vertex coordinates to obtain the first feature corresponding to the vertex coordinates.

For example, as shown in Figure 6, the deformation network layer includes an extraction layer, a CoordConv layer, a 6-layer point-by-point convolution layer, a self-attention layer, and a prediction convolution layer. The terminal determines the preset grid through the extraction layer in the deformation network layer. vertex coordinates (t _r , s _r ), and according to the vertex coordinates, perform vertex feature search in the training feature image to obtain the first feature corresponding to the vertex coordinates.

S504, the terminal determines the second feature corresponding to the vertex coordinate according to the first feature and the context information of the vertex coordinate through the deformation network layer in the neural network model to be trained, and predicts the initial displacement of the vertex coordinate according to the second feature, and according to the initial displacement Move the vertex coordinates to get the initial deformed mesh.

By deforming the CoordConv layer in the network layer, the terminal can first fuse the first feature and the vertex coordinates to obtain the initial candidate feature. Then, the terminal performs point-by-point convolution on the initial candidate features through the point-by-point convolution layer in the deformation network layer to obtain the initial intermediate features, and then obtains the context information of the vertex coordinates through the self-attention layer in the deformation network layer. The context information of the feature and vertex coordinates determines the second feature corresponding to the vertex coordinates, and the point-by-point convolution process and the process of the self-attention layer can be performed 6 times. Finally, through the prediction convolution layer, the terminal predicts the initial displacement (Δt _r , Δs _r ) of the vertex coordinates according to the second feature, and moves the vertex coordinates according to the initial displacement to obtain the initial deformed mesh.

S505: The terminal divides the training feature image according to the initial deformation grid through the pixel context layer in the neural network model to be trained, and obtains the training area image.

S506. The terminal determines the initial pixel context information in the training area image through the pixel context layer in the neural network model to be trained, and determines the first area image corresponding to the training image according to the initial pixel context information and the training area image.

则第一区域图像可以为

其中，K_j表示第j个训练区域图像中的像素，并且

则各个第一区域图像可以组成新特征图X'∈R^C×H×W，其中，R表示维度，C表示通道数，H表示训练图像的高度，W表示训练图像的宽度。For training area images

Then the first region image can be

where Kj represents the pixel in the _jth training region image, and

Then each first region image can form a new feature map X'∈R ^C×H×W , where R represents the dimension, C represents the number of channels, H represents the height of the training image, and W represents the width of the training image.

S507. The terminal determines the first region feature corresponding to the first region image according to the average value of the pixels of the first region image through the region pooling layer of the region context layer in the neural network model to be trained.

S508, the terminal determines the initial region context information between the images of the first region through the region-level self-attention mechanism layer of the region context layer in the neural network model to be trained, according to the first region feature, and according to the initial region context information and the first region context information A region feature is obtained, and a second region feature is obtained.

S509: The terminal maps the second region feature to the initial deformation grid through the region de-pooling layer of the region context layer in the neural network model to be trained to obtain the initial pixel feature of the training image.

S5010. The terminal determines the target category of the training pixel corresponding to the initial pixel feature according to the initial pixel feature, the first region image, and the training feature image through the segmentation layer in the neural network model to be trained.

S5011. The terminal determines a first target loss value according to the target category, the labels of the training pixels and the target weight.

S5012. The terminal determines the second target loss value according to the initial pixel feature and the mean value of the initial pixel feature.

S5013. The terminal determines the sub-areas of each grid in the initial deformation grid and the total area of the image to be segmented, and determines a third target loss value according to the sub-areas and the total area.

S5014. The terminal acquires the training times of the neural network model to be trained.

S5015. If the number of training times is less than the preset threshold, the terminal increases the number of training times by 1, and updates the network parameters of the feature extraction layer, the network parameters of the pixel context layer, and the network parameters of the region context layer in the neural network model to be trained according to the first target loss value. For the network parameters and the network parameters of the segmentation layer, update the network parameters of the deformation network layer in the neural network model to be trained according to the second target loss value and the third target loss value, and return to execute S502.

S5016. If the number of training times is equal to the preset threshold, the terminal takes the neural network model to be trained as the trained neural network type.

Referring to Figure 7, the application method of the trained neural network model includes:

S701. The terminal acquires the image to be segmented, and performs feature extraction on the image to be segmented through the feature extraction layer in the trained neural network model to obtain a feature image.

S702. The terminal obtains a preset grid, and determines the vertex coordinates in the preset grid through the deformation network layer in the trained neural network model, and performs vertex feature search in the feature image according to the vertex coordinates to obtain the corresponding vertex coordinates. initial features.

S703, the terminal determines the target feature corresponding to the vertex coordinate according to the initial feature and the context information of the vertex coordinate through the deformation network layer in the trained neural network model, and predicts the target displacement of the vertex coordinate according to the target feature, and determines the vertex coordinate according to the target displacement. Move to get the target deformed mesh.

S704. The terminal divides the feature image according to the target deformation grid through the pixel context layer in the trained neural network model, and obtains an initial region image corresponding to the image to be divided.

S705. The terminal determines the target pixel context information in the initial area image through the pixel context layer in the trained neural network model, and determines the target area image corresponding to the initial area image according to the target pixel context information and the initial area image.

S706. The terminal determines the initial region feature corresponding to the target region image according to the average value of the pixels of the target region image through the region pooling layer of the region context layer in the trained neural network model.

S707. The terminal determines the target area context information between the target area images through the area-level self-attention layer of the area context layer in the trained neural network model, and determines the target area feature according to the initial area feature and the target area context information.

S708. The terminal maps the target area feature to the target deformation grid through the area de-pooling layer of the area context layer in the trained neural network model to obtain the target pixel feature of the image to be segmented.

For example, the obtained target pixel feature can be X"∈R ^C×H×W .

S709. The terminal uses the segmentation layer in the trained neural network model to segment the image to be segmented according to the target pixel feature, the target area image, and the feature image to obtain a segmentation result.

Wherein, the segmentation layer can be a softmax layer. For example, the segmentation result obtained by the trained neural network model can be shown in Fig. 8, where the dotted box in Fig. 8 represents the region of interest.

For the beneficial effects and other achievable manners in this embodiment, reference may be made to the above-mentioned embodiment of the semantic segmentation method, and details are not described herein again in this embodiment.

To facilitate better implementation of the semantic segmentation method provided by the embodiment of the present application, the embodiment of the present application further provides an apparatus based on the above-mentioned semantic segmentation method. The meanings of the nouns are the same as those in the above-mentioned semantic segmentation method, and the specific implementation details can refer to the description in the method embodiment.

For example, as shown in Figure 9, the semantic segmentation device may include:

The image acquisition module 901 is configured to acquire an image to be segmented, and to divide the image to be segmented to obtain an initial region image corresponding to the image to be segmented.

The first determining module 902 is configured to determine the target pixel context information in the initial area image, and determine the target area image corresponding to the initial area image according to the target pixel context information and the initial area image.

The second determining module 903 is configured to determine the target area context information between the target area images, and determine the target pixel feature of the image to be segmented according to the target area image and the target area context information.

The image segmentation module 904 is configured to segment the image to be segmented according to the feature of the target pixel to obtain a segmentation result.

Optionally, the image acquisition module 901 is specifically configured to execute:

Perform feature extraction on the image to be segmented to obtain a feature image;

Obtain a preset mesh, and perform displacement prediction on the vertices of the preset mesh according to the feature image and the preset mesh to obtain the target deformation mesh;

The feature image is divided according to the target deformation grid to obtain the initial region image corresponding to the image to be divided.

Optionally, the image acquisition module 901 is specifically configured to execute:

Determine the vertex coordinates in the preset mesh;

According to the vertex coordinates, perform vertex feature search in the feature image to obtain the initial features corresponding to the vertex coordinates;

Determine the target feature corresponding to the vertex coordinate according to the initial feature and the context information of the vertex coordinate;

The target displacement of the vertex coordinates is predicted according to the target feature, and the vertex coordinates are moved according to the target displacement to obtain the target deformation mesh.

Optionally, the second determining module 903 is specifically configured to execute:

According to the mean value of the pixels of the target area image, determine the initial area feature corresponding to the target area image;

Determine the target area context information between the target area images according to the initial area features;

Determine the target area feature according to the initial area feature and the target area context information;

The target area features are mapped to the target deformation grid to obtain the target pixel features of the image to be segmented.

Optionally, the above-mentioned semantic segmentation device further includes:

A training module that executes:

Obtain the training sample set, and determine the target weight corresponding to the category of the training pixel according to the number of training pixels in the training image of the training sample set;

Divide the training image to obtain the training area image;

Determine the initial pixel context information in the training area image through the pixel context layer in the neural network model to be trained, and determine the first area image corresponding to the training image according to the initial pixel context information and the training area image;

Determine the initial area context information between the first area images through the area context layer in the neural network model to be trained, and determine the initial pixel feature of the training image according to the first area image and the initial area context information;

Determine the target category of the training pixels corresponding to the initial pixel features through the segmentation layer in the neural network model to be trained;

Determine the target loss value according to the target category, the label of the training pixel and the target weight;

Obtain the training times of the neural network model to be trained;

Based on the target loss value and the number of training times, the neural network model to be trained is trained to obtain the trained neural network model.

Optionally, the training module is specifically used to execute:

Through the feature extraction layer in the neural network model to be trained, feature extraction is performed on the training image to obtain the training feature image;

Through the deformation network layer in the neural network model to be trained, displacement prediction is performed on the vertices of the preset mesh according to the training feature image and the preset mesh to obtain the initial deformation mesh;

Through the pixel context layer in the neural network model to be trained, the training feature image is divided according to the initial deformation grid, and the training area image is obtained;

Determine the first target loss value according to the target category, the label of the training pixel and the target weight;

Determine the second target loss value according to the initial pixel feature and the mean value of the initial pixel feature;

Determine the target loss value according to the first target loss value and the second target loss value;

If the target loss value does not meet the preset conditions and/or the training times of the neural network model to be trained is less than the preset threshold, the training times are increased by 1, and the feature extraction layer in the neural network model to be trained is updated according to the first target loss value The network parameters of the pixel context layer, the network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer, update the network parameters of the deformation network layer in the neural network model to be trained according to the second target loss value, and return to execute The feature extraction layer in the trained neural network model performs feature extraction on the training image to obtain the training feature image;

If the target loss value satisfies the preset condition and/or the training times of the neural network model to be trained is equal to the preset threshold, the training is stopped, and the trained neural network model is obtained.

Optionally, the training module is specifically used to execute:

Determine the sub-areas of each grid in the initial deformed grid and the total area of the image to be segmented;

Determine the third target loss value according to the sub area and the total area;

Determine the target loss value according to the first target loss value, the second target loss value and the third target loss value;

If the target loss value does not meet the preset conditions and/or the training times of the neural network model to be trained is less than the preset threshold, the training times are increased by 1, and the feature extraction layer in the neural network model to be trained is updated according to the first target loss value The network parameters of the pixel context layer, the network parameters of the region context layer, and the network parameters of the segmentation layer are updated according to the second target loss value and the third target loss value. The network parameters of the deformation network layer in the neural network model to be trained , and return to execute the feature extraction layer in the neural network model to be trained to perform feature extraction on the training image to obtain the training feature image.

During specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined, implemented as the same or several entities. The specific implementation of the above modules and the corresponding beneficial effects can refer to the previous method embodiments. It is not repeated here.

The embodiment of the present application also provides an electronic device, which may be a server or a terminal, etc. As shown in FIG. 10 , which shows a schematic structural diagram of the electronic device involved in the embodiment of the present application, specifically:

The electronic device may include a processor 1001 of one or more processing cores, a memory 1002 of one or more computer-readable storage media, a power supply 1003 and an input unit 1004 and other components. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 10 does not constitute a limitation on the electronic device, and may include more or less components than the one shown, or combine some components, or arrange different components. in:

The processor 1001 is the control center of the electronic device, uses various interfaces and lines to connect various parts of the entire electronic device, runs or executes the computer programs and/or modules stored in the memory 1002, and invokes the computer program stored in the memory 1002. data, perform various functions of electronic equipment and process data. Optionally, the processor 1001 may include one or more processing cores; preferably, the processor 1001 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc. , the modem processor mainly deals with wireless communication. It can be understood that, the above-mentioned modulation and demodulation processor may not be integrated into the processor 1001.

The memory 1002 can be used to store computer programs and modules, and the processor 1001 executes various functional applications and data processing by running the computer programs and modules stored in the memory 1002 . The memory 1002 may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, a computer program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; Data created by the use of electronic equipment, etc. Additionally, memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, memory 1002 may also include a memory controller to provide processor 1001 access to memory 1002 .

The electronic device also includes a power supply 1003 for supplying power to various components. Preferably, the power supply 1003 can be logically connected to the processor 1001 through a power management system, so as to manage charging, discharging, and power consumption management functions through the power management system. Power supply 1003 may also include one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and any other components.

The electronic device may also include an input unit 1004, which may be used to receive input numerical or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

Although not shown, the electronic device may further include a display unit and the like, which will not be described here. Specifically, in this embodiment, the processor 1001 in the electronic device loads the executable files corresponding to the processes of one or more computer programs into the memory 1002 according to the following instructions, and the processor 1001 executes them and stores them in the memory 1002 . The computer program in the memory 1002, thereby realizing various functions, such as:

Obtain the image to be segmented, and divide the image to be segmented to obtain an initial area image corresponding to the image to be segmented;

Determine the target pixel context information in the initial area image, and determine the target area image corresponding to the initial area image according to the target pixel context information and the initial area image;

Determine the target area context information between the target area images, and determine the target pixel feature of the to-be-segmented image according to the target area image and the target area context information;

Segment the image to be segmented according to the feature of the target pixel, and obtain the segmentation result.

For specific implementations of the above operations and corresponding beneficial effects, reference may be made to the detailed description of the semantic segmentation method above, which will not be repeated here.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a computer program, or by a computer program that controls related hardware, and the computer program can be stored in a computer-readable storage. medium, and is loaded and executed by the processor.

To this end, the embodiments of the present application provide a computer-readable storage medium, in which a computer program is stored, and the computer program can be loaded by a processor to execute the steps in any of the semantic segmentation methods provided by the embodiments of the present application . For example, the computer program may perform the following steps:

Obtain the image to be segmented, and divide the image to be segmented to obtain an initial area image corresponding to the image to be segmented;

Determine the target pixel context information in the initial area image, and determine the target area image corresponding to the initial area image according to the target pixel context information and the initial area image;

Determine the target area context information between the target area images, and determine the target pixel feature of the to-be-segmented image according to the target area image and the target area context information;

Segment the image to be segmented according to the feature of the target pixel, and obtain the segmentation result.

For the specific implementation manners of the above operations and the corresponding beneficial effects, reference may be made to the foregoing embodiments, which will not be repeated here.

Wherein, the computer-readable storage medium may include: read only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and the like.

Because the computer program stored in the computer-readable storage medium can execute the steps in any of the semantic segmentation methods provided by the embodiments of the present application, it is possible to implement any of the semantic segmentation methods provided by the embodiments of the present application. For the beneficial effects that can be achieved, refer to the foregoing embodiments for details, which will not be repeated here.

Wherein, according to one aspect of the present application, there is provided a computer program product or computer program, the computer program product or computer program comprising computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned semantic segmentation method.

A semantic segmentation method, apparatus, electronic device, and computer-readable storage medium provided by the embodiments of the present application have been described above in detail. The principles and implementations of the present application are described with specific examples. The above embodiments The description is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, there will be changes in the specific embodiments and application scope. In summary, the above , the contents of this specification should not be construed as limiting the application.

Claims (10)

1. A method of semantic segmentation, comprising:

acquiring an image to be segmented, and dividing the image to be segmented to obtain an initial region image corresponding to the image to be segmented;

determining target pixel context information in the initial area image, and determining a target area image corresponding to the initial area image according to the target pixel context information and the initial area image;

determining target area context information between the target area images, and determining target pixel characteristics of the image to be segmented according to the target area images and the target area context information;

and segmenting the image to be segmented according to the target pixel characteristics to obtain a segmentation result.

2. The semantic segmentation method according to claim 1, wherein the step of dividing the image to be segmented to obtain an initial region image corresponding to the image to be segmented comprises:

performing feature extraction on the image to be segmented to obtain a feature image;

acquiring a preset grid, and performing displacement prediction on the vertex of the preset grid according to the characteristic image and the preset grid to obtain a target deformation grid;

and dividing the characteristic image according to the target deformation grid to obtain an initial area image corresponding to the image to be segmented.

3. The semantic segmentation method according to claim 2, wherein the determining of the context information of the target area between the target area images and the determining of the target pixel feature of the image to be segmented according to the target area images and the context information of the target area comprise:

determining initial area characteristics corresponding to the target area image according to the mean value of the pixels of the target area image;

determining target area context information between the target area images according to the initial area characteristics;

determining the target area characteristics according to the initial area characteristics and the target area context information;

and mapping the target region characteristics to the target deformation grid to obtain the target pixel characteristics of the image to be segmented.

4. The semantic segmentation method according to any one of claims 1-3, wherein the target pixel context information and the target area image are determined by a pixel context layer in a trained neural network model, the target area context information and the target pixel feature are determined by a region context layer in the trained neural network model, and the segmentation result is determined by a segmentation layer in the trained neural network model, the method further comprising:

acquiring a training sample set, and determining a target weight corresponding to the category of a training pixel according to the number of the training pixels in a training image of the training sample set;

dividing the training image to obtain a training area image;

determining initial pixel context information in the training area image through a pixel context layer in a neural network model to be trained, and determining a first area image corresponding to the training image according to the initial pixel context information and the training area image;

determining initial region context information between the first region images through a region context layer in the neural network model to be trained, and determining initial pixel characteristics of the training images according to the first region images and the initial region context information;

determining the target class of the training pixel corresponding to the initial pixel feature through a segmentation layer in the neural network model to be trained;

determining a target loss value according to the target category, the label of the training pixel and the target weight;

acquiring the training times of the neural network model to be trained;

and training the neural network model to be trained based on the target loss value and the training times to obtain the trained neural network model.

5. The semantic segmentation method according to claim 4, wherein the target deformation mesh is obtained through a deformation network layer in the trained neural network model;

the dividing the training image to obtain a training area image includes:

performing feature extraction on the training image through a feature extraction layer in the neural network model to be trained to obtain a training feature image;

performing displacement prediction on the vertex of the preset mesh according to the training characteristic image and the preset mesh through the deformation network layer in the neural network model to be trained to obtain an initial deformation mesh;

dividing the training characteristic image according to the initial deformation grid through the pixel context layer in the neural network model to be trained to obtain a training area image;

determining a target loss value according to the target class, the label of the training pixel and the target weight, including:

determining a first target loss value according to the target category, the label of the training pixel and the target weight;

determining a second target loss value according to the initial pixel characteristic and the average value of the initial pixel characteristic;

determining a target loss value according to the first target loss value and the second target loss value;

the training the neural network model to be trained based on the target loss value and the training times to obtain the trained neural network model, including:

if the target loss value does not meet a preset condition and/or the training times of the neural network model to be trained are smaller than a preset threshold value, increasing 1 for the training times, updating the network parameters of the feature extraction layer, the pixel context layer, the region context layer and the segmentation layer in the neural network model to be trained according to the first target loss value, updating the network parameters of the deformation network layer in the neural network model to be trained according to the second target loss value, returning to execute the feature extraction of the training image through the feature extraction layer in the neural network model to be trained, and obtaining a training feature image;

and if the target loss value meets a preset condition and/or the training times of the neural network model to be trained are equal to a preset threshold value, stopping training to obtain the trained neural network model.

6. The semantic segmentation method according to claim 5, wherein determining a target loss value based on the first target loss value and the second target loss value comprises:

determining the sub-area of each grid in the initial deformation grid and the total area of the image to be segmented;

determining a third target loss value according to the sub-area and the total area;

determining a target loss value according to the first target loss value, the second target loss value and the third target loss value;

if the target loss value does not meet a preset condition and/or the training frequency of the neural network model to be trained is smaller than a preset threshold value, adding 1 to the training frequency, updating the network parameters of the feature extraction layer, the pixel context layer, the region context layer and the segmentation layer in the neural network model to be trained according to the first target loss value, updating the network parameters of the deformation network layer in the neural network model to be trained according to the second target loss value, returning to execute the feature extraction through the feature extraction layer in the neural network model to be trained, and performing the feature extraction on the training image to obtain a training feature image, wherein the method comprises the following steps:

if the target loss value does not meet a preset condition and/or the training frequency of the neural network model to be trained is smaller than a preset threshold value, increasing 1 for the training frequency, updating the network parameters of the feature extraction layer, the pixel context layer, the region context layer and the segmentation layer in the neural network model to be trained according to the first target loss value, updating the network parameters of the deformation network layer in the neural network model to be trained according to the second target loss value and the third target loss value, and returning to execute the feature extraction of the training image through the feature extraction layer in the neural network model to be trained to obtain a training feature image.

7. A semantic segmentation apparatus, comprising:

the image acquisition module is used for acquiring an image to be segmented and dividing the image to be segmented to obtain an initial region image corresponding to the image to be segmented;

a first determining module, configured to determine context information of a target pixel in the initial area image, and determine a target area image corresponding to the initial area image according to the context information of the target pixel and the initial area image;

the second determining module is used for determining target area context information between the target area images and determining target pixel characteristics of the image to be segmented according to the target area images and the target area context information;

and the image segmentation module is used for segmenting the image to be segmented according to the target pixel characteristics to obtain a segmentation result.

8. An electronic device comprising a processor and a memory, the memory storing a computer program, the processor being configured to execute the computer program in the memory to perform the semantic segmentation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that it stores a computer program adapted to be loaded by a processor for performing the semantic segmentation method according to any one of claims 1 to 6.

10. A computer program product, characterized in that it stores a computer program adapted to be loaded by a processor for performing the semantic segmentation method according to any one of claims 1 to 6.

Images