CN113807354A - Image semantic segmentation method, device, equipment and storage medium - Google Patents
Image semantic segmentation method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113807354A CN113807354A CN202011592663.9A CN202011592663A CN113807354A CN 113807354 A CN113807354 A CN 113807354A CN 202011592663 A CN202011592663 A CN 202011592663A CN 113807354 A CN113807354 A CN 113807354A
- Authority
- CN
- China
- Prior art keywords
- vector
- semantic segmentation
- image
- basic feature
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000011218 segmentation Effects 0.000 title claims abstract description 184
- 238000000034 method Methods 0.000 title claims abstract description 69
- 239000013598 vector Substances 0.000 claims abstract description 351
- 238000012545 processing Methods 0.000 claims abstract description 77
- 230000008447 perception Effects 0.000 claims abstract description 40
- 238000004590 computer program Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 description 19
- 238000000605 extraction Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 8
- 238000001514 detection method Methods 0.000 description 6
- 238000007781 pre-processing Methods 0.000 description 6
- 238000003745 diagnosis Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 238000012549 training Methods 0.000 description 5
- 201000010099 disease Diseases 0.000 description 4
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000005728 strengthening Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000005562 fading Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application provides an image semantic segmentation method, an image semantic segmentation device, image semantic segmentation equipment and a storage medium, wherein the method is applied to the technical field of data processing and comprises the following steps: obtaining an image to be segmented; extracting the features of the image to be segmented to obtain a plurality of basic feature vectors; carrying out pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed; generating a target feature vector according to a plurality of feature vectors to be processed; and classifying the target characteristic vectors to obtain a semantic segmentation result of the image to be segmented. Therefore, the target characteristic vectors generated after the pixel perception and/or the boundary enhancement processing are classified to obtain the semantic segmentation result, the relevance learning and the content understanding between the pixels of the local area of the image are enhanced, the accurate identification of small objects and boundary information is ensured, and the accuracy of the semantic segmentation result of the image is improved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for semantic segmentation of an image.
Background
Generally, semantic segmentation is one of basic tasks of computer vision, and can be applied to various fields, such as smart agriculture, automatic driving, disease diagnosis, and the like.
In the related technology, the semantic segmentation algorithm based on the convolutional neural network receives more and more attention due to the characteristic of high prediction precision, however, most of the current semantic segmentation network models are realized based on a pixel-by-pixel classification method, the correlation among the characteristics is neglected, further the understanding of the image content is lacked, the accurate identification of small objects and boundary information is not ensured, and the defect of overlarge parameter amount exists, so that the method is not suitable for being applied to mobile equipment.
Disclosure of Invention
The present application is directed to solving, at least to some extent, one of the technical problems in the related art.
The application provides an image semantic segmentation method, device, equipment and storage medium, which are used for realizing classification processing of target characteristic vectors generated after pixel perception and/or boundary enhancement processing to obtain semantic segmentation results, strengthening relevance learning and content understanding between pixels in local regions of an image, ensuring accurate identification of small objects and boundary information, improving accuracy of image semantic segmentation results and solving the technical problems of inaccurate image semantic segmentation and large calculation amount in the prior art.
An embodiment of a first aspect of the present application provides an image semantic segmentation method, including:
acquiring an image to be segmented;
extracting the features of the image to be segmented to obtain a plurality of basic feature vectors;
performing pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed;
generating a target feature vector according to the plurality of feature vectors to be processed;
and classifying the target characteristic vector to obtain a semantic segmentation result of the image to be segmented.
According to the image semantic segmentation method, the image to be segmented is obtained; extracting the features of the image to be segmented to obtain a plurality of basic feature vectors; carrying out pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed; generating a target feature vector according to a plurality of feature vectors to be processed; and classifying the target characteristic vectors to obtain a semantic segmentation result of the image to be segmented. Therefore, the target characteristic vectors generated after the pixel perception and/or the boundary enhancement processing are classified to obtain the semantic segmentation result, the relevance learning and the content understanding between the pixels of the local area of the image are enhanced, the accurate identification of small objects and boundary information is ensured, and the accuracy of the semantic segmentation result of the image is improved.
The embodiment of the second aspect of the present application provides another image semantic segmentation apparatus, including:
the first acquisition module is used for acquiring an image to be segmented;
the second acquisition module is used for extracting the features of the image to be segmented to acquire a plurality of basic feature vectors;
the processing module is used for carrying out pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed;
the generating module is used for generating a target characteristic vector according to the plurality of characteristic vectors to be processed;
and the third acquisition module is used for classifying the target characteristic vector to acquire a semantic segmentation result of the image to be segmented.
The image semantic segmentation device of the embodiment of the application acquires an image to be segmented; extracting the features of the image to be segmented to obtain a plurality of basic feature vectors; carrying out pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed; generating a target feature vector according to a plurality of feature vectors to be processed; and classifying the target characteristic vectors to obtain a semantic segmentation result of the image to be segmented. Therefore, the target characteristic vectors generated after the pixel perception and/or the boundary enhancement processing are classified to obtain the semantic segmentation result, the relevance learning and the content understanding between the pixels of the local area of the image are enhanced, the accurate identification of small objects and boundary information is ensured, and the accuracy of the semantic segmentation result of the image is improved.
An embodiment of a third aspect of the present application provides an electronic device, including: the image semantic segmentation method comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein when the processor executes the program, the image semantic segmentation method is realized as set forth in the embodiment of the first aspect of the application.
An embodiment of a fourth aspect of the present application provides a non-transitory computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the image semantic segmentation method as set forth in the embodiment of the first aspect of the present application.
In a fifth aspect, the present application provides a computer program product, and when executed by an instruction processor in the computer program product, the method for semantic segmentation of an image provided in the first and second aspects of the present application is performed.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a schematic flowchart of an image semantic segmentation method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of an image semantic segmentation method according to a second embodiment of the present application;
FIG. 3 is an exemplary diagram of a convolutional network according to an embodiment of the present application;
FIG. 4 is an exemplary diagram of pixel-aware network processing according to an embodiment of the present application;
fig. 5 is a schematic flowchart of an image semantic segmentation method according to a third embodiment of the present application;
FIG. 6 is an exemplary diagram of a border enhanced network process according to an embodiment of the present application;
fig. 7 is a schematic flowchart of an image semantic segmentation method according to a fourth embodiment of the present application;
FIG. 8 is an exemplary diagram of semantic segmentation of an image provided by an embodiment of the present application;
FIG. 9 is an exemplary diagram of semantic segmentation of an image provided by an embodiment of the present application;
fig. 10 is a schematic structural diagram of an image semantic segmentation apparatus according to a fourth embodiment of the present application;
fig. 11 is a schematic structural diagram of an image semantic segmentation apparatus according to a fifth embodiment of the present application;
FIG. 12 illustrates a block diagram of an exemplary electronic device suitable for use in implementing embodiments of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary and intended to be used for explaining the present application and should not be construed as limiting the present application.
In practical application, semantic segmentation is required to be carried out on images through scenes such as intelligent agriculture, automatic driving and the like, in the related technology, the semantic segmentation algorithm based on the convolutional neural network neglects the correlation among features, further lacks understanding of image contents, and does not guarantee accurate identification of small objects and boundary information.
Aiming at the problems, the application provides an image semantic segmentation method, which is used for obtaining an image to be segmented; extracting features of an image to be segmented to obtain a plurality of basic feature vectors; carrying out pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed; generating a target feature vector according to a plurality of feature vectors to be processed; and classifying the target characteristic vectors by utilizing a classification network in the semantic segmentation network to obtain a semantic segmentation result of the image to be segmented.
Therefore, the target characteristic vectors generated after the pixel perception and/or the boundary enhancement processing are classified to obtain the semantic segmentation result, the relevance learning and the content understanding between the pixels of the local area of the image are enhanced, the accurate identification of small objects and boundary information is ensured, and the accuracy of the semantic segmentation result of the image is improved.
An image semantic segmentation method, an apparatus, a device, and a storage medium according to embodiments of the present application are described below with reference to the accompanying drawings.
Fig. 1 is a schematic flow chart of an image semantic segmentation method according to an embodiment of the present application.
The embodiment of the present application is exemplified by that the image semantic segmentation method is configured in an image semantic segmentation apparatus, and the image semantic segmentation apparatus can be applied to any electronic device, so that the electronic device can perform an image semantic segmentation function.
In the embodiment of the application, the semantic segmentation method refers to a method for segmenting a group of semantics into the smallest opposite components in comparison, and describing and analyzing the relationship between the smallest opposite components so as to determine a certain semantic content; pixel sensing may be understood as the acquisition of context information between pixels, the fusion of context information and the processing of context information.
As shown in fig. 1, the image semantic segmentation method may include the following steps:
In the embodiment of the application, the image to be segmented can be obtained by selecting and setting according to different application scenes, for example, in an automatic driving application scene, a picture within a preset distance in the driving process of a vehicle shot by a camera is subjected to pixel extraction and other preprocessing to be used as the image to be segmented; for example, in a disease diagnosis application scene, the image to be segmented is a picture taken by medical equipment, and is subjected to preprocessing such as pixel extraction and the like to be used as the image to be segmented.
And 102, extracting the features of the image to be segmented to obtain a plurality of basic feature vectors.
In the embodiment of the application, there are various ways of extracting features of an image to be segmented and obtaining a plurality of basic feature vectors, and as an example, a convolution network in a semantic segmentation network is used to extract features of the image to be segmented and obtain a plurality of basic feature vectors; as another example, a bp (back propagation) neural network in a semantic segmentation network is used to perform feature extraction on an image to be segmented to obtain a plurality of basic feature vectors, and the like, and the setting is selected according to an application scenario.
In the embodiment of the present application, as an example, the semantic segmentation network includes: the image classification method based on the pixel perception network comprises the following steps of performing convolution on a network, performing pixel perception network and/or boundary enhancement network, and performing classification network, thereby introducing the pixel perception network and/or the boundary enhancement network, strengthening the associated learning and content understanding of the semantic segmentation network on pixels in local areas of an image, and promoting the semantic segmentation network to better detect boundary information and small objects.
In the embodiment of the application, the preprocessed image to be segmented is firstly subjected to feature extraction through a convolution network, and the obtained basic feature vector with rich semantics is input into a subsequent network for processing.
In the embodiment of the present application, there are many ways of extracting features of an image to be segmented by using a convolutional network in a semantic segmentation network, and the setting may be selected according to the application scene, which is exemplified as follows.
In a first example, a plurality of convolution networks with different resolutions in a semantic segmentation network are used to perform feature extraction on an image to be segmented respectively, so as to obtain a plurality of basic feature vectors.
In a second example, a plurality of convolution networks with the same resolution in the semantic segmentation network are used to perform feature extraction on an image to be segmented respectively, so as to obtain a plurality of basic feature vectors.
And 103, carrying out pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed.
In the embodiment of the present application, there are many ways to perform pixel sensing and/or boundary enhancement processing on a plurality of basic feature vectors, which may be specifically selected according to an actual application scenario, and the following are exemplified:
in a first example, a plurality of basic feature vectors are subjected to pixel perception and/or boundary enhancement processing by utilizing a pixel perception network and/or a boundary enhancement network in a semantic segmentation network to generate a plurality of feature vectors to be processed.
Specifically, there are many ways to perform pixel sensing and/or boundary enhancement processing on a plurality of basic feature vectors by using a pixel sensing network and/or a boundary enhancement network in a semantic segmentation network to generate a plurality of feature vectors to be processed, which are illustrated as follows:
example one, a first basic feature sub-vector and a second basic feature sub-vector of each basic feature vector are obtained; generating a transposed characteristic component corresponding to the second basic characteristic sub-vector according to the second basic characteristic sub-vector; generating a first to-be-weighted feature vector corresponding to each basic feature vector according to the transposed feature components corresponding to the first basic feature sub-vector and the second basic feature sub-vector; and generating a plurality of feature vectors to be processed according to each basic feature vector and the corresponding first feature vector to be weighted.
Example two, a transposed feature vector corresponding to each basic feature vector is obtained; generating a second feature vector to be weighted corresponding to each basic feature vector according to each basic feature vector and the corresponding transposed feature vector; and generating a plurality of feature vectors to be processed according to each basic feature vector and the corresponding second feature vector to be weighted.
In a second example, a deep learning integrated neural network in a semantic segmentation network is used for simultaneously carrying out pixel perception and boundary enhancement processing on a plurality of basic feature vectors to obtain a plurality of feature vectors to be processed.
And 104, generating a target feature vector according to the plurality of feature vectors to be processed.
And 105, processing the target characteristic vector to obtain a semantic segmentation result of the image to be segmented.
In the embodiment of the present application, there are many ways to generate the target feature vector according to a plurality of feature vectors to be processed, which may be specifically selected according to practical applications, for example, as follows:
in a first example, a plurality of feature vectors to be processed are subjected to weighted summation to obtain a target feature vector.
In a second example, a target function is obtained according to a relationship between a plurality of historical to-be-processed feature vectors and a historical target feature vector, and the target feature vector is obtained by processing the plurality of to-be-processed feature vectors through the target function.
Further, the target feature vector is processed to obtain a semantic segmentation result of the image to be segmented, namely, the classification prediction of the image to be segmented is realized, so that the purpose of semantic segmentation is achieved.
According to the image semantic segmentation method, the image to be segmented is obtained; extracting the features of the image to be segmented to obtain a plurality of basic feature vectors; carrying out pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed; generating a target feature vector according to a plurality of feature vectors to be processed; and classifying the target characteristic vectors to obtain a semantic segmentation result of the image to be segmented. Therefore, the target characteristic vectors generated after the pixel perception and/or the boundary enhancement processing are classified to obtain the semantic segmentation result, the relevance learning and the content understanding between the pixels of the local area of the image are enhanced, the accurate identification of small objects and boundary information is ensured, and the accuracy of the semantic segmentation result of the image is improved.
Based on the description of the above embodiments, it can be understood that the pixel-aware network or the boundary enhancement network can be selected according to the requirements of the application scenario, and the pixel-aware network and the boundary enhancement network are simultaneously selected to perform the image semantic segmentation processing, and how to perform the image semantic segmentation processing based on the pixel-aware network is described in detail below with reference to fig. 2.
Fig. 2 is a schematic flow chart of an image semantic segmentation method according to a second embodiment of the present application.
As shown in fig. 2, the image semantic segmentation method may include the following steps:
In the embodiment of the application, the image to be segmented can be obtained by selecting and setting according to different application scenes, for example, in an automatic driving application scene, a picture within a preset distance in the driving process of a vehicle shot by a camera is subjected to pixel extraction and other preprocessing to be used as the image to be segmented; for example, in a disease diagnosis application scene, the image to be segmented is a picture taken by medical equipment, and is subjected to preprocessing such as pixel extraction and the like to be used as the image to be segmented.
In the embodiment of the present application, a plurality of convolution networks with different resolutions may be used to respectively perform feature extraction on an image to be segmented, and the semantic segmentation network of the present application has a lighter weight, so that the amount of computation and the complexity are greatly reduced, and therefore, in the embodiment of the present application, the selection of the convolution network may be as shown in fig. 2.
Specifically, as shown in fig. 3, the network includes 6 convolutional layers, the maximum number of channels is only 128, and compared with other networks which use dozens of convolutional layers and hundreds of channels, the network has the characteristic of being lighter in weight, and greatly reduces the amount of calculation and complexity.
In this embodiment, each basic feature vector may be subjected to one or more convolution processes to obtain a first basic feature sub-vector and a second basic feature sub-vector, which is described as follows.
In a first example, a convolution network with a first resolution ratio is used for carrying out convolution processing on basic feature vectors for the Nth time to obtain first basic feature sub-vectors; carrying out mth convolution processing on the basic characteristic vector by utilizing a convolution network with a second resolution ratio to obtain a second basic characteristic sub-vector; wherein N and M are positive integers.
In a second example, performing the xth convolution processing on the basic feature vector by using a convolution network with a first resolution to obtain a first basic feature sub-vector; carrying out the Y-th convolution processing on the basic characteristic vector by utilizing a convolution network with the first resolution to obtain a second basic characteristic sub-vector; wherein X and Y are different positive integers.
In the embodiment of the application, in the semantic segmentation task of the image, it is very important to maintain the context relationship between pixels, so as to understand the relationship between pixels inside an object and the relationship between the object and the environment, and thus correctly classify the pixels. The semantic segmentation network comprises a pixel perception network, so that the semantic segmentation network can realize the highest performance with the least calculation amount.
In this embodiment, the first basic feature sub-vector or the second basic feature sub-vector may be further transposed, for example, a transposed feature component corresponding to the second basic feature sub-vector is generated according to the second basic feature sub-vector, then a first to-be-weighted feature vector corresponding to each basic feature sub-vector is generated according to the first basic feature sub-vector and the transposed feature component corresponding to the second basic feature sub-vector, and a plurality of to-be-processed feature vectors are generated according to each basic feature vector and the corresponding first to-be-weighted feature vector.
For example, as shown in fig. 4, a convolution operation is performed one or more times on a basic feature vector a by using a 3 × 3 convolution, so as to obtain a first basic feature sub-vector B and a second basic feature sub-vector C, a matrix multiplication is performed on a transpose (transpose) of the first basic feature sub-vector B and the second basic feature sub-vector C, the feature vector obtained by the matrix multiplication is normalized by, for example, a softmax classification function, so as to obtain a first feature vector S to be weighted, and finally, a feature vector E to be processed is generated according to the basic feature vector a and the corresponding first feature vector S to be weighted.
The calculation formula of the first to-be-weighted feature vector S is as follows:
finally, a calculation formula for generating the feature vector E to be processed according to the basic feature vector a and the corresponding first feature vector S to be weighted is as follows:
Ej＝σ(Sj)+Aj (2)
the sigma initialization is 0, then the weight of the sigma initialization is gradually increased, each pixel in the finally output feature vector E to be processed keeps semantic information of adjacent pixels, the continuity of the features is ensured, and meanwhile, the pixel perception network only has one convolution layer needing to be trained, so that the parameter quantity is extremely small, and the lightweight of the module is ensured.
In the embodiment of the present application, there are many ways to generate the target feature vector according to a plurality of feature vectors to be processed, which may be specifically selected according to practical applications, for example, as follows:
in a first example, a plurality of feature vectors to be processed are subjected to weighted summation to obtain a target feature vector.
In a second example, a target function is obtained according to a relationship between a plurality of historical to-be-processed feature vectors and a historical target feature vector, and the target feature vector is obtained by processing the plurality of to-be-processed feature vectors through the target function.
Furthermore, a classification network in the semantic segmentation network is utilized to process the target feature vector, and a semantic segmentation result of the image to be segmented is obtained, namely, classification prediction of the image to be segmented is realized, so that the purpose of semantic segmentation is achieved.
The image semantic segmentation method of the embodiment of the application acquires an image to be segmented, respectively extracts features of the image to be segmented by using a plurality of convolution networks with different resolutions in the semantic segmentation network, acquires a plurality of basic feature vectors, acquires a first basic feature sub-vector and a second basic feature sub-vector of each basic feature vector, generates a transposed feature component corresponding to the second basic feature sub-vector according to the second basic feature sub-vector, generates a first to-be-weighted feature vector corresponding to each basic feature vector according to the first basic feature sub-vector and the transposed feature component corresponding to the second basic feature sub-vector, generates a plurality of feature vectors to be processed according to each basic feature vector and the corresponding first to-be-weighted feature vector, generates a target feature vector according to the plurality of feature vectors to be processed, and processes the target feature vector by using a classification network in the semantic segmentation network, and obtaining a semantic segmentation result of the image to be segmented. Therefore, relevance learning and content understanding among pixels of local areas of the image are enhanced by introducing the pixel perception network, and accuracy of image semantic segmentation results is improved.
Based on the description of the above embodiments, it can be understood that the pixel-aware network or the boundary enhancement network can be selected according to the requirements of the application scenario, and the pixel-aware network and the boundary enhancement network are simultaneously selected to perform the image semantic segmentation processing, and how to perform the image semantic segmentation processing based on the boundary enhancement network is described in detail below with reference to fig. 5.
Fig. 5 is a schematic flow chart of an image semantic segmentation method according to a third embodiment of the present application.
As shown in fig. 5, the image semantic segmentation method may include the following steps:
In the embodiment of the application, the image to be segmented can be obtained by selecting and setting according to different application scenes, for example, in an automatic driving application scene, a picture within a preset distance in the driving process of a vehicle shot by a camera is subjected to pixel extraction and other preprocessing to be used as the image to be segmented; for example, in a disease diagnosis application scene, the image to be segmented is a picture taken by medical equipment, and is subjected to preprocessing such as pixel extraction and the like to be used as the image to be segmented.
In the embodiment of the present application, a plurality of convolution networks with different resolutions may be used to respectively perform feature extraction on an image to be segmented, and the semantic segmentation network of the present application has a lighter weight, so that the amount of computation and the complexity are greatly reduced, and therefore, in the embodiment of the present application, the selection of the convolution network may be as shown in fig. 2.
Specifically, as shown in fig. 2, the network includes 6 convolutional layers, the maximum number of channels is only 128, and compared with other networks which use dozens of convolutional layers and hundreds of channels, the network has the characteristic of being lighter in weight, and greatly reduces the amount of calculation and complexity.
And 304, generating a plurality of feature vectors to be processed according to each basic feature vector and the corresponding second feature vector to be weighted.
It can be understood that, in general, after the picture is processed by the multi-convolution layer and the pooling layer of the neural network, a certain amount of small object information and boundary information is lost, which easily causes some internal features of the object or the small object to be incorrectly identified as background features, and greatly reduces the performance of the algorithm. Therefore, the boundary enhancement network is added to retain the boundary information, so that the object segmentation is correctly realized, and the loss of the boundary information is reduced.
For example, as shown in fig. 6, the boundary enhancement network is different from the pixel-aware network, and directly multiplies the basic feature vector a by its transpose without any convolution operation, normalizes the obtained feature vector by a classification function such as a softmax function to obtain a second feature vector B to be weighted, and finally adds B to the basic feature vector a to obtain a final output feature vector E to be processed.
The calculation formula of the second feature vector S to be weighted is as follows:
and finally, generating a calculation formula of the feature vector E to be processed according to the basic feature vector A and the corresponding second feature vector S to be weighted as follows:
Ej＝ω(Bj)+Aj (2)
the omega initialization is 0, then the weight of the omega initialization is gradually increased, and the boundary enhancement network does not contain a convolution layer, so that the problem of losing boundary information and small object information does not exist, correct object segmentation is realized, the finally obtained output E ensures the integrity of the small object information and the boundary information, and meanwhile, the boundary enhancement network does not have any convolution layer needing to be learned, and the light weight of the boundary enhancement network is ensured.
And 305, generating a target feature vector according to the plurality of feature vectors to be processed, and processing the target feature vector by using a classification network in the semantic segmentation network to obtain a semantic segmentation result of the image to be segmented.
In the embodiment of the present application, there are many ways to generate the target feature vector according to a plurality of feature vectors to be processed, which may be specifically selected according to practical applications, for example, as follows:
in a first example, a plurality of feature vectors to be processed are subjected to weighted summation to obtain a target feature vector.
In a second example, a target function is obtained according to a relationship between a plurality of historical to-be-processed feature vectors and a historical target feature vector, and the target feature vector is obtained by processing the plurality of to-be-processed feature vectors through the target function.
Furthermore, a classification network in the semantic segmentation network is utilized to process the target feature vector, and a semantic segmentation result of the image to be segmented is obtained, namely, classification prediction of the image to be segmented is realized, so that the purpose of semantic segmentation is achieved.
The image semantic segmentation method comprises the steps of obtaining an image to be segmented, extracting features of the image to be segmented by using a plurality of convolution networks with different resolutions in a semantic segmentation network, obtaining a plurality of basic feature vectors, obtaining a transposed feature vector corresponding to each basic feature vector, generating a second feature vector to be weighted corresponding to each basic feature vector according to each basic feature vector and the corresponding transposed feature vector, generating a plurality of feature vectors to be processed according to each basic feature vector and the corresponding second feature vector to be weighted, generating a target feature vector according to the plurality of feature vectors to be processed, processing the target feature vector by using a classification network in the semantic segmentation network, and obtaining a semantic segmentation result of the image to be segmented. Therefore, accurate identification of small objects and boundary information is guaranteed by introducing a boundary enhancement network, and accuracy of image semantic segmentation results is improved.
Based on the description of the above embodiments, it can be understood that the pixel-aware network or the boundary enhancement network can be selected according to the requirements of the application scenario, and the pixel-aware network and the boundary enhancement network are simultaneously selected to perform the image semantic segmentation processing, and how to simultaneously select the pixel-aware network and the boundary enhancement network to perform the image semantic segmentation processing is described in detail below with reference to fig. 6.
Fig. 7 is a schematic flowchart of an image semantic segmentation method according to a fourth embodiment of the present application.
As shown in fig. 7, the image semantic segmentation method may include the following steps:
And step 402, performing content understanding and small object and boundary detection on each basic feature vector by utilizing a pixel perception network and a boundary enhancement network which are connected in parallel in each branch.
And 403, adding and fusing the feature vectors of each branch to calculate loss, acquiring target feature vectors, classifying, and acquiring a semantic segmentation result of the image to be segmented.
In the embodiment of the present application, convolution kernels with different resolutions are, for example, 1 × 1 convolution kernel, 3 × 3 convolution kernel, and 5 × 5 convolution kernel shown in fig. 8, the feature vectors to be processed of each branch are added and fused to obtain a target feature vector, and in the process of adding and fusing, different weight coefficients, such as 0.2, 0.3, and 0.5, may be assigned to different branches according to actual application requirements for calculating loss; wherein, the sum of the weight coefficients corresponding to all branches is 1.
Therefore, the regional content understanding capacity and the small object and boundary detection capacity of the network are realized by utilizing the pixel perception network and the boundary enhancement network which are connected in parallel in each branch, and finally, the characteristics of each branch are added and fused to calculate the loss to obtain the semantic segmentation result, so that the generalization capacity and the identification performance of the semantic segmentation network are improved, meanwhile, each network of the semantic segmentation network has few parameters, and the light weight of the whole network is ensured.
In the embodiment of the present application, one or more semantic segmentation networks may also be combined to perform multitask training, as shown in fig. 9, in the present application, a multitask learning manner is adopted, two semantic segmentation networks are unified in one integrated architecture, and a weighted sum of losses of the two semantic segmentation networks is used as a final loss of a model in a training process, so as to implement end-to-end training, as shown in the following formula:
Loss＝θ·Loss1+(1-θ)Loss2 (5)
therefore, a plurality of tasks simultaneously learn and optimize the network, the risk of network overfitting can be effectively reduced, the generalization effect is improved, and the semantic segmentation network can achieve higher performance.
As an example of a scene, taking a large-scale city landscape data set as an example, street scenes of 50 different cities are recorded, 5000 real scene images of driving scenes in a city environment are provided, and the scene has 2975 training sets, 500 verification sets and 1525 test sets, and contains 19 classes of dense pixel labels.
Specifically, based on the data set, the processing is performed through the pixel sensing network and the boundary enhancement network respectively, and the accuracy and the detection efficiency of the semantic segmentation result are determined to be improved to a certain extent.
Specifically, for example, a poly learning strategy (a fading learning rate strategy) is adopted, the basic learning rate is set to 0.001, the parameter power is set to 0.9, an ADAM (adaptive moment estimation) optimizer is adopted, the batch-size (the size of each batch of data) is set to 12, the beta (code table) is set to (0.9, 0.999), and the weight offset is set to 0.0005. In addition, the training set can be subjected to data expansion by adopting cutting, random turning, scale transformation and average interpolation, the image scaling is set to be {0.5,0.75,1,1.25 and 1.5}, and the semantic segmentation network is trained through a cross entropy loss function, so that the accuracy of subsequent recognition of the semantic segmentation network is further improved.
In order to further understand the context understanding ability of the pixel-aware network and the boundary enhancement network for the pixels of the object region and the detection ability for the object boundary and the small object, the pixel-aware network and the boundary enhancement network are not used, only the pixel-aware network or the boundary enhancement network is used, and the pixel-aware network and the boundary enhancement network are simultaneously used for analysis comparison with specific image semantic segmentation data, as shown in table 1.
TABLE 1 semantic segmentation result comparison information
Specifically, in fig. 10, the pixel-aware network and the boundary-enhanced network are not added, the accuracy of the semantic segmentation network is 64.8%, when the pixel-aware network is added alone, the accuracy of the semantic segmentation network is improved by 1.3%, and when the boundary-enhanced network is added alone, the accuracy of the semantic segmentation network is improved by 1.7%, so that both the pixel-aware network and the boundary-enhanced network can play a certain role in the network, and the performance of the semantic segmentation network is improved. In addition, when the pixel-aware network and the boundary enhancement network are used simultaneously, the accuracy of the semantic segmentation network is improved by 2.5%, and thus, the common combination of the two pixel-aware networks and the boundary enhancement network can further enhance the performance of the semantic segmentation network.
The convolutional network, the pixel perception network and/or the boundary enhancement network and the classification network in the semantic segmentation network in the embodiment of the application have the characteristics of small parameter and calculated amount and high detection speed, so that the method is suitable for being deployed on mobile equipment to complete tasks such as auxiliary driving and auxiliary diagnosis and treatment.
Therefore, the pixel perception network and the boundary enhancement network are provided, the context understanding ability of the network for the pixels of the object area and the detection ability for the object boundary and the small object are improved, the pixel perception network and the boundary enhancement network are applied to the semantic segmentation task, and a good effect is achieved. Therefore, all components in the network architecture cooperate with each other and act together, higher performance can be achieved under the condition of few parameter quantity, the network architecture is a practical network taking parameters and accuracy into consideration, and the network architecture is more suitable for being deployed on mobile equipment.
In order to implement the above embodiments, the present application further provides an image semantic segmentation apparatus.
Fig. 10 is a schematic structural diagram of an image semantic segmentation apparatus according to a fourth embodiment of the present application.
As shown in fig. 10, the image semantic segmentation apparatus 1000 may include: a first obtaining module 1010, a second obtaining module 1020, a processing module 1030, a generating module 1040, and a third obtaining module 1050.
The first obtaining module 1010 is configured to obtain an image to be segmented.
The second obtaining module 1020 is configured to perform feature extraction on the image to be segmented, and obtain a plurality of basic feature vectors.
A processing module 1030, configured to perform pixel sensing and/or boundary enhancement processing on the multiple basic feature vectors to generate multiple feature vectors to be processed.
The generating module 1040 is configured to generate a target feature vector according to the multiple feature vectors to be processed.
And a third obtaining module 1050, configured to perform classification processing on the target feature vector, and obtain a semantic segmentation result of the image to be segmented.
Further, in a possible implementation manner of the embodiment of the present application, the second obtaining module 1020 is specifically configured to: and respectively extracting the features of the image to be segmented by utilizing a plurality of convolution networks with different resolutions in the semantic segmentation network to obtain a plurality of basic feature vectors.
Further, in a possible implementation manner of the embodiment of the present application, performing pixel perception and/or boundary enhancement processing on a plurality of basic feature vectors to generate a plurality of feature vectors to be processed includes: and carrying out pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors by utilizing a pixel perception network and/or a boundary enhancement network in the semantic segmentation network to generate a plurality of feature vectors to be processed.
Further, in a possible implementation manner of the embodiment of the present application, the classifying the target feature vector to obtain a semantic segmentation result of the image to be segmented includes: and classifying the target characteristic vectors by utilizing a classification network in the semantic segmentation network to obtain a semantic segmentation result of the image to be segmented.
Further, in a possible implementation manner of the embodiment of the present application, referring to fig. 11, on the basis of the embodiment shown in fig. 10, the processing module 1030 includes: an acquisition unit 1031, a first generation unit 1032, a second generation unit 1033, and a third generation unit 1034.
An obtaining unit 1031, configured to obtain the first basic feature sub-vector and the second basic feature sub-vector of each basic feature vector.
A first generating unit 1032 is configured to generate a transposed feature component corresponding to the second primitive feature sub-vector according to the second primitive feature sub-vector.
A second generating unit 1033, configured to generate a first to-be-weighted feature vector corresponding to each basic feature vector according to the transposed feature component corresponding to the first basic feature sub-vector and the second basic feature sub-vector.
A third generating unit 1034, configured to generate a plurality of feature vectors to be processed according to each basic feature vector and the corresponding first feature vector to be weighted.
Further, in a possible implementation manner of the embodiment of the present application, the first generating unit 1032 is specifically configured to: carrying out convolution processing on the basic characteristic vector for the Nth time by utilizing a convolution network with the first resolution to obtain a first basic characteristic sub-vector; carrying out mth convolution processing on the basic characteristic vector by utilizing a convolution network with a second resolution ratio to obtain a second basic characteristic sub-vector; wherein N and M are positive integers.
Further, in a possible implementation manner of the embodiment of the present application, the processing module 1030 is specifically configured to: acquiring a transposed feature vector corresponding to each basic feature vector; generating a second feature vector to be weighted corresponding to each basic feature vector according to each basic feature vector and the corresponding transposed feature vector; and generating a plurality of feature vectors to be processed according to each basic feature vector and the corresponding second feature vector to be weighted.
It should be noted that the foregoing explanation on the embodiment of the image semantic segmentation method is also applicable to the image semantic segmentation apparatus of the embodiment, and is not repeated here.
The image semantic segmentation device of the embodiment of the application acquires an image to be segmented; extracting the features of the image to be segmented to obtain a plurality of basic feature vectors; carrying out pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed; generating a target feature vector according to a plurality of feature vectors to be processed; and classifying the target characteristic vectors to obtain a semantic segmentation result of the image to be segmented. Therefore, the target characteristic vectors generated after the pixel perception and/or the boundary enhancement processing are classified to obtain the semantic segmentation result, the relevance learning and the content understanding between the pixels of the local area of the image are enhanced, the accurate identification of small objects and boundary information is ensured, and the accuracy of the semantic segmentation result of the image is improved.
In order to implement the above embodiments, the present application also provides an electronic device, including: the image semantic segmentation method comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein when the processor executes the program, the image semantic segmentation method is realized according to the embodiment of the application.
In order to achieve the above embodiments, the present application further proposes a non-transitory computer-readable storage medium storing a computer program, which when executed by a processor implements the image semantic segmentation method as proposed by the foregoing embodiments of the present application.
In order to implement the foregoing embodiments, the present application also provides a computer program product, which when executed by an instruction processor in the computer program product, executes the image semantic segmentation method as set forth in the foregoing embodiments of the present application.
FIG. 12 illustrates a block diagram of an exemplary server suitable for use in implementing embodiments of the present application. The server 12 shown in fig. 12 is only an example, and should not bring any limitation to the function and the scope of use of the embodiments of the present application.
As shown in fig. 12, the server 12 is in the form of a general purpose computing device. The components of the server 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.
The server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by server 12 and includes both volatile and nonvolatile media, removable and non-removable media.
A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally perform the functions and/or methodologies of the embodiments described herein.
The server 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with the server 12, and/or with any devices (e.g., network card, modem, etc.) that enable the server 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 22. Moreover, the server 12 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public Network such as the Internet) via the Network adapter 20. As shown in FIG. 12, the network adapter 20 communicates with the other modules of the server 12 via the bus 18. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the server 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The processing unit 16 executes various functional applications and data processing, such as implementing the image semantic segmentation method mentioned in the foregoing embodiments, by running a program stored in the system memory 28.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present application in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.
Claims (16)
1. An image semantic segmentation method, comprising:
acquiring an image to be segmented;
extracting the features of the image to be segmented to obtain a plurality of basic feature vectors;
performing pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed;
generating a target feature vector according to the plurality of feature vectors to be processed;
and classifying the target characteristic vector to obtain a semantic segmentation result of the image to be segmented.
2. The image semantic segmentation method according to claim 1, wherein the extracting features of the image to be segmented to obtain a plurality of basic feature vectors includes:
and respectively extracting the features of the image to be segmented by utilizing a plurality of convolution networks with different resolutions in the semantic segmentation network to obtain a plurality of basic feature vectors.
3. The image semantic segmentation method according to claim 1, wherein the performing pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed includes:
and performing pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors by utilizing a pixel perception network and/or a boundary enhancement network in a semantic segmentation network to generate the plurality of feature vectors to be processed.
4. The image semantic segmentation method according to claim 3, wherein the performing pixel-aware processing on the plurality of basic feature vectors by using a pixel-aware network in the semantic segmentation network to generate a plurality of feature vectors to be processed includes:
acquiring a first basic feature sub-vector and a second basic feature sub-vector of each basic feature vector;
generating a transposed feature component corresponding to the second basic feature sub-vector according to the second basic feature sub-vector;
generating a first to-be-weighted feature vector corresponding to each basic feature vector according to the transposed feature components corresponding to the first basic feature sub-vector and the second basic feature sub-vector;
and generating the plurality of feature vectors to be processed according to each basic feature vector and the corresponding first feature vector to be weighted.
5. The image semantic segmentation method according to claim 4, wherein the obtaining of the first primitive feature sub-vector and the second primitive feature sub-vector of each primitive feature vector comprises:
performing convolution processing on the basic feature vector for the Nth time by using a convolution network with a first resolution ratio to obtain a first basic feature sub-vector;
performing convolution processing on the basic feature vector for the Mth time by using a convolution network with a second resolution to obtain a second basic feature sub-vector; wherein N and M are positive integers.
6. The image semantic segmentation method according to claim 3, wherein performing boundary enhancement processing on the plurality of basic feature vectors by using a boundary enhancement network in the semantic segmentation network to generate a plurality of feature vectors to be processed includes:
acquiring a transposed feature vector corresponding to each basic feature vector;
generating a second feature vector to be weighted corresponding to each basic feature vector according to each basic feature vector and the corresponding transposed feature vector;
and generating the plurality of feature vectors to be processed according to each basic feature vector and the corresponding second feature vector to be weighted.
7. The image semantic segmentation method according to claim 1, wherein the classifying the target feature vector to obtain the semantic segmentation result of the image to be segmented comprises:
and classifying the target characteristic vectors by utilizing a classification network in the semantic segmentation network to obtain a semantic segmentation result of the image to be segmented.
8. An image semantic segmentation apparatus, comprising:
the first acquisition module is used for acquiring an image to be segmented;
the second acquisition module is used for extracting the features of the image to be segmented to acquire a plurality of basic feature vectors;
the processing module is used for carrying out pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors to generate a plurality of feature vectors to be processed;
the generating module is used for generating a target characteristic vector according to the plurality of characteristic vectors to be processed;
and the third acquisition module is used for classifying the target characteristic vector to acquire a semantic segmentation result of the image to be segmented.
9. The image semantic segmentation apparatus according to claim 8, wherein the second obtaining module is specifically configured to:
and respectively extracting the features of the image to be segmented by utilizing a plurality of convolution networks with different resolutions in the semantic segmentation network to obtain a plurality of basic feature vectors.
10. The image semantic segmentation apparatus according to claim 8, wherein the processing module is specifically configured to:
and performing pixel perception and/or boundary enhancement processing on the plurality of basic feature vectors by utilizing a pixel perception network and/or a boundary enhancement network in a semantic segmentation network to generate the plurality of feature vectors to be processed.
11. The image semantic segmentation apparatus according to claim 10, wherein the processing module includes:
an obtaining unit, configured to obtain a first basic feature sub-vector and a second basic feature sub-vector of each basic feature vector;
a first generating unit, configured to generate a transposed feature component corresponding to the second basic feature sub-vector according to the second basic feature sub-vector;
a second generating unit, configured to generate a first to-be-weighted feature vector corresponding to each basic feature vector according to a transposed feature component corresponding to the first basic feature sub-vector and the second basic feature sub-vector;
a third generating unit, configured to generate the multiple feature vectors to be processed according to each basic feature vector and the corresponding first feature vector to be weighted.
12. The image semantic segmentation apparatus according to claim 11, wherein the first generation unit is specifically configured to:
performing convolution processing on the basic feature vector for the Nth time by using a convolution network with a first resolution ratio to obtain a first basic feature sub-vector;
performing convolution processing on the basic feature vector for the Mth time by using a convolution network with a second resolution to obtain a second basic feature sub-vector; wherein N and M are positive integers.
13. The image semantic segmentation apparatus according to claim 10, wherein the processing module is specifically configured to:
acquiring a transposed feature vector corresponding to each basic feature vector;
generating a second feature vector to be weighted corresponding to each basic feature vector according to each basic feature vector and the corresponding transposed feature vector;
and generating the plurality of feature vectors to be processed according to each basic feature vector and the corresponding second feature vector to be weighted.
14. The image semantic segmentation apparatus according to claim 8, wherein the third obtaining module is specifically configured to:
and classifying the target characteristic vectors by utilizing a classification network in the semantic segmentation network to obtain a semantic segmentation result of the image to be segmented.
15. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the image semantic segmentation method according to any one of claims 1 to 7 when executing the program.
16. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method for semantic segmentation of images according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011592663.9A CN113807354B (en) | 2020-12-29 | 2020-12-29 | Image semantic segmentation method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011592663.9A CN113807354B (en) | 2020-12-29 | 2020-12-29 | Image semantic segmentation method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113807354A true CN113807354A (en) | 2021-12-17 |
| CN113807354B CN113807354B (en) | 2023-11-03 |
Family
ID=78943582
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011592663.9A Active CN113807354B (en) | 2020-12-29 | 2020-12-29 | Image semantic segmentation method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113807354B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115170934A (en) * | 2022-09-05 | 2022-10-11 | 粤港澳大湾区数字经济研究院(福田) | Image segmentation method, system, equipment and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180129912A1 (en) * | 2016-11-07 | 2018-05-10 | Nec Laboratories America, Inc. | System and Method for Learning Random-Walk Label Propagation for Weakly-Supervised Semantic Segmentation |
| CN110245665A (en) * | 2019-05-13 | 2019-09-17 | 天津大学 | Image, semantic dividing method based on attention mechanism |
| CN111144418A (en) * | 2019-12-31 | 2020-05-12 | 北京交通大学 | Railway track area segmentation and extraction method |
| CN111582175A (en) * | 2020-05-09 | 2020-08-25 | 中南大学 | High-resolution remote sensing image semantic segmentation method sharing multi-scale countermeasure characteristics |
| CN111612802A (en) * | 2020-04-29 | 2020-09-01 | 杭州电子科技大学 | Re-optimization training method based on existing image semantic segmentation model and application |
| CN111666921A (en) * | 2020-06-30 | 2020-09-15 | 腾讯科技（深圳）有限公司 | Vehicle control method, apparatus, computer device, and computer-readable storage medium |
| CN111738310A (en) * | 2020-06-04 | 2020-10-02 | 科大讯飞股份有限公司 | Material classification method and device, electronic equipment and storage medium |
| CN111833273A (en) * | 2020-07-17 | 2020-10-27 | 华东师范大学 | Semantic boundary enhancement method based on long-distance dependence |
| US20200372660A1 (en) * | 2019-05-21 | 2020-11-26 | Beihang University | Image salient object segmentation method and apparatus based on reciprocal attention between foreground and background |
-
2020
- 2020-12-29 CN CN202011592663.9A patent/CN113807354B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180129912A1 (en) * | 2016-11-07 | 2018-05-10 | Nec Laboratories America, Inc. | System and Method for Learning Random-Walk Label Propagation for Weakly-Supervised Semantic Segmentation |
| CN110245665A (en) * | 2019-05-13 | 2019-09-17 | 天津大学 | Image, semantic dividing method based on attention mechanism |
| US20200372660A1 (en) * | 2019-05-21 | 2020-11-26 | Beihang University | Image salient object segmentation method and apparatus based on reciprocal attention between foreground and background |
| CN111144418A (en) * | 2019-12-31 | 2020-05-12 | 北京交通大学 | Railway track area segmentation and extraction method |
| CN111612802A (en) * | 2020-04-29 | 2020-09-01 | 杭州电子科技大学 | Re-optimization training method based on existing image semantic segmentation model and application |
| CN111582175A (en) * | 2020-05-09 | 2020-08-25 | 中南大学 | High-resolution remote sensing image semantic segmentation method sharing multi-scale countermeasure characteristics |
| CN111738310A (en) * | 2020-06-04 | 2020-10-02 | 科大讯飞股份有限公司 | Material classification method and device, electronic equipment and storage medium |
| CN111666921A (en) * | 2020-06-30 | 2020-09-15 | 腾讯科技（深圳）有限公司 | Vehicle control method, apparatus, computer device, and computer-readable storage medium |
| CN111833273A (en) * | 2020-07-17 | 2020-10-27 | 华东师范大学 | Semantic boundary enhancement method based on long-distance dependence |
Non-Patent Citations (2)
| Title |
|---|
| JUN FU ET AL: "Dual Attention Network for Scene Segmentation", ARXIV:1809.02983V4 * |
| 罗学刚;吕俊瑞;彭真明;: "超像素分割及评价的最新研究进展", 激光与光电子学进展, no. 09 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115170934A (en) * | 2022-09-05 | 2022-10-11 | 粤港澳大湾区数字经济研究院(福田) | Image segmentation method, system, equipment and storage medium |
| CN115170934B (en) * | 2022-09-05 | 2022-12-23 | 粤港澳大湾区数字经济研究院(福田) | Image segmentation method, system, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113807354B (en) | 2023-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11200424B2 (en) | Space-time memory network for locating target object in video content | |
| CN109086811B (en) | Multi-label image classification method and device and electronic equipment | |
| CN110782420A (en) | Small target feature representation enhancement method based on deep learning | |
| CN112465828A (en) | Image semantic segmentation method and device, electronic equipment and storage medium | |
| CN111210446B (en) | Video target segmentation method, device and equipment | |
| CN108961180B (en) | Infrared image enhancement method and system | |
| CN113591968A (en) | Infrared weak and small target detection method based on asymmetric attention feature fusion | |
| CN112329702B (en) | Method and device for rapid face density prediction and face detection, electronic equipment and storage medium | |
| CN111696110B (en) | Scene segmentation method and system | |
| CN111523463B (en) | Target tracking method and training method based on matching-regression network | |
| CN111126385A (en) | Deep learning intelligent identification method for deformable living body small target | |
| CN109977834B (en) | Method and device for segmenting human hand and interactive object from depth image | |
| Shu et al. | LVC-Net: Medical image segmentation with noisy label based on local visual cues | |
| CN112132164B (en) | Target detection method, system, computer device and storage medium | |
| Xie et al. | Deepmatcher: a deep transformer-based network for robust and accurate local feature matching | |
| CN111428664A (en) | Real-time multi-person posture estimation method based on artificial intelligence deep learning technology for computer vision | |
| GB2579262A (en) | Space-time memory network for locating target object in video content | |
| CN115205150A (en) | Image deblurring method, device, equipment, medium and computer program product | |
| CN114596440A (en) | Semantic segmentation model generation method and device, electronic equipment and storage medium | |
| CN111914949B (en) | Zero sample learning model training method and device based on reinforcement learning | |
| CN113807354B (en) | Image semantic segmentation method, device, equipment and storage medium | |
| CN114943834B (en) | Full-field Jing Yuyi segmentation method based on prototype queue learning under few labeling samples | |
| CN113762231B (en) | End-to-end multi-pedestrian posture tracking method and device and electronic equipment | |
| Pei et al. | FGO-Net: Feature and Gaussian Optimization Network for visual saliency prediction | |
| CN116670687A (en) | Method and system for adapting trained object detection models to domain offsets |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |