WO2023169334A1 - 图像的语义分割方法、装置、电子设备及存储介质 - Google Patents

图像的语义分割方法、装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2023169334A1
WO2023169334A1 PCT/CN2023/079602 CN2023079602W WO2023169334A1 WO 2023169334 A1 WO2023169334 A1 WO 2023169334A1 CN 2023079602 W CN2023079602 W CN 2023079602W WO 2023169334 A1 WO2023169334 A1 WO 2023169334A1
Authority
WO
WIPO (PCT)
Prior art keywords
model
loss
segmentation result
student model
student
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/079602
Other languages
English (en)
French (fr)
Inventor
吴捷
覃杰
肖学锋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Priority to US18/844,755 priority Critical patent/US20250182436A1/en
Publication of WO2023169334A1 publication Critical patent/WO2023169334A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/778—Active pattern-learning, e.g. online learning of image or video features
    • G06V10/7784—Active pattern-learning, e.g. online learning of image or video features based on feedback from supervisors
    • G06V10/7792—Active pattern-learning, e.g. online learning of image or video features based on feedback from supervisors the supervisor being an automated module, e.g. "intelligent oracle"
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Definitions

  • the present disclosure relates to the field of computer technology, for example, to methods, devices, electronic devices, and storage media for semantic segmentation of images.
  • Image semantic segmentation technology is a technology that uses semantic attributes as the dividing standard to achieve pixel-by-pixel classification prediction.
  • the depth and width of the semantic segmentation model are usually larger.
  • the depth of the model can be thought of as the number of network layers of the model
  • the width of the model can be thought of as the number of channels in each layer of the network.
  • the shortcomings of related technologies include at least the following: applying large-volume semantic segmentation models requires a large amount of resources at the cost, such as consuming a large amount of computing resources and deploying space resources. This poses a huge challenge to deploying semantic segmentation models to resource-constrained devices.
  • the present disclosure provides an image semantic segmentation method, device, electronic equipment and storage medium, which can use a lightweight model to achieve image semantic segmentation on the basis of ensuring the semantic segmentation effect, greatly reducing resource consumption and conducive to resource constraints. Model deployment on device.
  • the present disclosure provides a semantic segmentation method for images, including:
  • the image to be segmented is input into the student model; wherein the student model is trained according to the supervision information provided by the first teacher model and the second teacher model, and the depth of the first teacher model is greater than the depth of the student model and the depth of the third teacher model.
  • the depth of the second teacher model, the width of the second teacher model is greater than the width of the student model and the width of the first teacher model;
  • the semantic segmentation result of the image to be segmented is output based on the student model.
  • the present disclosure also provides a device for semantic segmentation of images, including:
  • the input module is configured to input the image to be segmented into the student model; wherein the student model is trained according to the supervision information provided by the first teacher model and the second teacher model, and the depth of the first teacher model is greater than that of the student model. depth and the depth of the second teacher model, the second teacher model The width is greater than the width of the student model and the width of the first teacher model;
  • An output module is configured to output the semantic segmentation result of the image to be segmented based on the student model.
  • the present disclosure also provides an electronic device, which includes:
  • processors one or more processors
  • a storage device configured to store one or more programs
  • the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the above-mentioned semantic segmentation method of images.
  • the present disclosure also provides a storage medium containing computer-executable instructions, which when executed by a computer processor are used to perform the above-mentioned semantic segmentation method of images.
  • the present disclosure also provides a computer program product, including a computer program carried on a non-transitory computer-readable medium, where the computer program includes program code for executing the above-mentioned semantic segmentation method for images.
  • Figure 1 is a schematic flowchart of an image semantic segmentation method provided by Embodiment 1 of the present disclosure
  • Figure 2 is a schematic flowchart of the training steps of the student model in a semantic segmentation method for images provided in Embodiment 2 of the present disclosure
  • Figure 3 is a schematic structural diagram of an image semantic segmentation device provided in Embodiment 4 of the present disclosure.
  • FIG. 4 is a schematic structural diagram of an electronic device provided by Embodiment 5 of the present disclosure.
  • the term “include” and its variations are open-ended, ie, “including but not limited to.”
  • the term “based on” means “based at least in part on.”
  • the term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; and the term “some embodiments” means “at least some embodiments”. Relevant definitions of other terms will be given in the description below.
  • Figure 1 is a schematic flowchart of a semantic segmentation method for images provided in Embodiment 1 of the present disclosure.
  • This embodiment of the present disclosure is suitable for semantic segmentation of images based on lightweight models.
  • This method can be performed by a semantic segmentation device for images, which can be implemented in the form of software and/or hardware, and which can be configured in electronic devices, such as mobile phones, computers and other electronic devices.
  • the semantic segmentation method of images provided by this embodiment may include:
  • the semantics and position coordinates of each object in the image can be obtained, so it has great practical value in many fields surrounding scene understanding.
  • the images to be segmented are different.
  • the image to be segmented can be a real-time road image.
  • real-time road images such as segmenting pedestrians and vehicles in the image
  • a solid foundation can be laid for autonomous driving tasks.
  • the semantic segmentation method of this embodiment can also perform semantic segmentation on images to be segmented corresponding to other fields, and this is not exhaustive here.
  • the student model can be considered as a shallower and narrower lightweight model; the first teacher model can be considered as a deeper and narrower large-volume model; and the second teacher model can be considered as a shallower and wider model.
  • large-scale model the first teacher model can be larger than the student model and the second teacher model in the depth dimension, and the depth of the second teacher model can be larger than the student model; the second teacher model can be larger than the student model and the first teacher model in the width dimension.
  • the width of the first teacher model may not be smaller than the student model.
  • the depth and width values of the first teacher model, the second teacher model and the student model can be set according to the actual application scenario.
  • the depth of the first teacher model may be 101 layers
  • the depth of the second teacher model may be 34 layers
  • the depth of the student model may be 17 layers
  • the width of the first teacher model may be equal to the student model, and may be half of the width of the second teacher model.
  • the first teacher model and the second teacher model are two complementary network structures. Deeper first teacher models can have better ability to extract high-level semantics and global classification abstractions, which helps achieve effective results in classification-oriented tasks. Wider second teacher models can be better at capturing diverse parts Content-aware information, which facilitates modeling contextual relationships between pixels. Supervising student model training based on these two complementary teacher models can provide comprehensive supervision information for student models from both deeper and wider dimensions. Deeper-dimensional supervision information can enhance the classification ability of the student model, and wider-dimensional supervision information can help the student model model the contextual relationship between pixels. By fully utilizing the performance advantages of large models to complete the process of knowledge distillation, the performance of lightweight student models can be greatly improved.
  • the first teacher model and the second teacher model may be pre-trained models, and the parameters are fixed during the training process of the student model.
  • the first teacher model and the second teacher model can be obtained through fully supervised training or semi-supervised training in advance. Since fully supervised training requires pre-annotation of massive pixel-level labels, the semi-supervised training method can be preferred to train the first teacher model and the second teacher model.
  • the semi-supervised training method can be considered as using a small number of labeled images and a large number of unlabeled images to train the first teacher model and the second teacher model.
  • the first teacher model and the second teacher model can be trained in advance, and the parameters of the trained first teacher model and the second teacher model can be fixed to perform a knowledge distillation process to improve the student model. performance.
  • the first teacher model and the second teacher model when using labeled data to train the student model, can also adjust parameters appropriately, so that the first teacher model and the second teacher model can be adjusted to a certain extent.
  • the teacher model achieves better supervision results when training the student model using unlabeled data.
  • the embodiment of the present disclosure improves the performance of the lightweight student model by providing complementary supervision information through two teacher models. It can enable the lightweight student model to achieve good semantic segmentation performance while ensuring that the Low resource consumption. Since the student model has a small amount of parameters and calculations, it can be easily deployed on resource-constrained devices.
  • the method before inputting the image to be segmented into the student model, the method further includes: in response to the remaining resource amount of the local device meeting the preset range, deploying the student model in the local device.
  • the resources of the electronic device deploying the segmentation model may be limited.
  • the computing resources of the mobile phone may be limited.
  • the electronic device before deploying the semantic segmentation model, the electronic device can obtain the remaining resources of the local device, such as the remaining amount of computing resources, the remaining amount of storage resources, etc. If the remaining resources of the local device meet the preset range, it can be considered that the current available resources of the local device Limited resources. At this time, the lightweight student model can be obtained and deployed to the local device, thereby enabling model deployment on resource-constrained devices.
  • the model selection that can be deployed is relatively wide, and the student model or the traditional semantic segmentation model provided in this embodiment can be deployed in the local device.
  • the image to be segmented is input into the student model; the student model is trained based on the supervision information provided by the first teacher model and the second teacher model, wherein the depth of the first teacher model is greater than the depth of the student model and the second teacher model.
  • the depth of the teacher model and the width of the second teacher model are greater than the width of the student model and the width of the first teacher model; the semantic segmentation results of the image to be segmented are output based on the student model.
  • the embodiments of the present disclosure can be combined with multiple solutions of the semantic segmentation method for images provided in the above embodiments.
  • the semantic segmentation method for images provided in this embodiment describes the steps of training a student model based on supervision information.
  • the global semantic loss, local feature loss, and consistency loss of the student model can be determined. Training the student model based on global semantic loss can help the student model learn to distinguish high-level semantic categories; training the student model based on local feature loss can help the student model capture the information of local detail texture of the image; training the student model based on consistency loss is beneficial to achieve the same input.
  • the results are consistent, thereby improving semantic segmentation accuracy.
  • the student model can be trained according to the following steps: based on the first teacher model, the second teacher model and the student model, output the first segmentation result, the second segmentation result and the third segmentation result of the sample image respectively; Determine the global semantic loss, local feature loss and consistency loss of the student model according to the first segmentation result, the second segmentation result and the third segmentation result; take the global semantic loss, the local feature loss and the consistency loss as Supervisory information is used to train the student model.
  • the global semantic loss can be considered as the difference in high-dimensional semantic features between the third segmentation result and the first segmentation result and/or the second segmentation result.
  • the segmentation results of multiple models for the same image will usually tend to be consistent. Since the first teacher model and the second teacher model perform better than the student model in terms of global classification abstraction ability and capturing diverse local features, it can be determined based on the difference between the third segmentation result and the first segmentation result and the second segmentation result. Loss of consistency.
  • training the student model based on global semantic loss can help the student model learn to distinguish high-level semantic categories; training the student model based on local feature loss can help the student model capture the information of local detail texture of the image; training the student model based on consistency loss has It is conducive to achieving consistency among multiple results of the same input, thereby improving semantic segmentation accuracy.
  • FIG. 2 is a schematic flowchart of the training steps of the student model in a semantic segmentation method for images provided in Embodiment 2 of the present disclosure.
  • the student model can be trained according to the following steps:
  • the first segmentation results of the sample image can be output respectively based on the first teacher model TD , the second teacher model TW and the student model S. Second segmentation result and the third segmentation result Y S .
  • the overall structure used in the training process is a three-branch network structure, consisting of two complementary large-scale teacher models and a lightweight student model.
  • the depth of the first teacher model T D (represented by Deep in the figure) is greater than the depth of the student model S and the depth of the second teacher model T W
  • the width of the second teacher model T W (represented by Wide in the figure) is greater than the student model S.
  • the width of model S and the width of first teacher model T D is a three-branch network structure, consisting of two complementary large-scale teacher models and a lightweight student model.
  • the first teacher model T D can provide the student model S with global semantic category abstractions, which is beneficial to the student model S's ability to learn classification; the second teacher model T W can extract richer local content perception by using a wider number of channels. Information, assists in supervising the student model S, and helps the student model S model the contextual relationship between pixels. That is, multi-granularity knowledge distillation from two complex teacher models to a simple student model can be achieved, which is beneficial to breaking through the learning ability bottleneck of lightweight models to ensure that the student model trained based on supervised information has better semantic segmentation effects.
  • the global semantic loss of the student model S can be determined based on the difference between the third segmentation result Y S and the first segmentation result Y TD (represented by Global Semantic-sensitive Loss in the figure).
  • This global semantic loss can be considered as the supervisory information provided by the first teacher model T D to the student model S, which can be used to characterize the difference in deeper high-dimensional semantic feature knowledge between the teacher model and the student model.
  • Determined with the second teacher model T W is used to generate the second segmentation result Feature image of The difference between them determines the local feature loss of the student model S (represented by Local Content-aware Loss in the figure).
  • This local feature loss can be considered as the supervisory information provided to the student model S by the second teacher model T W , which can be used to characterize the difference in the wider local context relationship between the teacher model and the student model.
  • the third segmentation result Y S can also be compared with the first segmentation result respectively. and the second segmentation result The difference determines the consistency loss of the student model S (represented by Complementary Consistency Loss in the figure).
  • This consistency loss can be considered as the supervision information provided by the first teacher model T D and the second teacher model T W to the student model S at the same time.
  • the first segmentation result and the second segmentation result The pixel values in multiple channel images can represent the probability value of the corresponding segmentation category.
  • the first pseudo label can be obtained by taking the maximum value of the pixel values of the multiple channel images. and the second pseudo label Pass the first pseudo tag and the second pseudo label It can assist in training the student model S.
  • the first segmentation result can be determined Second segmentation result and the third segmentation result Y S , these losses are calculated simultaneously; for another example, the characteristic image can be determined and First calculate the local feature loss, and then determine the first segmentation result Second segmentation result After the third segmentation result Y S , the global semantic loss and consistency loss are calculated.
  • local feature loss is used to assist in supervising the student network in the feature layer output by the decoder in the model
  • global semantic loss is used in the prediction output layer to improve the semantic category recognition ability of the student network, that is, multi-layer and multi-granularity are achieved.
  • the knowledge distillation scheme is used to train lightweight student models to achieve high performance and low computational load of student models.
  • the global semantic loss can be determined according to the following steps: convert the first segmentation result Perform channel-by-channel pooling (for example, channel-by-channel global average pooling (GAP)) with the third segmentation result Y S to obtain the first global vector respectively. and the second global vector will be the first global vector and the second global vector The sum of the differences in multiple dimensions is used as the global semantic loss of the student model S.
  • GAP channel-by-channel global average pooling
  • the first global vector can be determined by the following formula:
  • the global semantic loss can be determined by the following formula:
  • the value of the i-th dimension in; N represents the total number of segmentation categories.
  • global semantic loss allows the student model to attempt to learn higher-dimensional semantic category representations, which helps provide global guidance for the discrimination of semantic categories in semantic segmentation tasks.
  • the local feature loss is determined according to the following steps: the feature image determined by the second teacher model T W and the characteristic image determined by the student model S The feature difference is calculated channel by channel and pixel by pixel, and the local feature loss is determined based on multiple feature differences.
  • the local feature loss can be determined by the following formula:
  • C ⁇ H ⁇ W represents feature image and feature images size of
  • the local feature loss aims to take advantage of the channel of the wider teacher model to provide rich local contextual information, which can provide auxiliary supervision to guide the student model to model contextual relationships between pixels.
  • the consistency loss is determined according to the following steps: separate the third segmentation result Y S with the first segmentation result and the second segmentation result The sum of cross-entropy losses is used as the consistency loss of the student model.
  • the first pseudo label and the second pseudo label is based on the first segmentation result and the second segmentation result Definite pseudo-label. Accordingly, the consistency loss can be determined by the following formula:
  • the pixel values in the multiple channel images in Y S can represent the probability value of the corresponding segmentation category, and the prediction result Y can be obtained by taking the maximum value of the pixel values in the multiple channel images.
  • the consistency loss L Com (Y, Y p ) can be determined by Y and cross entropy loss with Y and cross entropy loss of and composition.
  • H ⁇ W can represent the prediction result and the total number of pixels of the two pseudo labels, y i , and can respectively represent the prediction result Y and the first pseudo label and the second pseudo label The predicted segmentation class of the i-th pixel in .
  • other kinds of inter-image losses can also be calculated to determine the consistency loss.
  • the consistency of multiple predictions for the same input can be maintained and the performance of the student model can be improved.
  • the technical solution of the embodiment of the present disclosure describes the steps of training a student model based on supervision information.
  • the global semantic loss, local feature loss, and consistency loss of the student model can be determined.
  • Training the student model based on global semantic loss can help the student model learn to distinguish high-level semantic categories; training the student model based on local feature loss can help the student model capture the information of local detail texture of the image; training the student model based on consistency loss is beneficial to achieve the same input.
  • the results are consistent, thereby improving semantic segmentation accuracy.
  • the semantic segmentation method of images provided by the embodiments of the present disclosure belongs to the same concept as the semantic segmentation method of images provided by the above-mentioned embodiments.
  • Technical details that are not described in detail in this embodiment can be referred to the above-mentioned embodiments, and the same technical features This embodiment has the same effect as the above-mentioned embodiment.
  • the embodiments of the present disclosure can be combined with multiple solutions of the semantic segmentation method for images provided in the above embodiments.
  • the semantic segmentation method of images provided in this embodiment supplements the supervision information when the sample image is a labeled sample image. By basing the difference between the segmentation results and labels of the student model, supervised learning of the student model can be achieved and the semantic segmentation accuracy of the student model can be improved.
  • the training step of the student model also includes: determining the supervision of the student model based on the difference between the third segmentation result and the label of the first sample image. Loss; accordingly, using global semantic loss, local feature loss, and consistency loss as supervision information to train the student model includes: using global semantic loss, local feature loss, consistency loss, and supervision loss as supervision information to train the student model.
  • the training method of the student model can be considered as fully supervised training; when the sample image contains both the labeled first sample image and the unlabeled third sample image
  • the training method of the student model can be considered as semi-supervised training.
  • the pseudo labels can be determined based on the prediction results output by the first teacher model and the second teacher model to train the student model.
  • the supervision loss can also be determined.
  • the student model can be trained in combination with the above losses to improve the accuracy of the student model.
  • the supervised loss can be determined according to the following steps: the cross-entropy loss of the third segmentation result and the label of the first sample image is used as the supervised loss of the student model.
  • the supervision loss can be determined by the following formula:
  • H ⁇ W can represent the prediction result Y and label
  • the total number of pixels, y i and The prediction result Y and label can be expressed separately The predicted segmentation class of the i-th pixel in .
  • other kinds of inter-image losses can also be calculated to determine the supervised loss.
  • the total loss of student model training can be expressed by the formula means; where L can be the total loss; It can represent the supervised loss corresponding to the labeled first sample image; It can represent the consistency loss corresponding to the labeled first sample image. Consistency loss corresponding to the unlabeled second sample image comprehensive losses; It can represent the global semantic loss corresponding to the unlabeled second sample image; It can represent the local feature loss corresponding to the unlabeled second sample image; ⁇ 1 and ⁇ 2 are the weighted parameters of the loss function, and these two parameters can be set based on empirical values or experimental values.
  • the technical solution of the embodiment of the present disclosure supplements the supervision information when the sample image is a labeled sample image.
  • the semantic segmentation method of images provided by the embodiments of the present disclosure belongs to the same concept as the semantic segmentation method of images provided by the above-mentioned embodiments.
  • Technical details that are not described in detail in this embodiment can be referred to the above-mentioned embodiments, and the same technical features This embodiment has the same effect as the above-mentioned embodiment.
  • FIG. 3 is a schematic structural diagram of an image semantic segmentation device provided in Embodiment 4 of the present disclosure.
  • the embodiments of the present disclosure are suitable for image semantic segmentation based on lightweight models.
  • the semantic segmentation device for images provided by this embodiment may include:
  • the input module 310 is configured to input the image to be segmented into the student model; the student model is trained according to the supervision information provided by the first teacher model and the second teacher model, where the depth of the first teacher model is greater than the depth of the student model and the second teacher model. The depth of the second teacher model is greater than the width of the student model and the width of the first teacher model; the output module 320 is configured to output the semantic segmentation result of the image to be segmented based on the student model.
  • the semantic segmentation device for images may include:
  • the model training module can be set to train the student model according to the following steps:
  • the first segmentation result, the second segmentation result and the third segmentation result of the sample image are respectively output; determined based on the first segmentation result, the second segmentation result and the third segmentation result.
  • Global semantic loss, local feature loss and consistency loss of the student model use global semantic loss, local feature loss and consistency loss as supervision information to train the student model.
  • the model training module can be set to:
  • the global semantic loss of the student model is determined; the feature image determined by the student model for generating the third segmentation result is compared with the feature image determined by the second teacher model for generating the second segmentation.
  • the difference between the resulting feature images determines the local feature loss of the student model; based on the difference between the third segmentation result and the first segmentation result and the second segmentation result, the consistency loss of the student model is determined.
  • the model training module can be configured to determine the global semantic loss according to the following steps:
  • the model training module can be configured to determine local feature loss according to the following steps:
  • the feature images determined by the second teacher model and the feature images determined by the student model are used to calculate feature differences channel by channel and pixel by pixel, and local feature losses are determined based on multiple feature differences.
  • the model training module can be configured to determine the consistency loss according to the following steps:
  • the third segmentation result is the sum of the cross-entropy losses of the first segmentation result and the second segmentation result, Consistency loss as a student model.
  • the model training module can also be set to:
  • the model training module can be set as:
  • Global semantic loss, local feature loss, consistency loss and supervision loss are used as supervision information to train the student model.
  • the model training module can be configured to determine the supervision loss according to the following steps:
  • the cross-entropy loss of the third segmentation result and the label of the first sample image is used as the supervised loss of the student model.
  • the first teacher model and the second teacher model are pre-trained models, and the parameters are fixed during the training process of the student model.
  • the image semantic segmentation device may also include:
  • the deployment module can be configured to deploy the student model in the local device in response to the remaining resource amount of the local device meeting the preset range before inputting the image to be segmented into the student model.
  • the device for semantic segmentation of images provided by the embodiments of the present disclosure can execute the semantic segmentation method of images provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method.
  • the multiple units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned divisions, as long as they can achieve the corresponding functions; in addition, the names of the multiple functional units are only for the convenience of distinguishing each other. , are not used to limit the protection scope of the embodiments of the present disclosure.
  • Terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (Personal Digital Assistant, PDA), tablet computers (Portable Android Device, PAD), portable multimedia players Mobile terminals such as (Portable Media Player, PMP), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital television (TV), desktop computers, etc.
  • PDA Personal Digital Assistant
  • PAD Portable Multimedia Players Mobile terminals
  • PMP Portable Multimedia Player
  • vehicle-mounted terminals such as vehicle-mounted navigation terminals
  • fixed terminals such as digital television (TV), desktop computers, etc.
  • TV digital television
  • the electronic device 400 shown in FIG. 4 is only an example and should not bring any limitations to the functions and usage scope of the embodiments of the present disclosure.
  • the electronic device 400 may include a processing device (such as a central processing unit, a graphics processor, etc.) 401, which may be stored in a read-only memory (Read-Only Memory, ROM) 402.
  • ROM Read-Only Memory
  • a variety of appropriate actions and processes are performed by the program in the program or the program loaded from the storage device 408 into the random access memory (Random Access Memory, RAM) 403 .
  • RAM Random Access Memory
  • various programs and data required for the operation of the electronic device 400 are also stored.
  • the processing device 401, ROM 402 and RAM 403 are connected to each other via a bus 404.
  • An input/output (I/O) interface 405 is also connected to bus 404.
  • the following devices can be connected to the I/O interface 405: input devices 406 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a Liquid Crystal Display (LCD) , an output device 407 such as a speaker, a vibrator, etc.; a storage device 408 including a magnetic tape, a hard disk, etc.; and a communication device 409.
  • the communication device 409 may allow the electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data.
  • FIG. 4 illustrates electronic device 400 with various means, implementation or availability of all illustrated means is not required. More or fewer means may alternatively be implemented or provided.
  • embodiments of the present disclosure include a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via communication device 409, or from storage device 408, or from ROM 402.
  • the processing device 401 When the computer program is executed by the processing device 401, the above-mentioned functions defined in the semantic segmentation method of images according to the embodiment of the present disclosure are performed.
  • Embodiments of the present disclosure provide a computer storage medium on which a computer program is stored.
  • the program is executed by a processor, the semantic segmentation method for images provided in the above embodiments is implemented.
  • the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • the computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof.
  • Examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard drives, RAM, ROM, Erasable Programmable Read-Only Memory (EPROM) ) or flash memory (FLASH), optical fiber, portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • a computer-readable storage medium may be any tangible medium that contains or stores a program that can be executed by instructions Use by or in connection with a system, apparatus or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above.
  • a computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device .
  • Program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequency (Radio Frequency, RF), etc., or any suitable combination of the above.
  • the client and server can communicate using any currently known or future developed network protocol, such as HyperText Transfer Protocol (HTTP), and can communicate with digital data in any form or medium.
  • HTTP HyperText Transfer Protocol
  • Data communications e.g., communications network
  • Examples of communication networks include Local Area Networks (LANs), Wide Area Networks (WANs), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any current network for knowledge or future research and development.
  • LANs Local Area Networks
  • WANs Wide Area Networks
  • the Internet e.g., the Internet
  • end-to-end networks e.g., ad hoc end-to-end networks
  • the above-mentioned computer-readable medium may be included in the above-mentioned electronic device; it may also exist independently without being assembled into the electronic device.
  • the above-mentioned computer-readable medium carries one or more programs.
  • the electronic device executes the above-mentioned one or more programs.
  • the depth of the first teacher model is greater than the depth of the student model and the depth of the second teacher model.
  • the second teacher model The width of the model is greater than the width of the student model and the width of the first teacher model; the semantic segmentation result of the image to be segmented is output based on the student model.
  • Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and Includes conventional procedural programming languages—such as "C" or similar programming languages.
  • the program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
  • the remote computer may be connected to the user computer through any kind of network, including a LAN or WAN, or may be connected to an external computer (eg, through the Internet using an Internet service provider).
  • each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration can be implemented by special purpose hardware-based systems that perform the specified functions or operations. , or can be implemented using a combination of specialized hardware and computer instructions.
  • the units involved in the embodiments of the present disclosure can be implemented in software or hardware. Among them, the name of a unit or module does not constitute a limitation on the unit or module itself.
  • exemplary types of hardware logic components include: field programmable gate array (Field Programmable Gate Array, FPGA), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), application specific standard product (Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programming Logic Device (CPLD), etc.
  • a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.
  • the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium.
  • Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing. Examples of machine-readable storage media would include an electrical connection based on one or more wires, a portable computer disk, a hard drive, RAM, ROM, EPROM or flash memory, optical fiber, CD-ROM, optical storage device, magnetic storage device, or Any suitable combination of the above.
  • Example 1 provides a semantic segmentation method for images, which method includes:
  • the image to be segmented is input into the student model; the student model is trained according to the supervision information provided by the first teacher model and the second teacher model, wherein the depth of the first teacher model is greater than the depth of the student model and the second teacher model.
  • the depth of the teacher model, the width of the second teacher model is greater than the width of the student model and the width of the first teacher model;
  • the semantic segmentation result of the image to be segmented is output based on the student model.
  • Example 2 provides a semantic segmentation of images Methods also include:
  • the student model is trained according to the following steps:
  • the second teacher model and the student model Based on the first teacher model, the second teacher model and the student model, output the first segmentation result, the second segmentation result and the third segmentation result of the sample image respectively;
  • the student model is trained using the global semantic loss, the local feature loss and the consistency loss as supervision information.
  • Example 3 provides a semantic segmentation method for images, further including:
  • the global semantic loss, local feature loss and consistency loss of the student model are determined based on the first segmentation result, the second segmentation result and the third segmentation result, including:
  • the student is determined Local feature loss of the model
  • the consistency loss of the student model is determined according to the differences between the third segmentation result and the first segmentation result and the second segmentation result respectively.
  • Example 4 provides a semantic segmentation method for images, further including:
  • the global semantic loss is determined according to the following steps:
  • the sum of the differences in multiple dimensions between the first global vector and the second global vector is used as the global semantic loss of the student model.
  • Example 5 provides a semantic segmentation method for images, further including:
  • the local feature loss is determined according to the following steps:
  • the feature images determined by the second teacher model and the feature images determined by the student model are used to obtain feature differences channel by channel and pixel by pixel, and local feature losses are determined based on multiple feature differences.
  • Example 6 provides a semantic segmentation method for images, further including:
  • the consistency loss is determined according to the following steps:
  • the sum of the cross-entropy losses of the third segmentation result and the first segmentation result and the second segmentation result is used as the consistency loss of the student model.
  • Example 7 provides a semantic segmentation method for images, further including:
  • the training step of the student model further includes:
  • the global semantic loss, the local feature loss and the consistency loss as supervision information to train the student model includes:
  • the student model is trained using the global semantic loss, the local feature loss, the consistency loss and the supervision loss as supervision information.
  • Example 8 provides a semantic segmentation method for images, further including:
  • the supervision loss is determined according to the following steps:
  • the cross-entropy loss between the third segmentation result and the label of the first sample image is used as the supervision loss of the student model.
  • Example 9 provides a semantic segmentation method for images, further including:
  • the first teacher model and the second teacher model are pre-trained models, and the parameters are fixed during the training process of the student model.
  • Example 10 provides a semantic segmentation method for images, further including:
  • the method before inputting the image to be segmented into the student model, the method further includes:
  • the student model is deployed in the local device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Biology (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Computational Linguistics (AREA)
  • Image Analysis (AREA)

Abstract

本公开提供了一种图像的语义分割方法、装置、电子设备及存储介质。该方法包括：将待分割图像输入学生模型；学生模型根据第一教师模型和第二教师模型提供的监督信息训练得到，其中第一教师模型的深度大于学生模型的深度和第二教师模型的深度，第二教师模型的宽度大于学生模型的宽度和第一教师模型的宽度；基于学生模型输出待分割图像的语义分割结果。

Description

图像的语义分割方法、装置、电子设备及存储介质
本申请要求在2022年03月09日提交中国专利局、申请号为202210225180.8的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本公开涉及计算机技术领域，例如涉及图像的语义分割方法、装置、电子设备及存储介质。
背景技术
图像语义分割技术是以语义属性作为划分标准实现逐像素分类预测的技术。
相关技术中，为保证语义分割效果，语义分割模型的深度、宽度通常较大。其中，模型的深度可以认为是模型的网络层数，模型的宽度可以认为是每层网络中的通道数。
相关技术的不足之处至少包括：应用大体量的语义分割模型需要以大量的资源作为代价，例如需要消耗大量计算资源和部署空间资源等。这对将语义分割模型部署至资源受限的设备带来巨大挑战。
发明内容
本公开提供了一种图像的语义分割方法、装置、电子设备及存储介质，能够在保证语义分割效果的基础上，采用轻量化模型实现图像语义分割，大大减少了资源耗费，有利于资源受限设备上的模型部署。
第一方面，本公开提供了一种图像的语义分割方法，包括：
将待分割图像输入学生模型；其中，所述学生模型根据第一教师模型和第二教师模型提供的监督信息训练得到，所述第一教师模型的深度大于所述学生模型的深度和所述第二教师模型的深度，所述第二教师模型的宽度大于所述学生模型的宽度和所述第一教师模型的宽度；
基于所述学生模型输出所述待分割图像的语义分割结果。
第二方面，本公开还提供了一种图像的语义分割装置，包括：
输入模块，设置为将待分割图像输入学生模型；其中，所述学生模型根据第一教师模型和第二教师模型提供的监督信息训练得到，所述第一教师模型的深度大于所述学生模型的深度和所述第二教师模型的深度，所述第二教师模型 的宽度大于所述学生模型的宽度和所述第一教师模型的宽度；
输出模块，设置为基于所述学生模型输出所述待分割图像的语义分割结果。
第三方面，本公开还提供了一种电子设备，所述电子设备包括：
一个或多个处理器；
存储装置，设置为存储一个或多个程序；
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现上述的图像的语义分割方法。
第四方面，本公开还提供了一种包含计算机可执行指令的存储介质，所述计算机可执行指令在由计算机处理器执行时用于执行上述的图像的语义分割方法。
第五方面，本公开还提供了一种计算机程序产品，包括承载在非暂态计算机可读介质上的计算机程序，所述计算机程序包含用于执行上述的图像的语义分割方法的程序代码。
附图说明
图1为本公开实施例一所提供的一种图像的语义分割方法的流程示意图；
图2为本公开实施例二所提供的一种图像的语义分割方法中学生模型的训练步骤的流程示意图；
图3为本公开实施例四所提供的一种图像的语义分割装置的结构示意图；
图4为本公开实施例五所提供的一种电子设备的结构示意图。
具体实施方式
下面将参照附图描述本公开的实施例。虽然附图中显示了本公开的一些实施例，然而本公开可以通过多种形式来实现，提供这些实施例是为了理解本公开。本公开的附图及实施例仅用于示例性作用。
本公开的方法实施方式中记载的多个步骤可以按照不同的顺序执行，和/或并行执行。此外，方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括，即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”；术语“另一实施例”表示“至少一个另外的实施例”；术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。
本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分，并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。
本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的，本领域技术人员应当理解，除非在上下文另有指出，否则应该理解为“一个或多个”。
实施例一
图1为本公开实施例一所提供的一种图像的语义分割方法流程示意图，本公开实施例适用于基于轻量化模型进行图像语义分割的情形。该方法可以由图像的语义分割装置来执行，该装置可以通过软件和/或硬件的形式实现，该装置可配置于电子设备中，例如配置于手机、电脑等电子设备中。
如图1所示，本实施例提供的图像的语义分割方法，可以包括：
S110、将待分割图像输入学生模型；学生模型根据第一教师模型和第二教师模型提供的监督信息训练得到，其中第一教师模型的深度大于学生模型的深度和第二教师模型的深度，第二教师模型的宽度大于学生模型的宽度和第一教师模型的宽度。
通过图像语义分割能够得到图像中每个对象的语义以及位置坐标，因此在围绕场景理解展开的诸多领域中具有巨大实用价值。针对不同领域，待分割图像各不相同。例如，在自动驾驶领域，待分割图像可以为实时的道路图像。通过对实时道路图像进行语义分割(例如对图像中的行人、车辆进行分割)，可以为自动驾驶任务奠定坚实的基础。此外，本实施例的语义分割方法也可以对其他领域对应的待分割图像进行语义分割，在此不做穷举。
本公开实施例中，学生模型可以认为是较浅、较窄的轻量化模型；第一教师模型可以认为是较深、较窄的大体量模型；第二教师模型可以认为是较浅、较宽的大体量模型。其中，第一教师模型在深度维度上可以大于学生模型和第二教师模型，且第二教师模型的深度可以大于学生模型；第二教师模型在宽度维度上可以大于学生模型和第一教师模型，且第一教师模型的宽度可以不小于学生模型。其中，第一教师模型、第二教师模型和学生模型的深度、宽度的数值可以根据实际应用场景进行设置。例如，第一教师模型的深度可以为101层，第二教师模型的深度可以为34层，以及学生模型的深度可以为17层等。又如，第一教师模型的宽度可以等于学生模型，且可以为第二教师模型的宽度的一半。
第一教师模型和第二教师模型为两个互补的网络结构。更深的第一教师模型可以具有更好地提取高级语义和全局分类抽象的能力，这有助于在面向分类的任务中取得有效的结果。更宽的第二教师模型可以更擅于捕获多样化的局部 内容感知信息，这有利于对像素之间的上下文关系进行建模。基于这两个互补的教师模型监督学生模型训练，可以从更深、更宽的两个维度为学生模型提供全面的监督信息。更深维度的监督信息可增强学生模型的分类能力，更宽维度的监督信息可帮助学生模型建模像素间的上下文关系。通过充分利用大模型的性能优势完成知识蒸馏的过程，能够大大提升轻量化的学生模型的表现。
在一些实现方式中，第一教师模型和第二教师模型可以为预先训练的模型，且在学生模型训练过程中参数固定。
第一教师模型和第二教师模型可以预先经过全监督训练或半监督训练得到。由于全监督训练需要预先标注海量像素级的标签，可优先选用半监督的训练方式训练第一教师模型和第二教师模型。半监督训练方式，可以认为是利用少量有标签的图像和大量无标签的图像，对第一教师模型和第二教师模型进行训练。在半监督训练过程中，可以采用生成无标签数据的伪标签的方式，和/或采用一致性正则的方式，利用无标签数据缩小标签数据较少而带来的性能衰减。
在这些实现方式中，可以预先对第一教师模型和第二教师模型进行训练，并可以固定训练完成的第一教师模型的参数和第二教师模型的参数，以执行知识蒸馏过程来改进学生模型的性能。此外，在一些其他实现方式中，当利用有标签数据训练学生模型时，第一教师模型和第二教师模型也可以适当的进行参数调整，从而可在一定程度上使第一教师模型和第二教师模型在利用无标签数据训练学生模型时达到更好的监督效果。
S120、基于学生模型输出待分割图像的语义分割结果。
如果将传统大体量的语义分割模型中的主干网络直接用简化网络替代，将使语义分割性能急剧下降。相较于该方式，本公开实施例通过两个教师模型提供互为补充的监督信息来提高轻量化学生模型的性能，能够使轻量化的学生模型实现很好的语义分割性能的同时，保证很低的资源消耗量。由于学生模型具有很小的参数量和计算量，可以很方便地部署到资源受限的设备上。
本公开实施例提供的图像的语义分割方法，在数据集上进行了广泛实验，表明该方法具有有效性，为轻量级语义分割模型的训练开辟了先河。
在一些实现方式中，在将待分割图像输入学生模型之前，还包括：响应于本端设备的资源剩余量符合预设范围，将学生模型部署于本端设备中。
在实际的分割场景中，部署分割模型的电子设备的资源可能有限，例如手机端的计算资源可能有限。在本实现方式中，电子设备在部署语义分割模型之前，可以获取本端设备的资源剩余量，例如计算资源剩余量、存储资源剩余量等。若本端设备的资源剩余量符合预设范围，则可以认为本端设备当前可用的 资源有限。此时，可以获取并部署轻量化的学生模型至本端设备，从而可以实现资源受限设备上的模型部署。
此外，若本端设备资源剩余量超出预设范围，则可以认为本端设备当前可用的资源充裕。此时可部署的模型选择较为广泛，可以将本实施例提供的学生模型或传统的语义分割模型部署于本端设备中。
本公开实施例的技术方案，将待分割图像输入学生模型；学生模型根据第一教师模型和第二教师模型提供的监督信息训练得到，其中第一教师模型的深度大于学生模型的深度和第二教师模型的深度，第二教师模型的宽度大于学生模型的宽度和第一教师模型的宽度；基于学生模型输出待分割图像的语义分割结果。通过利用更深、更宽的两个教师模型，为轻量化的学生模型提供不同方面的监督信息，能够实现从两个复杂模型到简单模型的知识蒸馏，可以保证基于监督信息训练的学生模型具有较佳的语义分割效果。并且，轻量化的学生模型可以大大减少资源耗费，有利于资源受限设备上的模型部署。
实施例二
本公开实施例与上述实施例中所提供的图像的语义分割方法中多个方案可以结合。本实施例所提供的图像的语义分割方法，对基于监督信息训练学生模型的步骤进行了描述。通过根据学生模型和第一教师模型、第二教师模型的分割结果，可以确定学生模型的全局语义损失、局部特征损失和一致性损失。基于全局语义损失训练学生模型可帮助学生模型学习判别高层语义类别；基于局部特征损失训练学生模型可帮助学生模型捕获图像局部细节纹理的信息；基于一致性损失训练学生模型有利于实现同一输入的多个结果保持一致，从而提高语义分割精度。
在一些实现方式中，学生模型可以根据下述步骤训练得到：基于第一教师模型、第二教师模型和学生模型，分别输出样本图像的第一分割结果、第二分割结果和第三分割结果；根据第一分割结果、第二分割结果和第三分割结果确定学生模型的全局语义损失，局部特征损失和一致性损失；将所述全局语义损失、所述局部特征损失和所述一致性损失作为监督信息，训练所述学生模型。
模型的深度越大，可以具有更好地提取高级语义和全局分类抽象的能力。由于第一教师模型在深度维度上大于学生模型，可以根据第一教师模型的分割结果，为学生模型的分割结果提供更多的高级语义。并且，若第二教师模型的深度也大于学生模型，则也可以根据第二教师模型的分割结果，为学生模型的分割结果提供更多的高级语义。全局语义损失即可以认为是，第三分割结果与第一分割结果和/或第二分割结果相比，在高维语义特征方面的差异。
模型的宽度越大，可以更擅于捕获多样化的局部内容感知信息。由于第二教师模型在宽度维度上大于学生模型，可以根据第二教师模型在生成分割结果过程中的特征图的局部特征，为学生模型在生成分割结果过程中的特征图提供更细节的局部内容感知信息。并且，若第一教室模型的宽度也大于学生模型，则也可以根据第一教师模型在生成分割结果过程中的特征图的局部特征，为学生模型在生成分割结果过程中的特征图提供更细节的局部内容感知信息。局部特征损失即可以认为是，在生成第三分割结果过程中的特征图，与在生成第一分割结果和/或第二分割结果过程中的特征图相比，在局部上下文关系方面的差异。
在模型具备一定准确率的情况下，多模型针对同一图像的分割结果通常会趋于一致。由于第一教师模型和第二教师模型分别在全局分类抽象能力、捕获多样化的局部特征方面表现优于学生模型，可以根据第三分割结果与第一分割结果和第二分割结果的差异，确定一致性损失。
在这些实现方式中，基于全局语义损失训练学生模型可帮助学生模型学习判别高层语义类别；基于局部特征损失训练学生模型可帮助学生模型捕获图像局部细节纹理的信息；基于一致性损失训练学生模型有利于实现同一输入的多个结果保持一致，从而提高语义分割精度。
示例性的，图2为本公开实施例二所提供的一种图像的语义分割方法中学生模型的训练步骤的流程示意图。如图2所示，在一些实现方式中，学生模型可以根据下述步骤训练得到：
首先，可以基于第一教师模型TD、第二教师模型TW和学生模型S，分别输出样本图像的第一分割结果第二分割结果和第三分割结果YS。
图2中可以看出，该训练过程采用的整体结构是一个三分支的网络结构，由两个互补的大体量教师模型和一个轻量化的学生模型组成。其中，第一教师模型TD的深度(图中用Deep表示)大于学生模型S的深度和第二教师模型TW的深度，第二教师模型TW的宽度(图中用Wide表示)大于学生模型S的宽度和第一教师模型TD的宽度。
第一教师模型TD可以为学生模型S提供全局的语义类别抽象概念，有利于学生模型S学习分类的能力；第二教师模型TW利用更宽的通道数可以提取出更丰富的局部内容感知信息，辅助监督学生模型S，有助于学生模型S建模像素间的上下文关系。即，可以实现从两个复杂教师模型到简单学生模型的多粒度知识蒸馏，有利于突破轻量级模型的学习能力瓶颈，以保证基于监督信息训练的学生模型具有较佳的语义分割效果。
然后，可以根据第三分割结果YS与第一分割结果YTD的差异，确定学生模型S的全局语义损失(图中用Global Semantic-sensitive Loss表示)。该全局语义损失可以认为是由第一教师模型TD提供给学生模型S的监督信息，可用于表征更深的教师模型和学生模型之间的高维语义特征知识的差异。
也可以根据学生模型S确定的用于生成第三分割结果YS的特征图像与第二教师模型TW确定的用于生成第二分割结果的特征图像之间的差异，确定学生模型S的局部特征损失(图中用Local Content-aware Loss表示)。该局部特征损失可以认为是由第二教师模型TW提供给学生模型S的监督信息，可用于表征更宽的教师模型和学生模型之间的局部上下文关系的差异。
还可以根据第三分割结果YS分别与第一分割结果和第二分割结果的差异，确定学生模型S的一致性损失(图中用Complementary Consistency Loss表示)。该一致性损失可以认为是第一教师模型TD和第二教师模型TW同时提供给学生模型S的监督信息。其中，第一分割结果和第二分割结果中多个通道图像中的像素值可表征对应分割类别的概率值，可以通过多个通道图像的像素值取最大值的方式，分别得到第一伪标签和第二伪标签通过第一伪标签和第二伪标签可辅助训练学生模型S。
上述全局语义损失、局部特征损失和一致性损失的求取步骤并无严格时序限制。例如，可以在确定出第一分割结果第二分割结果和第三分割结果YS之后，同步计算该些损失；又如，可以在确定出特征图像和时先计算局部特征损失，在确定出第一分割结果第二分割结果和第三分割结果YS之后，再计算全局语义损失和一致性损失等。
最后，将全局语义损失、局部特征损失和一致性损失作为监督信息，训练学生模型S。
本公开实施例中，在模型内解码器输出的特征层，使用局部特征损失辅助监督学生网络，在预测输出层使用全局语义损失提高学生网络的语义类别识别能力，即实现了多层、多粒度的知识蒸馏方案来训练轻量化的学生模型，以实现学生模型的高性能和低计算量。
再次参见图2，在一些实现方式中，全局语义损失可以根据下述步骤确定：将第一分割结果和第三分割结果YS进行逐通道池化(例如可以为逐通道全局平均池化(Global Average Pooling，GAP))，分别得到第一全局向量和第二全局向量将第一全局向量和第二全局向量间多个维度的差值之和，作为学生模型S的全局语义损失。
可以通过下述公式来确定第一全局向量：
其中，表示第一分割结果的图像尺寸为N×H×W；N表示通道数量，H表示图像高度，W表示图像宽度，且后续相同格式上标所表示的含义与此处相同，将不再赘述。其中，G(·)表示逐通道的全局平均池化运算，且运算得到的可表示N个分割类别的全局语义类别向量。基于同种运算方式，可确定第二全局向量
可通过下述公式来确定全局语义损失：
其中，表示全局语义损失；和可以分别表示第二全局向量和第一全局向量中第i个维度的数值；N表示分割类别的总数量。
在这些实现方式中，通过全局语义损失可使学生模型尝试学习更高维的语义类别表示，这有助于在语义分割任务中为语义类别的判别提供全局指导。
再次参见图2，在一些实现方式中，局部特征损失根据下述步骤确定：将第二教师模型TW确定的特征图像和学生模型S确定的特征图像进行逐通道、逐像素求取特征差值，并根据多个特征差值确定局部特征损失。
可以通过下述公式确定局部特征损失：
其中，表示局部特征损失；C×H×W表示特征图像和特征图像的尺寸；和分别表示特征图像和特征图像在第i个通道、第j个高度、第q个宽度上像素的特征值。
在这些实现方式中，局部特征损失旨在利用更宽的教师模型的通道优势来提供丰富的局部上下文信息，它可以提供辅助监督来指导学生模型对像素之间的上下文关系进行建模。
再次参见图2，在一些实现方式中，一致性损失根据下述步骤确定：将第三分割结果YS分别与第一分割结果和第二分割结果的交叉熵损失之和，作为学生模型的一致性损失。
图中，第一伪标签和第二伪标签为基于第一分割结果和第二分割结果确定的伪标签。相应的，可以通过下述公式来确定一致性损失：

其中，YS中多个通道图像中的像素值可表征对应分割类别的概率值，可以通过多个通道图像的像素值取最大值的方式，得到预测结果Y。其中，一致性损失LCom(Y，Yp)可以由Y和的交叉熵损失与Y和的交叉熵损失的和构成。其中，H×W可以表示预测结果和两个伪标签的像素总数，yi、和可分别表示预测结果Y、第一伪标签和第二伪标签中第i个像素的预测的分割类别。此外，除计算Y分别与和的交叉熵损失之外，还可计算其他种类的图像间损失，以确定一致性损失。
在这些实现方式中，通过计算第一教师模型、第二教师模型和学生模型之间的互补的一致性损失，可以保持同一输入的多个预测的一致性，提高学生模型的性能。
本公开实施例的技术方案，对基于监督信息训练学生模型的步骤进行了描述。通过根据学生模型和第一教师模型、第二教师模型的分割结果，可以确定学生模型的全局语义损失、局部特征损失和一致性损失。基于全局语义损失训练学生模型可帮助学生模型学习判别高层语义类别；基于局部特征损失训练学生模型可帮助学生模型捕获图像局部细节纹理的信息；基于一致性损失训练学生模型有利于实现同一输入的多个结果保持一致，从而提高语义分割精度。
此外，本公开实施例提供的图像的语义分割方法与上述实施例提供的图像的语义分割方法属于同一构思，未在本实施例中详尽描述的技术细节可参见上述实施例，并且相同的技术特征在本实施例与上述实施例中具有相同的效果。
实施例三
本公开实施例与上述实施例中所提供的图像的语义分割方法中多个方案可以结合。本实施例所提供的图像的语义分割方法，对样本图像为有标签样本图像时的监督信息进行了补充。通过根据学生模型的分割结果与标签之间的差异，可以实现学生模型的有监督学习，提高学生模型的语义分割精度。
在一些实现方式中，若样本图像包括有标签的第一样本图像，则学生模型的训练步骤，还包括：根据第三分割结果与第一样本图像的标签的差异，确定学生模型的监督损失；相应的，将全局语义损失、局部特征损失和一致性损失作为监督信息，训练学生模型，包括：将全局语义损失、局部特征损失、一致性损失和监督损失作为监督信息，训练学生模型。
当样本图像仅包含有标签的第一样本图像时，学生模型的训练方式可认为是全监督训练；当样本图像既包含有标签的第一样本图像，又包含无标签的第 二样本图像时，学生模型的训练方式可认为是半监督训练。以半监督训练方式训练学生模型时，可以根据第一教师模型和第二教师模型输出的预测结果确定伪标签，对学生模型进行训练。
当样本图像包含第一样本图像时，除了可以确定全局语义损失、局部特征损失和一致性损失之外，还可以确定监督损失。并且，可以结合上述损失训练学生模型，以提高学生模型精度。
监督损失根据下述步骤可以确定：将第三分割结果与第一样本图像的标签的交叉熵损失，作为学生模型的监督损失。可以通过下述公式来确定监督损失：
其中，可以表示学生模型的预测结果Y与第一样本图像的标签间的监督损失，且上标l可表示该监督损失在有标签的样本数据时确定。其中，H×W可以表示预测结果Y和标签的像素总数，yi和可分别表示预测结果Y和标签中第i个像素的预测的分割类别。此外，除计算Y与的交叉熵损失之外，还可计算其他种类的图像间损失，以确定监督损失。
此外，针对第一样本图像和第二样本图像，可以从上述损失中分别选用不同的损失训练学生模型。
示例性的，采用第一样本图像训练学生模型时，可以选用监督损失和一致性损失训练学生模型；采用第二样本图像训练学生模型时，可以选用全局语义损失、局部特征损失和一致性损失训练学生模型。
此时，学生模型训练的总损失可以用公式 表示；其中，L可以为总损失；可以表示有标签的第一样本图像对应的监督损失；可以表示有标签的第一样本图像对应的一致性损失和无标签的第二样本图像对应的一致性损失的综合损失；可以表示无标签的第二样本图像对应的全局语义损失；可以表示无标签的第二样本图像对应的局部特征损失；λ1和λ2是损失函数的加权参数，且该两参数可以根据经验值或实验值进行设置。
本公开实施例的技术方案，对样本图像为有标签样本图像时的监督信息进行了补充。通过根据学生模型的分割结果与标签之间的差异，可以实现学生模型的有监督学习，提高学生模型的语义分割精度。此外，本公开实施例提供的图像的语义分割方法与上述实施例提供的图像的语义分割方法属于同一构思，未在本实施例中详尽描述的技术细节可参见上述实施例，并且相同的技术特征在本实施例与上述实施例中具有相同的效果。
实施例四
图3为本公开实施例四所提供的一种图像的语义分割装置的结构示意图。本公开实施例适用于基于轻量化模型进行图像语义分割的情形。
如图3所示，本实施例提供的图像的语义分割装置，可以包括：
输入模块310，设置为将待分割图像输入学生模型；学生模型根据第一教师模型和第二教师模型提供的监督信息训练得到，其中第一教师模型的深度大于学生模型的深度和第二教师模型的深度，第二教师模型的宽度大于学生模型的宽度和第一教师模型的宽度；输出模块320，设置为基于学生模型输出待分割图像的语义分割结果。
在一些实现方式中，图像的语义分割装置，可以包括：
模型训练模块，可以设置为根据下述步骤训练得到学生模型：
基于第一教师模型、第二教师模型和学生模型，分别输出样本图像的第一分割结果、第二分割结果和第三分割结果；根据第一分割结果、第二分割结果和第三分割结果确定学生模型的全局语义损失，局部特征损失和一致性损失；将全局语义损失、局部特征损失和一致性损失作为监督信息，训练学生模型。
在一些实现方式中，模型训练模块，可以设置为：
根据第三分割结果与第一分割结果的差异，确定学生模型的全局语义损失；根据学生模型确定的用于生成第三分割结果的特征图像，与第二教师模型确定的用于生成第二分割结果的特征图像之间的差异，确定学生模型的局部特征损失；根据第三分割结果分别与第一分割结果和第二分割结果的差异，确定学生模型的一致性损失。
在一些实现方式中，模型训练模块，可以设置为根据下述步骤确定全局语义损失：
将第一分割结果和第三分割结果进行逐通道池化，分别得到第一全局向量和第二全局向量；将第一全局向量和第二全局向量间多个维度的差值之和，作为学生模型的全局语义损失。
在一些实现方式中，模型训练模块，可以设置为根据下述步骤确定局部特征损失：
将第二教师模型确定的特征图像和学生模型确定的特征图像进行逐通道、逐像素求取特征差值，并根据多个特征差值确定局部特征损失。
在一些实现方式中，模型训练模块，可以设置为根据下述步骤确定一致性损失：
将第三分割结果分别与第一分割结果和第二分割结果的交叉熵损失之和， 作为学生模型的一致性损失。
在一些实现方式中，若样本图像包括有标签的第一样本图像，则模型训练模块，还可以设置为：
根据第三分割结果与第一样本图像的标签的差异，确定学生模型的监督损失；相应的，模型训练模块，可以设置为：
将全局语义损失、局部特征损失、一致性损失和监督损失作为监督信息，训练学生模型。
在一些实现方式中，模型训练模块，可以设置为根据下述步骤确定监督损失：
将第三分割结果与第一样本图像的标签的交叉熵损失，作为学生模型的监督损失。
在一些实现方式中，第一教师模型和第二教师模型为预先训练的模型，且在学生模型训练过程中参数固定。
在一些实现方式中，图像的语义分割装置，还可以包括：
部署模块，可以设置为在将待分割图像输入学生模型之前，响应于本端设备的资源剩余量符合预设范围，将学生模型部署于本端设备中。
本公开实施例所提供的图像的语义分割装置，可执行本公开任意实施例所提供的图像的语义分割方法，具备执行方法相应的功能模块和有益效果。
上述装置所包括的多个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，多个功能单元的名称也只是为了便于相互区分，并不用于限制本公开实施例的保护范围。
实施例五
下面参考图4，其示出了适于用来实现本公开实施例的电子设备(例如图4中的终端设备或服务器)400的结构示意图。本公开实施例中的终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、个人数字助理(Personal Digital Assistant，PDA)、平板电脑(Portable Android Device，PAD)、便携式多媒体播放器(Portable Media Player，PMP)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字电视(Television，TV)、台式计算机等等的固定终端。图4示出的电子设备400仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图4所示，电子设备400可以包括处理装置(例如中央处理器、图形处理器等)401，其可以根据存储在只读存储器(Read-Only Memory，ROM)402 中的程序或者从存储装置408加载到随机访问存储器(Random Access Memory，RAM)403中的程序而执行多种适当的动作和处理。在RAM 403中，还存储有电子设备400操作所需的多种程序和数据。处理装置401、ROM 402以及RAM 403通过总线404彼此相连。输入/输出(Input/Output，I/O)接口405也连接至总线404。
通常，以下装置可以连接至I/O接口405：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置406；包括例如液晶显示器(Liquid Crystal Display，LCD)、扬声器、振动器等的输出装置407；包括例如磁带、硬盘等的存储装置408；以及通信装置409。通信装置409可以允许电子设备400与其他设备进行无线或有线通信以交换数据。虽然图4示出了具有多种装置的电子设备400，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置409从网络上被下载和安装，或者从存储装置408被安装，或者从ROM 402被安装。在该计算机程序被处理装置401执行时，执行本公开实施例的图像的语义分割方法中限定的上述功能。
本公开实施例提供的电子设备与上述实施例提供的图像的语义分割方法属于同一构思，未在本实施例中详尽描述的技术细节可参见上述实施例，并且本实施例与上述实施例具有相同的效果。
实施例六
本公开实施例提供了一种计算机存储介质，其上存储有计算机程序，该程序被处理器执行时实现上述实施例所提供的图像的语义分割方法。
本公开上述的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、RAM、ROM、可擦式可编程只读存储器(Erasable Programmable Read-Only Memory，EPROM)或闪存(FLASH)、光纤、便携式紧凑磁盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行 系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：电线、光缆、射频(Radio Frequency，RF)等等，或者上述的任意合适的组合。
在一些实施方式中，客户端、服务器可以利用诸如超文本传输协议(Hyper Text Transfer Protocol，HTTP)之类的任何当前已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字数据通信(例如，通信网络)互连。通信网络的示例包括局域网(Local Area Network，LAN)，广域网(Wide Area Network，WAN)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端对端网络)，以及任何当前已知或未来研发的网络。
上述计算机可读介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被该电子设备执行时，使得该电子设备：
将待分割图像输入学生模型；学生模型根据第一教师模型和第二教师模型提供的监督信息训练得到，其中第一教师模型的深度大于学生模型的深度和第二教师模型的深度，第二教师模型的宽度大于学生模型的宽度和第一教师模型的宽度；基于学生模型输出待分割图像的语义分割结果。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的计算机程序代码，上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括LAN或WAN—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开多种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图 中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，单元、模块的名称在一种情况下并不构成对该单元、模块本身的限定。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(Field Programmable Gate Array，FPGA)、专用集成电路(Application Specific Integrated Circuit，ASIC)、专用标准产品(Application Specific Standard Parts，ASSP)、片上系统(System on Chip，SOC)、复杂可编程逻辑设备(Complex Programming Logic Device，CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、RAM、ROM、EPROM或快闪存储器、光纤、CD-ROM、光学储存设备、磁储存设备、或上述内容的任何合适组合。
根据本公开的一个或多个实施例，【示例一】提供了一种图像的语义分割方法，该方法包括：
将待分割图像输入学生模型；所述学生模型根据第一教师模型和第二教师模型提供的监督信息训练得到，其中所述第一教师模型的深度大于所述学生模型的深度和所述第二教师模型的深度，所述第二教师模型的宽度大于所述学生模型的宽度和所述第一教师模型的宽度；
基于所述学生模型输出所述待分割图像的语义分割结果。
根据本公开的一个或多个实施例，【示例二】提供了一种图像的语义分割 方法，还包括：
在一些实现方式中，所述学生模型根据下述步骤训练得到：
基于所述第一教师模型、所述第二教师模型和所述学生模型，分别输出样本图像的第一分割结果、第二分割结果和第三分割结果；
根据所述第一分割结果、第二分割结果和第三分割结果确定学生模型的全局语义损失，局部特征损失和一致性损失；
将所述全局语义损失、所述局部特征损失和所述一致性损失作为监督信息，训练所述学生模型。
根据本公开的一个或多个实施例，【示例三】提供了一种图像的语义分割方法，还包括：
在一些实现方式中，所述根据所述第一分割结果、第二分割结果和第三分割结果确定学生模型的全局语义损失，局部特征损失和一致性损失，包括：
根据所述第三分割结果与所述第一分割结果的差异，确定所述学生模型的全局语义损失；
根据所述学生模型确定的用于生成所述第三分割结果的特征图像，与所述第二教师模型确定的用于生成所述第二分割结果的特征图像之间的差异，确定所述学生模型的局部特征损失；
根据所述第三分割结果分别与所述第一分割结果和所述第二分割结果的差异，确定所述学生模型的一致性损失。
根据本公开的一个或多个实施例，【示例四】提供了一种图像的语义分割方法，还包括：
在一些实现方式中，所述全局语义损失根据下述步骤确定：
将所述第一分割结果和所述第三分割结果进行逐通道池化，分别得到第一全局向量和第二全局向量；
将所述第一全局向量和所述第二全局向量间多个维度的差值之和，作为所述学生模型的全局语义损失。
根据本公开的一个或多个实施例，【示例五】提供了一种图像的语义分割方法，还包括：
在一些实现方式中，所述局部特征损失根据下述步骤确定：
将所述第二教师模型确定的特征图像和所述学生模型确定的特征图像进行逐通道、逐像素求取特征差值，并根据多个特征差值确定局部特征损失。
根据本公开的一个或多个实施例，【示例六】提供了一种图像的语义分割方法，还包括：
在一些实现方式中，所述一致性损失根据下述步骤确定：
将所述第三分割结果分别与所述第一分割结果和所述第二分割结果的交叉熵损失之和，作为所述学生模型的一致性损失。
根据本公开的一个或多个实施例，【示例七】提供了一种图像的语义分割方法，还包括：
在一些实现方式中，若所述样本图像包括有标签的第一样本图像，则所述学生模型的训练步骤，还包括：
根据所述第三分割结果与所述第一样本图像的标签的差异，确定所述学生模型的监督损失；
相应的，所述将所述全局语义损失、所述局部特征损失和一致性损失作为监督信息，训练所述学生模型，包括：
将所述全局语义损失、所述局部特征损失、所述一致性损失和所述监督损失作为监督信息，训练所述学生模型。
根据本公开的一个或多个实施例，【示例八】提供了一种图像的语义分割方法，还包括：
在一些实现方式中，所述监督损失根据下述步骤确定：
将所述第三分割结果与所述第一样本图像的标签的交叉熵损失，作为所述学生模型的监督损失。
根据本公开的一个或多个实施例，【示例九】提供了一种图像的语义分割方法，还包括：
在一些实现方式中，所述第一教师模型和所述第二教师模型为预先训练的模型，且在所述学生模型训练过程中参数固定。
根据本公开的一个或多个实施例，【示例十】提供了一种图像的语义分割方法，还包括：
在一些实现方式中，在所述将待分割图像输入学生模型之前，还包括：
响应于本端设备的资源剩余量符合预设范围，将所述学生模型部署于所述本端设备中。
此外，虽然采用特定次序描绘了多个操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和 并行处理可能是有利的。同样地，虽然在上面论述中包含了多个实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的一些特征还可以组合地实现在单个实施例中。相反地，在单个实施例的上下文中描述的多种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。

Claims (14)

  1. 一种图像的语义分割方法，包括：
    将待分割图像输入学生模型；其中，所述学生模型根据第一教师模型和第二教师模型提供的监督信息训练得到，所述第一教师模型的深度大于所述学生模型的深度和所述第二教师模型的深度，所述第二教师模型的宽度大于所述学生模型的宽度和所述第一教师模型的宽度；
    基于所述学生模型输出所述待分割图像的语义分割结果。
  2. 根据权利要求1所述的方法，其中，所述学生模型根据下述方法训练得到：
    基于所述第一教师模型、所述第二教师模型和所述学生模型，分别输出样本图像的第一分割结果、第二分割结果和第三分割结果；
    根据所述第一分割结果、所述第二分割结果和所述第三分割结果确定所述学生模型的全局语义损失，局部特征损失和一致性损失；
    将所述全局语义损失、所述局部特征损失和所述一致性损失作为监督信息，训练所述学生模型。
  3. 根据权利要求2所述的方法，其中，所述根据所述第一分割结果、所述第二分割结果和所述第三分割结果确定所述学生模型的全局语义损失，局部特征损失和一致性损失，包括：
    根据所述第三分割结果与所述第一分割结果的差异，确定所述学生模型的全局语义损失；
    根据所述学生模型确定的用于生成所述第三分割结果的特征图像，与所述第二教师模型确定的用于生成所述第二分割结果的特征图像之间的差异，确定所述学生模型的局部特征损失；
    根据所述第三分割结果分别与所述第一分割结果和所述第二分割结果的差异，确定所述学生模型的一致性损失。
  4. 根据权利要求3所述的方法，其中，所述根据所述第三分割结果与所述第一分割结果的差异，确定所述学生模型的全局语义损失，包括：
    将所述第一分割结果和所述第三分割结果进行逐通道池化，分别得到第一全局向量和第二全局向量；
    将所述第一全局向量和所述第二全局向量间多个维度的差值之和，作为所述学生模型的全局语义损失。
  5. 根据权利要求3所述的方法，其中，所述根据所述学生模型确定的用于 生成所述第三分割结果的特征图像，与所述第二教师模型确定的用于生成所述第二分割结果的特征图像之间的差异，确定所述学生模型的局部特征损失，包括：
    将所述第二教师模型确定的特征图像和所述学生模型确定的特征图像进行逐通道、逐像素求取特征差值，并根据多个特征差值确定局部特征损失。
  6. 根据权利要求3所述的方法，其中，所述根据所述第三分割结果分别与所述第一分割结果和所述第二分割结果的差异，确定所述学生模型的一致性损失，包括：
    将所述第三分割结果分别与所述第一分割结果和所述第二分割结果的交叉熵损失之和，作为所述学生模型的一致性损失。
  7. 根据权利要求2所述的方法，其中，在所述样本图像包括有标签的第一样本图像的情况下，所述学生模型的训练方法，还包括：
    根据所述第三分割结果与所述第一样本图像的标签的差异，确定所述学生模型的监督损失；
    所述将所述全局语义损失、所述局部特征损失和所述一致性损失作为监督信息，训练所述学生模型，包括：
    将所述全局语义损失、所述局部特征损失、所述一致性损失和所述监督损失作为监督信息，训练所述学生模型。
  8. 根据权利要求7所述的方法，其中，所述根据所述第三分割结果与所述第一样本图像的标签的差异，确定所述学生模型的监督损失，包括：
    将所述第三分割结果与所述第一样本图像的标签的交叉熵损失，作为所述学生模型的监督损失。
  9. 根据权利要求1-8中任一所述的方法，其中，所述第一教师模型和所述第二教师模型为预先训练的模型，且在所述学生模型训练过程中参数固定。
  10. 根据权利要求1-8中任一所述的方法，在所述将待分割图像输入学生模型之前，还包括：
    响应于本端设备的资源剩余量符合预设范围，将所述学生模型部署于所述本端设备中。
  11. 一种图像的语义分割装置，包括：
    输入模块，设置为将待分割图像输入学生模型；其中，所述学生模型根据第一教师模型和第二教师模型提供的监督信息训练得到，所述第一教师模型的 深度大于所述学生模型的深度和所述第二教师模型的深度，所述第二教师模型的宽度大于所述学生模型的宽度和所述第一教师模型的宽度；
    输出模块，设置为基于所述学生模型输出所述待分割图像的语义分割结果。
  12. 一种电子设备，包括：
    至少一个处理器；
    存储装置，设置为存储至少一个程序；
    当所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现如权利要求1-10中任一所述的图像的语义分割方法。
  13. 一种包含计算机可执行指令的存储介质，所述计算机可执行指令在由计算机处理器执行时用于执行如权利要求1-10中任一所述的图像的语义分割方法。
  14. 一种计算机程序产品，包括承载在非暂态计算机可读介质上的计算机程序，所述计算机程序包含用于执行如权利要求1-10中任一所述的图像的语义分割方法的程序代码。
PCT/CN2023/079602 2022-03-09 2023-03-03 图像的语义分割方法、装置、电子设备及存储介质 Ceased WO2023169334A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/844,755 US20250182436A1 (en) 2022-03-09 2023-03-03 Semantic segmentation method and apparatus for image, and electronic device and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210225180.8 2022-03-09
CN202210225180.8A CN116797782A (zh) 2022-03-09 2022-03-09 一种图像的语义分割方法、装置、电子设备及存储介质

Publications (1)

Publication Number Publication Date
WO2023169334A1 true WO2023169334A1 (zh) 2023-09-14

Family

ID=87937239

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/079602 Ceased WO2023169334A1 (zh) 2022-03-09 2023-03-03 图像的语义分割方法、装置、电子设备及存储介质

Country Status (3)

Country Link
US (1) US20250182436A1 (zh)
CN (1) CN116797782A (zh)
WO (1) WO2023169334A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118629038A (zh) * 2024-06-20 2024-09-10 深圳市梓健生物科技有限公司 一种用于妇科微生物荧光图像的识别模型训练方法
CN118823834A (zh) * 2024-09-18 2024-10-22 紫光摩度教育科技有限公司 一种课堂行为分析方法、系统、计算机及存储介质
CN118968457A (zh) * 2024-10-18 2024-11-15 浙江有鹿机器人科技有限公司 一种使用涂鸦标注的高效语义场景补全方法及系统
CN119904472A (zh) * 2025-01-17 2025-04-29 湖南大学 一种基于视觉语言模型的半监督医学图像分割方法、系统

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240005648A1 (en) * 2022-06-29 2024-01-04 Objectvideo Labs, Llc Selective knowledge distillation
US20240290332A1 (en) * 2023-02-28 2024-08-29 Qualcomm Incorporated Knowledge distillation from non-streaming to streaming encoder

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111950638A (zh) * 2020-08-14 2020-11-17 厦门美图之家科技有限公司 基于模型蒸馏的图像分类方法、装置和电子设备
KR102225579B1 (ko) * 2020-05-14 2021-03-10 아주대학교산학협력단 학습성능이 향상된 지식 증류법 기반 의미론적 영상 분할 방법
CN113449851A (zh) * 2021-07-15 2021-09-28 北京字跳网络技术有限公司 数据处理方法及设备
CN113627545A (zh) * 2021-08-16 2021-11-09 山东大学 一种基于同构多教师指导知识蒸馏的图像分类方法及系统
CN113705362A (zh) * 2021-08-03 2021-11-26 北京百度网讯科技有限公司 图像检测模型的训练方法、装置、电子设备及存储介质

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112465111B (zh) * 2020-11-17 2024-06-21 大连理工大学 一种基于知识蒸馏和对抗训练的三维体素图像分割方法
CN113538334B (zh) * 2021-06-09 2025-01-03 香港中文大学深圳研究院 一种胶囊内窥镜图像病变识别装置及训练方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102225579B1 (ko) * 2020-05-14 2021-03-10 아주대학교산학협력단 학습성능이 향상된 지식 증류법 기반 의미론적 영상 분할 방법
CN111950638A (zh) * 2020-08-14 2020-11-17 厦门美图之家科技有限公司 基于模型蒸馏的图像分类方法、装置和电子设备
CN113449851A (zh) * 2021-07-15 2021-09-28 北京字跳网络技术有限公司 数据处理方法及设备
CN113705362A (zh) * 2021-08-03 2021-11-26 北京百度网讯科技有限公司 图像检测模型的训练方法、装置、电子设备及存储介质
CN113627545A (zh) * 2021-08-16 2021-11-09 山东大学 一种基于同构多教师指导知识蒸馏的图像分类方法及系统

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118629038A (zh) * 2024-06-20 2024-09-10 深圳市梓健生物科技有限公司 一种用于妇科微生物荧光图像的识别模型训练方法
CN118823834A (zh) * 2024-09-18 2024-10-22 紫光摩度教育科技有限公司 一种课堂行为分析方法、系统、计算机及存储介质
CN118968457A (zh) * 2024-10-18 2024-11-15 浙江有鹿机器人科技有限公司 一种使用涂鸦标注的高效语义场景补全方法及系统
CN119904472A (zh) * 2025-01-17 2025-04-29 湖南大学 一种基于视觉语言模型的半监督医学图像分割方法、系统

Also Published As

Publication number Publication date
US20250182436A1 (en) 2025-06-05
CN116797782A (zh) 2023-09-22

Similar Documents

Publication Publication Date Title
WO2023169334A1 (zh) 图像的语义分割方法、装置、电子设备及存储介质
WO2024012255A1 (zh) 语义分割模型训练方法、装置、电子设备及存储介质
US20260024320A1 (en) Semantic segmentation model training method, electronic device and storage medium
CN111696176B (zh) 图像处理方法、装置、电子设备及计算机可读介质
WO2023232056A1 (zh) 图像处理方法、装置、存储介质及电子设备
CN114422698B (zh) 视频生成方法、装置、设备及存储介质
CN115761443A (zh) 货物检测方法、装置、系统、电子设备及存储介质
CN116168207A (zh) 图像剪裁方法、模型训练方法、装置、电子设备及介质
WO2023179420A1 (zh) 一种图像处理方法、装置、电子设备及存储介质
US20240281930A1 (en) Network model compression method, apparatus and device, image generation method, and medium
US20260051096A1 (en) Video editing method, device, and medium
WO2023116744A1 (zh) 图像处理方法、装置、设备及介质
US12437502B2 (en) Method for generating objective function, apparatus, electronic device and computer readable medium
CN114510911A (zh) 文本处理方法、装置、计算机设备及存储介质
US20240330769A1 (en) Object processing method, device, readable medium and electronic device
CN117097797B (zh) 云边端协同方法、装置、系统、电子设备及可读存储介质
US20240290135A1 (en) Method, electornic device, and storage medium for image processing
WO2023202361A1 (zh) 视频生成方法、装置、介质及电子设备
CN114037715B (zh) 图像分割方法、装置、设备及存储介质
CN117765250A (zh) 一种图像的语义分割方法、装置、电子设备及存储介质
WO2024007938A1 (zh) 一种多任务预测方法、装置、电子设备及存储介质
CN118332326A (zh) 多模态数据的处理方法、装置、电子设备及存储介质
CN114283060B (zh) 视频生成方法、装置、设备及存储介质
CN116758359B (zh) 图像识别方法、装置及电子设备
CN121074927A (zh) 基于多模态大模型的图像识别方法、装置、电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23765912

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18844755

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 05.12.2024)

WWP Wipo information: published in national office

Ref document number: 18844755

Country of ref document: US

122 Ep: pct application non-entry in european phase

Ref document number: 23765912

Country of ref document: EP

Kind code of ref document: A1