WO2024017199A1 - 模型训练方法、实例分割方法、装置、设备及介质 - Google Patents

模型训练方法、实例分割方法、装置、设备及介质 Download PDF

Info

Publication number
WO2024017199A1
WO2024017199A1 PCT/CN2023/107725 CN2023107725W WO2024017199A1 WO 2024017199 A1 WO2024017199 A1 WO 2024017199A1 CN 2023107725 W CN2023107725 W CN 2023107725W WO 2024017199 A1 WO2024017199 A1 WO 2024017199A1
Authority
WO
WIPO (PCT)
Prior art keywords
instance
category
sample image
point
label
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/107725
Other languages
English (en)
French (fr)
Inventor
朱敏昭
刘乐遥
孔涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Youzhuju Network Technology Co Ltd
Original Assignee
Beijing Youzhuju Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Youzhuju Network Technology Co Ltd filed Critical Beijing Youzhuju Network Technology Co Ltd
Publication of WO2024017199A1 publication Critical patent/WO2024017199A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks

Definitions

  • the present disclosure relates to the field of artificial intelligence technology, and in particular, to a model training method, instance segmentation method, device, equipment and medium.
  • the instance segmentation task is an important task in computer vision. It can predict the semantic category and instance category to which each point in the image belongs, that is, it can point out the various object categories contained in the image and further distinguish different instance objects.
  • Existing instance segmentation models that perform instance segmentation tasks need to label a large number of points in the training sample images during training, that is, very dense labeling information is required, and the required labeling cost is very high.
  • the present disclosure provides a model training method, instance segmentation method, device, equipment and medium.
  • Embodiments of the present disclosure provide a model training method, including: obtaining a sample image; obtaining an initial label of the sample image; wherein the initial label includes a semantic category and an instance category to which the target point belongs, and each of the sample images Each object is marked with a target point respectively, and the target point marked by the object is a point contained in the object; an extended label of the sample image is generated based on the initial label; wherein the extended label includes the The semantic category and instance category to which the associated point of the target point belongs, and the semantic category and instance category to which the associated point belongs are determined based on the semantic category and instance category to which the target point belongs; based on the initial label and the The extended label trains the initial neural network model, and uses the neural network model at the end of training as an instance segmentation model.
  • Embodiments of the present disclosure also provide an instance segmentation method, which includes: acquiring a target image; inputting the target image into a pre-trained instance segmentation model; wherein the instance segmentation model is obtained using the above model training method; Obtain the instance segmentation result output by the instance segmentation model; the instance segmentation result includes the semantic category and instance category to which each point in the target image belongs.
  • Embodiments of the present disclosure also provide a model training device, including: a sample acquisition module, used to acquire a sample image; an initial label acquisition module, used to acquire an initial label of the sample image; wherein the initial label includes a target point The semantic category and instance category it belongs to, each object in the sample image is marked with a target point, and the target point marked by the object is a point contained in the object; the extended label generation module is used to The initial label generates an extended label of the sample image; wherein the extended label includes the semantic category and instance category to which the associated point of the target point belongs, and the semantic category and instance category to which the associated point belongs are based on the The semantic category and instance category to which the target point belongs are determined; a model training module is used to train the initial neural network model based on the initial label and the extended label, and use the neural network model at the end of training as Instance segmentation model.
  • Embodiments of the present disclosure also provide an instance segmentation device, including: an image acquisition module for acquiring a target image; a model input module for inputting the target image into a pre-trained instance segmentation model; wherein, The instance segmentation model is obtained by using the model training method described in any one of the above; the result acquisition module is used to obtain the instance segmentation result output by the instance segmentation model; the instance segmentation result includes the location of each point in the target image semantic categories and instance categories.
  • An embodiment of the present disclosure also provides an electronic device.
  • the electronic device includes: a processor; a memory used to store instructions executable by the processor; and the processor is used to read the instruction from the memory.
  • the instructions can be executed and executed to implement the model training method or the instance segmentation method as provided by the embodiments of the present disclosure.
  • Embodiments of the present disclosure also provide a computer-readable storage medium, the storage medium stores a computer program, and the computer program is used to execute the model training method or the instance segmentation method as provided by the embodiments of the present disclosure.
  • the above technical solutions provided by the embodiments of the present disclosure can only label one target point for each object in the sample image during model training, and can then be expanded based on the initial label (the semantic category and instance category to which the target point belongs). , obtain the extended label (the semantic category and instance category to which the associated point of the target point belongs), and then perform model training based on the initial label and extended label, and then the instance segmentation model can be trained and subsequent instance segmentation can be performed.
  • the above method only needs to initially mark one point for each object, which greatly saves the labeling cost. And because the instance segmentation model used in the instance segmentation method requires lower annotation costs, that is, it is more convenient to obtain the instance segmentation model, which also helps to improve the universality of the instance segmentation method.
  • Figure 1 is a schematic flow chart of a model training method provided by an embodiment of the present disclosure
  • Figure 2 is a schematic diagram of model training provided by an embodiment of the present disclosure
  • Figure 3 is a schematic flowchart of an instance segmentation method provided by an embodiment of the present disclosure
  • Figure 4 is a schematic structural diagram of a model training device provided by an embodiment of the present disclosure.
  • Figure 5 is a schematic structural diagram of an example segmentation device provided by an embodiment of the present disclosure.
  • FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.
  • Figure 1 is a schematic flowchart of a model training method provided by an embodiment of the present disclosure.
  • the method can be executed by a model training device, where the device can be implemented using software and/or hardware, and can generally be integrated in electronic equipment.
  • the method mainly includes the following steps S102 to S108:
  • Step S102 Obtain a sample image.
  • the sample image may be a two-dimensional image (2D plane image) or a three-dimensional image (such as a three-dimensional point cloud).
  • the embodiment of the present disclosure does not limit the form of the sample image. In practical applications, the number of sample images is usually multiple, and each sample image contains at least one object.
  • the object can be a person, animal, plant, static object, or dynamic object such as a vehicle. It should be specified It should be noted that objects such as the sky and the ground can also be regarded as objects, and the embodiments of the present disclosure do not limit the type/number of objects included in the sample image.
  • Step S104 obtain the initial label of the sample image; where the initial label includes the semantic category and instance category to which the target point belongs, each object in the sample image is labeled with a target point, and the target point labeled by the object is the target point contained in the object. one point.
  • the initial label is the label obtained by initially labeling the sample image.
  • each object in the sample image only needs to be labeled with one point.
  • the point marked on the object is the target point, and the target point can be It can be any point in the object, it can also be the center point of the object or other key positions of the object, and there is no restriction here.
  • the semantic category and instance category to which a point on each object belongs can be pre-labeled, so that the initial label can be obtained conveniently and quickly.
  • the above semantic categories can be understood as object categories.
  • the semantic category to which the target point belongs can indicate the type of object to which the target point belongs, such as people, cats, dogs, vehicles, etc.; the instance category to which the target point belongs can be understood as based on the semantic category. It further points out which specific object the target point belongs to. For example, when the sample image contains multiple people, the semantic category only indicates that the target point belongs to the person, while the instance category can further distinguish between multiple people and indicate which person the target point specifically belongs to. .
  • Step S106 generate an extended label of the sample image based on the initial label; wherein the extended label includes the semantic category and instance category to which the associated point of the target point belongs, and the semantic category and instance category to which the associated point belongs is based on the semantic category and the instance category to which the target point belongs. Determined by the instance category.
  • the associated points of the target point can be found.
  • the associated points of the target point are points that have a certain correlation with the target point, such as points that have the same characteristics (such as color, etc.) as the target point and/or are located in the target point. Points within a specified distance range.
  • the semantic category to which the associated point belongs and The instance category is consistent with the semantic category and instance category to which the target point belongs.
  • Step S108 Train an initial neural network model based on the initial labels and extended labels, and use the neural network model at the end of training as an instance segmentation model.
  • the process of training a neural network model is essentially a process of adjusting the network parameters of the neural network model until the preset conditions are reached (such as the loss function converges and the neural network model can output the expected instance segmentation results). Train and obtain the instance segmentation model.
  • the embodiment of the present disclosure does not limit the model structure, such as it can be Convolutional neural network or other network structures, etc., any network structure that can be used as instance segmentation can be used.
  • model training can be performed by appropriately expanding on this basis.
  • This training method can also be called weakly supervised training.
  • pseudo labels can also be generated based on the output results of the neural network model for self-training to further improve the model training accuracy.
  • the above technical solutions provided by the embodiments of the present disclosure can only label one target point for each object in the sample image during model training, and can then be expanded based on the initial label (the semantic category and instance category to which the target point belongs). , obtain the extended label (the semantic category and instance category to which the associated point of the target point belongs), and then perform model training based on the initial label and extended label, and then the instance segmentation model can be trained and subsequent instance segmentation can be performed.
  • the above method only needs to initially mark one point for each object, which greatly saves the labeling cost.
  • the step of generating an extended label of the sample image based on the initial label can be implemented with reference to the following steps a to c:
  • Step a obtain low-level features of the sample image.
  • low-level features may include color features
  • low-level features may include color features
  • normal vector features etc.
  • low-level features may also include features such as Texture features, shape features, etc. are not limited here.
  • Step b Segment the sample image according to low-level features, and obtain the associated points of the target points based on the segmentation results. For example, if the sample image is segmented into multiple local units based on low-level features, then the points located in the same local unit as the target point will be regarded as the associated points of the target point. In order to quickly and accurately find the associated points of the target point, the sample image is segmented based on low-level features, and the steps of obtaining the associated points of the target point based on the segmentation results include at least one of the following (1) and (2):
  • sample image is a two-dimensional image
  • a two-dimensional image can be subdivided into multiple local units based on low-level features such as color, brightness, texture, etc.
  • Each local unit can be regarded as a super pixel, and the super pixel is composed of a series of Small areas composed of pixels with adjacent positions and similar low-level features. Most of these small areas retain effective information for further image segmentation, and generally do not destroy the boundary information of objects in the image.
  • a superpixel segmentation algorithm can be used to segment the sample image, so as to obtain the associated points of the target point based on the segmentation results.
  • points in the sample image that belong to the same superpixel as the target point can be used as associated points of the target point.
  • the superpixel segmentation result includes multiple superpixels, and each superpixel may It will contain multiple points with similar characteristics and adjacent positions.
  • the remaining points belonging to the same superpixel as the target point can be regarded as the associated points of the target point, or the target point can be Some points among the remaining points belonging to the same superpixel (such as points adjacent to the target point in the superpixel) serve as associated points of the target point, and are not limited here.
  • the sample image When the sample image is a three-dimensional image, it can be represented by point cloud data, that is, the point cloud can be segmented by super-voxel based on low-level features such as color and normal vector. Multiple local units of the point cloud can also be obtained. Each local unit can be regarded as a super voxel.
  • a supervoxel is a small region composed of a series of points in a point cloud that are adjacent in location and have similar low-level features.
  • a supervoxel segmentation algorithm can be used to segment the sample image, so as to obtain the associated points of the target points based on the segmentation results.
  • points in the sample image that belong to the same supervoxel as the target point can be used as associated points of the target point.
  • the superpixel segmentation result includes multiple supervoxels, and each supervoxel A voxel may contain multiple points with similar characteristics and adjacent positions.
  • the remaining points belonging to the same supervoxel as the target point can be regarded as the associated points of the target point, or Some points among the remaining points belonging to the same supervoxel as the target point (such as points adjacent to the target point in the supervoxel) are regarded as associated points of the target point, and there is no restriction here.
  • step c the semantic category and instance category to which the target point belongs are used as the semantic category and instance category to which the associated point belongs, respectively, to obtain the extended label of the sample image.
  • the low-level features of the associated point and the target point are similar and the associated point is usually adjacent to the target point. Therefore, the probability that the associated point and the target point belong to the same object is very high, so the target point can be
  • the semantic category and instance category of the genus are respectively used as the semantic category and instance category to which the associated point belongs.
  • the step of training a preset neural network model based on initial labels and extended labels, and using the neural network model at the end of training as an instance segmentation model can be implemented by referring to the following steps 1 to 3:
  • Step 1 Perform first training on the initial neural network model based on the initial labels and extended labels, and use the neural network model that reaches the first training end condition as the first neural network model.
  • the first training end condition may include one or more of the following: the total loss function value converges to the first preset threshold, reaches the first training times threshold, and reaches the first training duration threshold. Among them, the total loss function value is determined based on the label and model output results.
  • the sample image can be input into the neural network model, the segmentation result output by the neural network model for the sample image is obtained, and the segmentation result is evaluated based on the label (initial label and extended label) and the preset loss function, and we get Total loss function value.
  • the segmentation results output by the neural network model during training can be represented only by information such as instance feature values, semantic probabilities, and offset vectors of each point in the sample image, where the semantic probabilities can be directly used Semantic segmentation, instance feature values and offset vectors can be used for instance segmentation.
  • the segmentation result is evaluated based on the label (initial label and extended label), such as determining the first loss value corresponding to the instance feature value based on the label and the discriminative loss (Discriminative Loss) function, based on the label and Cross Entropy Loss (Cross Entropy Loss)
  • the function determines the second loss value corresponding to the semantic probability, determines the third loss value corresponding to the offset vector based on the L2 loss function, and weights and sums the first loss value, the second loss value, and the third loss value to obtain the total loss value (also That is, the abbreviation of the aforementioned total loss function value), the parameters of the neural network model are adjusted based on the total loss value until the first training end condition is reached.
  • the initially trained first neural network model can be obtained, and the first neural network model has certain instance segmentation capabilities.
  • Step 2 Obtain the output result of the first neural network model for each point to be measured in the sample image, and generate a pseudo label based on the output result; the pseudo label includes the location of each point in the sample image.
  • Semantic categories and instance categories of genus In practical applications, each point in the sample image can be used as a point to be measured, or some points in the sample image can be used as points to be measured. Some points in the sample image can be multiple randomly determined points, or It can be multiple designated points; for example, a preset number of points in the sample image can be randomly determined as the points to be measured, or multiple points distributed at different positions in the sample image can be specified as the points to be measured according to requirements. If the points to be tested are only some points in the sample image, the quantity threshold can be further set so that the number of points to be tested needs to be higher than the preset quantity threshold to ensure the reliability of model training.
  • the output results include instance feature values, semantic probabilities and offset vectors; on this basis, when generating pseudo labels based on the output results, it can be based on the instance feature values, semantic probabilities and offset vectors of each point to be measured in the sample image, As well as the semantic category and instance category to which the target point belongs, a preset clustering algorithm is used to generate pseudo labels.
  • pseudo labels are not labels that are actually labeled by humans, but are the prediction results of the model for unlabeled data during the training process. Since they are not necessarily accurate, they can be called pseudo labels (or pseudo labels). However, pseudo-labels can help the model learn the information hidden in unlabeled data.
  • clustering can be performed according to the clustering algorithm based on the instance feature value, semantic probability and offset vector of each test point in the sample image output by the neural network model. According to the clustering results and the known The semantic category and instance category to which the target point belongs can be easily obtained, and the generated pseudo-labels are relatively accurate.
  • the embodiments of the present disclosure may not limit the clustering algorithm, including but not limited to K-means clustering algorithm (K-means clustering), etc.
  • the embodiments of the present disclosure further provide specific implementation methods for generating pseudo labels using a preset clustering algorithm. For example, this can be implemented with reference to the following steps A to B:
  • step A the target point is used as the initial clustering center in the preset clustering algorithm, and clustering is performed based on the initial clustering center and the instance feature value, semantic probability and offset vector of each point to be measured in the sample image.
  • the clustering center can also be called the seed, and the initial clustering center can also be called the initial seed.
  • the target point can be used as the initial clustering center, and then multiple iterations of clustering are performed until the clustering center no longer changes. Specifically, you can refer to the following implementation:
  • the first similarity (also called is the first distance) determines the new cluster center based on the first similarity. Specifically, categories can be divided according to the first similarity between each point to be tested and each initial clustering center, and each point to be tested can be classified into the class to which the clustering center has the closest similarity, and then Recalculate the position of each cluster center based on the initially obtained clusters (such as averaging all points within each cluster, etc.).
  • the third distance between each point to be measured and the cluster center obtained from the previous clustering process is obtained.
  • the second degree of similarity (can also be called the second distance) determines the new clustering center based on the second degree of similarity (please refer to the above-mentioned specific implementation method of determining the new clustering center), and ends the clustering until the clustering center no longer changes.
  • class processing Among them, the number of cluster centers is equal to the number of target points, that is, equal to the number of objects in the sample image.
  • the above similarity can also be called distance in clustering algorithms.
  • the clustering center is repeatedly adjusted, and finally the clustering is stopped when the clustering center no longer changes, and a more accurate clustering result can be obtained.
  • this method helps to improve the clustering efficiency, and the final result can be found in a smaller number of iterations.
  • the clustering center can effectively save computing overhead, and better avoid the problem of initially selecting an inappropriate initial clustering center that causes more computing overhead; in addition, the above method can also ensure the accuracy of the clustering results to a certain extent. Reliability, each cluster finally obtained can correspond to an object relatively accurately.
  • Step B According to the clustering results and the semantic category and instance category to which the target point belongs, the semantic category and instance category to which each test point in the sample image belongs is obtained.
  • the same cluster points of the target point can be obtained according to the clustering results; it can be understood that the clustering results include multiple clusters, and the same cluster points are the same cluster points in the sample image as The target points are test points that belong to the same cluster.
  • the clusters in the clustering results correspond to the objects in the sample image one-to-one (that is, the objects corresponding to different clusters are different); in other words, through the above clustering results, the objects corresponding to each cluster can be found.
  • the target points belong to the same object as the remaining test points, and then the semantic category and instance category to which the target point belongs can be directly used as the semantic category and instance category of the same cluster points respectively.
  • each point to be measured in the sample image has an object to which it belongs, and each object has been marked with a target point, so each point to be measured will correspond to a target point that belongs to the same object.
  • the cluster of the object is obtained through the above clustering method, and the semantic category and instance category of the target point in the cluster corresponding to the object are known, and the semantic categories and instance categories of the remaining points to be measured are Theoretically, it is consistent with the semantic category and instance category of the target point.
  • the semantic category and instance category of the remaining points to be measured in the object can be directly determined based on the semantic category and instance category of the target point in the object.
  • the semantic categories and instance categories of all test points in the sample image can be known.
  • Step 3 Perform a second training on the first neural network model based on the pseudo labels, and use the first neural network model when the second training end condition is reached as an instance segmentation model.
  • the process of conducting the second training of the first neural network model based on pseudo labels is similar to the process of conducting the first training of the initial neural network model based on the initial labels and extended labels. The difference is that the first training is the initial training, and the initial training is used. labels and labels based on expansion of the initial labels.
  • the second training is model self-training, using pseudo labels obtained from the output results of the initially trained first neural network model, that is, the labels used during training are different.
  • the above second training end condition may include one or more of the following: total loss function The value converges to the second preset threshold, reaches the second training times threshold, and reaches the second training duration threshold.
  • the above calculation method of the total loss function value also refers to the method of calculating the total loss function value in the first training process, such as requiring the first loss value corresponding to the instance feature value, the second loss value corresponding to the semantic probability and the third loss value corresponding to the offset vector.
  • the three loss values are weighted and summed to obtain the total loss function value, which will not be described again here.
  • the first training and the second training both represent the training stage, and do not represent the number of trainings. Whether it is the first training or the second training, they will be trained multiple times (that is, adjusted multiple times through backpropagation). Model parameters), pseudo-labels will be output for each training, and the output pseudo-labels are used to supervise the output results of the model in the next training, thereby adjusting the model parameters until the preset training end conditions are reached to stop training.
  • the neural network model is initially trained using initial labels and extended labels.
  • pseudo labels are generated based on the output results of the neural network model.
  • the pseudo labels are used to The neural network model performs self-training.
  • the above method only needs to label one point for each object initially. Subsequent labels can be automatically generated based on the initially labeled points, which greatly reduces the labeling cost.
  • whether the above method obtains extended labels based on low-level features The method of obtaining pseudo labels based on clustering is more reliable, which can effectively ensure the accuracy of the labels used to train the model, and also ensure the accuracy of the trained model to a certain extent.
  • the sample image is input to the neural network model.
  • the neural network model can output instance feature values for each point in the sample image.
  • semantic probability and offset vector pseudo labels can be generated based on the output results.
  • the pseudo labels include the semantic category and instance category to which each point in the sample image belongs. Among them, the semantic category is obtained based on the semantic probability, and the instance category is based on the instance feature value and The offset vector is obtained; then pseudo labels can be used to supervise the neural network model to adjust the parameters of the neural network model.
  • Figure 2 shows that each object only needs to be marked with one target point, and also shows the way to generate pseudo labels based on the output results of the model.
  • the semantic probability and offset vector are clustered using a clustering algorithm, where the target
  • the points are the initial clustering centers, which are O1, O2 and O3 respectively (indicated by solid circles of different grayscales). Taking a random point as an example, the similarity between this point and O1 is 0.8, and the similarity between this point and O2 is 0.8. The degree is 0.6, and the similarity with O3 is 0.2. By calculating the similarity between each point and O1, O2 and O3, it can be re-clustered.
  • the gray levels corresponding to different clusters are different; then find new The cluster centers O1', O2', and O3' of the cluster (indicated by solid circles of different grayscales), and then calculate the similarity between each point and the new cluster centers O1', O2', and O3' again. , and then re-clustering based on similarity. After multiple clusterings, clusters whose cluster centers do not change are finally obtained.
  • Each cluster corresponds to an object, thus based on the semantic category of the target point in the cluster to which each point belongs. and instance categories, to obtain the semantic category and instance category of each point, and generate pseudo labels more conveniently and reliably.
  • Figure 2 is only a simple example, and not all clusters/points are shown.
  • embodiments of the present disclosure provide an instance segmentation method.
  • FIG 3 the schematic flow chart of an instance segmentation method shown in Figure 3, which mainly includes the following steps S302 to S306:
  • Step S302 Obtain the target image.
  • the target image is the image to be instance segmented.
  • Step S304 Input the target image into the pre-trained instance segmentation model; wherein the instance segmentation model is obtained by using any of the above model training methods.
  • the specific training method can refer to the above related content and is not limited here.
  • Step S306 Obtain the instance segmentation result output by the instance segmentation model; the instance segmentation result includes the semantic category and instance category to which each point in the target image belongs.
  • the instance segmentation model can output the instance feature value, semantic probability and offset vector corresponding to each point in the target image. Based on the semantic probability, the semantic category to which each point belongs can be obtained. Based on the instance feature value and semantic probability, that is The similarity between different points can be calculated to obtain the instance category to which each point belongs through methods such as Mean-shift clustering.
  • the instance segmentation model used in the instance segmentation method requires lower annotation costs, it is more convenient to obtain the instance segmentation model, which also helps to improve the universality of the instance segmentation method. Furthermore, based on the aforementioned method, whether it is to obtain extended labels based on low-level features or obtain pseudo labels based on clustering methods, they are more reliable. Compared with conventional weak supervision, In terms of training methods, the above-mentioned methods provided by the embodiments of the present disclosure can effectively ensure the accuracy of the labels used to train the model. Therefore, the accuracy of the instance segmentation model obtained by training is relatively high, that is, the embodiments of the present disclosure only require very few training methods. Labeling information can achieve higher segmentation accuracy.
  • FIG. 4 is a schematic structural diagram of a model training device provided by an embodiment of the present disclosure.
  • the device can be implemented by software and/or hardware. Generally, it can be integrated into electronic equipment, as shown in Figure 4, including:
  • Sample acquisition module 402 used to acquire sample images
  • the initial label acquisition module 404 is used to obtain the initial label of the sample image; wherein the initial label includes the semantic category and instance category to which the target point belongs.
  • Each object in the sample image is labeled with a target point, and the target point labeled by the object is A point contained in an object;
  • the extended label generation module 406 is used to generate an extended label of the sample image based on the initial label; wherein the extended label includes the semantic category and instance category to which the associated point of the target point belongs, and the semantic category and instance category to which the associated point belongs is based on the target point. Determined by the semantic category and instance category it belongs to;
  • the model training module 408 is used to train the initial neural network model based on the initial label and the extended label, and use the neural network model at the end of the training as an instance segmentation model.
  • the above technical solutions provided by the embodiments of the present disclosure can only label one target point for each object in the sample image during model training, and can then be expanded based on the initial label (the semantic category and instance category to which the target point belongs). , obtain the extended label (the semantic category and instance category to which the associated point of the target point belongs), and then perform model training based on the initial label and extended label, and then the instance segmentation model can be trained and subsequent instance segmentation can be performed.
  • the above method only needs to initially mark one point for each object, which greatly saves the labeling cost.
  • the extended label generation module 406 is specifically configured to: obtain low-level features of the sample image; segment the sample image according to the low-level features, and obtain the associated points of the target points based on the segmentation results;
  • the word to which the target point belongs is
  • the semantic category and instance category are respectively used as the semantic category and instance category to which the associated point belongs, and the extended label of the sample image is obtained.
  • the extended label generation module 406 is specifically used for at least one of the following: when the sample image is a two-dimensional image, perform super-pixel segmentation processing on the sample image according to the low-level features, based on The segmentation result is to obtain the correlation point of the target point; when the sample image is a three-dimensional image, super-voxel segmentation processing is performed on the sample image according to the low-level features, and the correlation point of the target point is obtained based on the segmentation result. point.
  • the model training module 408 is specifically configured to: perform a first training on the initial neural network model based on the initial label and the extended label, and use the neural network model when the first training end condition is reached as the first Neural network model; obtain the output result of the first neural network model for each test point in the sample image, and generate a pseudo label based on the output result; the pseudo label includes each of the points in the sample image The semantic category and instance category to which the point to be measured belongs; perform a second training on the first neural network model based on the pseudo label, and use the first neural network model when the second training end condition is reached as an instance segmentation model.
  • the output results include instance feature values, semantic probabilities and offset vectors; the model training module 408 is specifically configured to: based on the instance feature values, semantic probabilities of each of the points to be measured in the sample image and offset vector, as well as the semantic category and instance category to which the target point belongs, and a preset clustering algorithm is used to generate pseudo labels.
  • the model training module 408 is specifically configured to: use the target point as an initial clustering center in a preset clustering algorithm, based on the initial clustering center and each of the sample images to be The instance feature values, semantic probabilities and offset vectors of the measured points are clustered; according to the clustering results and the semantic categories and instance categories to which the target points belong, the sample image to which each of the measured points belongs is obtained. Semantic categories and instance categories.
  • the model training module 408 is specifically configured to: in the first clustering process, obtain each of the measured points based on the instance feature value, semantic probability and offset vector of each of the test points in the sample image. The first phase between the point to be measured and the initial cluster center Similarity, determine a new clustering center based on the first similarity; during non-first clustering processing, based on the instance feature value, semantic probability and offset vector of each point to be measured in the sample image, Obtain the second degree of similarity between each of the points to be measured and the cluster center obtained from the last clustering process, and determine a new cluster center based on the second degree of similarity until the cluster center no longer changes. Clustering processing.
  • the model training module 408 is specifically configured to: obtain the same cluster points of the target point according to the clustering results; wherein the same cluster points are the same cluster as the target point in the sample image.
  • the points to be measured, the clusters in the clustering results correspond to the objects in the sample image one-to-one; the semantic categories and instance categories to which the target points belong are respectively used as the semantic categories of the same cluster points and instance classes.
  • the model training device provided by the embodiments of the present disclosure can execute the model training method provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution method.
  • FIG. 5 is a schematic structural diagram of an instance segmentation device provided by an embodiment of the present disclosure.
  • the device can be implemented by software and/or hardware, and can generally be integrated in electronic equipment. As shown in Figure 5, it includes:
  • Image acquisition module 502 used to acquire target images
  • the model input module 504 is used to input the target image into a pre-trained instance segmentation model; wherein the instance segmentation model is obtained by using any one of the model training methods;
  • the result acquisition module 506 is used to obtain the instance segmentation result output by the instance segmentation model; the instance segmentation result includes the semantic category and instance category to which each point in the target image belongs.
  • instance segmentation model used by the instance segmentation device requires lower annotation costs, that is, it is more convenient to obtain the instance segmentation model, which also helps to improve the universality of the instance segmentation method.
  • the instance segmentation device provided by the embodiments of the present disclosure can execute the instance segmentation method provided by any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing the method.
  • Embodiments of the present disclosure provide an electronic device.
  • the electronic device includes: a processor; a memory for storing executable instructions by the processor; and a processor for reading executable instructions from the memory and executing the instructions to implement any of the above. Model training method or instance segmentation method.
  • FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. As shown in FIG. 6 , electronic device 600 includes one or more processors 601 and memory 602 .
  • the processor 601 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 600 to perform desired functions.
  • CPU central processing unit
  • the processor 601 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 600 to perform desired functions.
  • Memory 602 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory.
  • the volatile memory may include, for example, random access memory (RAM) and/or cache memory (cache).
  • the non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.
  • One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 601 may execute the program instructions to implement the model training method or instance segmentation method of the embodiments of the present disclosure described above and /or other desired functionality.
  • Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.
  • the electronic device 600 may also include an input device 603 and an output device 604, and these components are interconnected through a bus system and/or other forms of connection mechanisms (not shown).
  • the input device 603 may also include, for example, a keyboard, a mouse, and the like.
  • the output device 604 can output various information to the outside, including determined distance information, direction information, etc.
  • the output device 604 may include, for example, a display, a speaker, a printer, a communication network and remote output devices connected thereto, and the like.
  • the electronic device 600 may also include any other appropriate components depending on the specific application.
  • embodiments of the present disclosure may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to execute the model provided by the embodiments of the present disclosure. Training method or instance segmentation method.
  • the computer program product may be written with program code for performing operations of embodiments of the present disclosure in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc. , also includes conventional procedural programming languages, such as the "C" language or similar programming languages.
  • the program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.
  • embodiments of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored.
  • the computer program instructions When the computer program instructions are run by a processor, the computer program instructions cause the processor to perform the model training provided by the embodiments of the present disclosure. method or instance splitting method.
  • the computer-readable storage medium may be any combination of one or more readable media.
  • the readable medium may be a readable signal medium or a readable storage medium.
  • the readable storage medium may include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.
  • Embodiments of the present disclosure also provide a computer program product, which includes a computer program/instruction.
  • a computer program product which includes a computer program/instruction.
  • the model training method or instance segmentation method in the embodiment of the present disclosure is implemented.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

本公开实施例涉及一种模型训练方法、实例分割方法、装置、设备及介质，其中该模型训练方法包括：获取样本图像；获取样本图像的初始标签；其中，初始标签包括目标点所属的语义类别和实例类别，样本图像中每个物体分别标注有一个目标点，且物体标注的目标点为物体中所包含的一个点；基于初始标签生成样本图像的扩展标签；其中，扩展标签包括目标点的关联点所属的语义类别和实例类别，且关联点所属的语义类别和实例类别是基于目标点所属的语义类别和实例类别确定的；基于初始标签和扩展标签对初始的神经网络模型进行训练，并将训练结束时的神经网络模型作为实例分割模型。本公开实施例可有效节约训练模型所需的标注成本。

Description

模型训练方法、实例分割方法、装置、设备及介质
本申请要求2022年7月18日递交的，标题为“模型训练方法、实例分割方法、装置、设备及介质”、申请号为202210843130.6的中国发明专利申请的优先权。
技术领域
本公开涉及人工智能技术领域，尤其涉及一种模型训练方法、实例分割方法、装置、设备及介质。
背景技术
实例分割任务是计算机视觉中的重要任务，可以预测出图像中每个点所属的语义类别和实例类别，也即可以指出图像中所包含的各种物体类别并进一步区别开不同的实例对象。现有执行实例分割任务的实例分割模型在训练时都需要针对训练样本图像中的大量点进行标注，也即需要非常密集的标注信息，所需标注成本非常高。
发明内容
为了解决上述技术问题或者至少部分地解决上述技术问题，本公开提供了一种模型训练方法、实例分割方法、装置、设备及介质。
本公开实施例提供了一种模型训练方法，包括：获取样本图像；获取所述样本图像的初始标签；其中，所述初始标签包括目标点所属的语义类别和实例类别，所述样本图像中每个物体分别标注有一个目标点，且所述物体标注的目标点为所述物体中所包含的一个点；基于所述初始标签生成所述样本图像的扩展标签；其中，所述扩展标签包括所述目标点的关联点所属的语义类别和实例类别，且所述关联点所属的语义类别和实例类别是基于所述目标点所属的语义类别和实例类别确定的；基于所述初始标签和所述扩展标签对初始的神经网络模型进行训练，并将训练结束时的所述神经网络模型作为实例分割模型。
本公开实施例还提供了一种实例分割方法，包括：获取目标图像；将所述目标图像输入至预先训练得到的实例分割模型；其中，所述实例分割模型是采用上述模型训练方法得到的；获取所述实例分割模型输出的实例分割结果；所述实例分割结果包括所述目标图像中每个点所属的语义类别和实例类别。
本公开实施例还提供了一种模型训练装置，包括：样本获取模块，用于获取样本图像；初始标签获取模块，用于获取所述样本图像的初始标签；其中，所述初始标签包括目标点所属的语义类别和实例类别，所述样本图像中每个物体分别标注有一个目标点，且所述物体标注的目标点为所述物体中所包含的一个点；扩展标签生成模块，用于基于所述初始标签生成所述样本图像的扩展标签；其中，所述扩展标签包括所述目标点的关联点所属的语义类别和实例类别，且所述关联点所属的语义类别和实例类别是基于所述目标点所属的语义类别和实例类别确定的；模型训练模块，用于基于所述初始标签和所述扩展标签对初始的神经网络模型进行训练，并将训练结束时的所述神经网络模型作为实例分割模型。
本公开实施例还提供了一种实例分割装置，包括：图像获取模块，用于获取目标图像；模型输入模块，用于将所述目标图像输入至预先训练得到的实例分割模型；其中，所述实例分割模型是采用任一项所述的模型训练方法得到的；结果获取模块，用于获取所述实例分割模型输出的实例分割结果；所述实例分割结果包括所述目标图像中每个点所属的语义类别和实例类别。
本公开实施例还提供了一种电子设备，所述电子设备包括：处理器；用于存储所述处理器可执行指令的存储器；所述处理器，用于从所述存储器中读取所述可执行指令，并执行所述指令以实现如本公开实施例提供的模型训练方法或者实例分割方法。
本公开实施例还提供了一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序用于执行如本公开实施例提供的模型训练方法或者实例分割方法。
本公开实施例提供的上述技术方案，在模型训练时可以只针对样本图像中每个物体分别标注一个目标点即可，之后便可基于初始标签(目标点所属的语义类别和实例类别)进行扩展，得到扩展标签(目标点的关联点所属的语义类别和实例类别)，从而基于初始标签和扩展标签进行模型训练，便可训练得到实例分割模型及进行后续实例分割。上述方式只需最初针对每个物体标注一个点即可，极大节约了标注成本。且由于实例分割方法所采用的实例分割模型所需的标注成本较低，也即获取实例分割模型更为方便，因此也有助于提升实例分割方法的普适性。
应当理解，本部分所描述的内容并非旨在标识本公开的实施例的关键或重要特征，也不用于限制本公开的范围。本公开的其它特征将通过以下的说明书而变得容易理解。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。
为了更清楚地说明本公开实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本公开实施例提供的一种模型训练方法的流程示意图；
图2为本公开实施例提供的一种模型训练示意图；
图3为本公开实施例提供的一种实例分割方法的流程示意图；
图4为本公开实施例提供的一种模型训练装置的结构示意图；
图5为本公开实施例提供的一种实例分割装置的结构示意图；
图6为本公开实施例提供的一种电子设备的结构示意图。
具体实施方式
为了能够更清楚地理解本公开的上述目的、特征和优点，下面将对本公开的方案进行进一步描述。需要说明的是，在不冲突的情况下，本公开的实施例及实施例中的特征可以相互组合。
图1为本公开实施例提供的一种模型训练方法的流程示意图，该方法可以由模型训练装置执行，其中该装置可以采用软件和/或硬件实现，一般可集成在电子设备中。如图1所示，该方法主要包括如下步骤S102～步骤S108：
步骤S102，获取样本图像。
样本图像可以为二维图像(2D平面图像)，也可以为三维图像(诸如三维点云)，本公开实施例对样本图像的形式不进行限定。在实际应用中，样本图像的数量通常为多个，且每个样本图像中都会包含至少一种物体，该物体可以为人物、动物、植物、静止的物体或者诸如车辆等动态的物体，应当说明的是，诸如天空、地面等也都可被视为物体，本公开实施例对样本图像中所包含的物体种类/数量不进行限定。
步骤S104，获取样本图像的初始标签；其中，初始标签包括目标点所属的语义类别和实例类别，样本图像中每个物体分别标注有一个目标点，且物体标注的目标点为物体中所包含的一个点。
初始标签即为最初对样本图像进行标注所得的标签，在本公开实施例中针对样本图像中的每个物体都只需标注一个点即可，物体上标注的点即为目标点，目标点可以为物体中的任意一点，也可以为物体中心点或者物体其它关键位置的点，在此不进行限制。在实际应用中，可以预先标注每个物体上的一个点所属的语义类别和实例类别，从而方便快捷地得到初始标签。上述语义类别可以理解为物体种类，诸如目标点所属的语义类别可以指出目标点所属物体的种类为人物、猫、狗、车辆等；目标点所属的实例类别可以理解为在语义类别的基础上 进一步指出目标点属于具体的哪个物体，诸如在样本图像中包含有多个人时，语义类别仅指出目标点属于人物，而实例类别还可以进一步对多个人物进行区分，指出目标点具体属于哪个人。
在本公开实施例中，只需为每个物体标注一个点的信息即可，假设样本图像中有N个物体，则每个物体标注一个点，最终只需标注N个点(目标点)，与相关技术中需要人工在样本图像上标注非常密集的点相比，上述只需针对每个物体标注一个点的方式非常方便快捷，易于实现。
步骤S106，基于初始标签生成样本图像的扩展标签；其中，扩展标签包括目标点的关联点所属的语义类别和实例类别，且关联点所属的语义类别和实例类别是基于目标点所属的语义类别和实例类别确定的。
由于初始只针对每个物体标注一个点，所得的初始标签数量较少，为了能够正常训练模型，因此可以基于初始标签进行扩展，自动生成扩展标签。具体而言，可以找到目标点的关联点，目标点的关联点即为与目标点具有一定关联性的点，诸如，与目标点具有相同特征(诸如颜色等特征)的点和/或位于目标点指定距离范围内的点，已知目标点所属的语义类别和实例类别的基础上，可以进一步扩展获知目标点的关联点所属的语义类别和实例类别，诸如，令关联点所属的语义类别和实例类别与目标点所属的语义类别和实例类别保持一致。通过上述方式可以快速有效地自动扩充标签数量，并在一定程度上保证扩展标签的准确性。
步骤S108，基于初始标签和扩展标签对初始的神经网络模型进行训练，并将训练结束时的神经网络模型作为实例分割模型。
可以理解的是，对神经网络模型进行训练的过程实质为调整神经网络模型的网络参数的过程，直至达到预设条件时(诸如损失函数收敛，神经网络模型可以输出符合预期的实例分割结果)结束训练，得到实例分割模型。本公开实施例对模型结构不进行限制，诸如可以是 卷积神经网络或者其它网络结构等，任何可用于作为实例分割的网络结构均可。在本公开实施例中，最初只需极少量的标注信息，然后在此基础上适当扩展便可进行模型训练，这种训练方式也可称为弱监督训练。在具体训练过程中，还可以基于神经网络模型的输出结果生成伪标签进行自训练，以进一步提升模型训练精度。
本公开实施例提供的上述技术方案，在模型训练时可以只针对样本图像中每个物体分别标注一个目标点即可，之后便可基于初始标签(目标点所属的语义类别和实例类别)进行扩展，得到扩展标签(目标点的关联点所属的语义类别和实例类别)，从而基于初始标签和扩展标签进行模型训练，便可训练得到实例分割模型及进行后续实例分割。上述方式只需最初针对每个物体标注一个点即可，极大节约了标注成本。
在一些实施方式中，基于初始标签生成样本图像的扩展标签的步骤，可以参照如下步骤a～步骤c实现：
步骤a，获取样本图像的低级特征。示例性地，样本图像是二维平面图像时，低级特征诸如可以包括颜色特征；样本图像是三维点云时，低级特征诸如可以包括颜色特征、法向量特征等；此外，低级特征还可以包括诸如纹理特征、形状特征等，在此不进行限制。
步骤b，根据低级特征对样本图像进行分割处理，基于分割结果得到目标点的关联点。诸如，根据低级特征将样本图像分割为多个局部单元，则将与目标点位于同一局部单元的点作为目标点的关联点。为了能够较为快速准确地找到目标点的关联点，根据低级特征对样本图像进行分割处理，基于分割结果得到目标点的关联点的步骤，包括下述(1)和(2)中至少一项：
(1)在样本图像为二维图像的情况下，根据低级特征对样本图像进行超像素分割处理，基于分割结果得到目标点的关联点。
诸如，可以根据颜色、亮度、纹理等低级特征将二维图像细分为多个局部单元，每个局部单元均可视为一个超像素，超像素由一系列 位置相邻且低级特征相似的像素点组成的小区域，这些小区域大多保留了后续进一步进行图像分割的有效信息，且一般不会破坏图像中物体的边界信息。具体可以采用超像素分割算法对样本图像进行分割处理，从而基于分割结果得到目标点的关联点。在一种具体的实施示例中，可以将样本图像中与目标点同属于一个超像素的点作为目标点的关联点，具体的，超像素分割结果包括多个超像素，每个超像素都可能会包含多个特征相似且位置相邻的点，对于每个目标点而言，可以将与该目标点同属于一个超像素的其余点均作为该目标点的关联点，也可以将与目标点同属于一个超像素的其余点中的部分点(诸如在超像素中与目标点相邻的点)作为目标点的关联点，在此不进行限制。
(2)在样本图像为三维图像的情况下，根据低级特征对样本图像进行超体素分割处理，基于分割结果得到目标点的关联点。
样本图像为三维图像时，可以采用点云数据表示，也即可以根据诸如颜色、法向量等低级特征对点云进行超体素分割，同样可得到点云的多个局部单元，每个局部单元均可视为一个超体素。超体素由点云中一系列位置相邻且低级特征相似的点组成的小区域。具体可以采用超体素分割算法对样本图像进行分割处理，从而基于分割结果得到目标点的关联点。在一种具体的实施示例中，可以将样本图像中与目标点同属于一个超体素的点作为目标点的关联点，具体的，超像素分割结果包括多个超体素，每个超体素都可能会包含多个特征相似且位置相邻的点，对于每个目标点而言，可以将与该目标点同属于一个超体素的其余点均作为该目标点的关联点，也可以将与目标点同属于一个超体素的其余点中的部分点(诸如在超体素中与目标点相邻的点)作为目标点的关联点，在此不进行限制。
步骤c，将目标点所属的语义类别和实例类别分别作为关联点所属的语义类别和实例类别，得到样本图像的扩展标签。
关联点与目标点的低级特征相似且关联点通常与目标点相邻，因此关联点与目标点属于同一个物体的概率极大，所以可以将目标点所 属的语义类别和实例类别分别作为关联点所属的语义类别和实例类别，通过上述步骤a～步骤c所提供的方式可以在初期得到较为可靠准确的扩展标签，以便于后续采用数量相对较多的标签进行模型的初始训练。
在一些实施方式中，基于初始标签和扩展标签对预设的神经网络模型进行训练，并将训练结束时的神经网络模型作为实例分割模型的步骤，可以参照如下步骤一～步骤三实现:
步骤一，基于初始标签和扩展标签对初始的神经网络模型进行第一训练，将达到第一训练结束条件时的神经网络模型作为第一神经网络模型。其中，第一训练结束条件可以包括以下中的一种或多种：总损失函数值收敛至第一预设阈值，达到第一训练次数阈值、达到第一训练时长阈值。其中，总损失函数值基于标签与模型输出结果确定。
在实际应用中，可以将样本图像输入至神经网络模型中，获取神经网络模型针对样本图像输出的分割结果，基于标签(初始标签和扩展标签)和预设的损失函数对分割结果进行评估，得到总损失函数值。。在一些具体的实施示例中，在训练期间神经网络模型输出的分割结果可以仅采用样本图像中每个点的实例特征值、语义概率和偏移向量等信息表征，其中，语义概率可直接用于语义分割，实例特征值和偏移向量可用于进行实例分割。然后基于标签(初始标签和扩展标签)对分割结果进行评估，诸如，基于标签和判别损失(Discriminative Loss)函数确定实例特征值对应的第一损失值，基于标签和交叉熵损失(Cross Entropy Loss)函数确定语义概率对应的第二损失值，基于L2损失函数确定偏移向量对应的第三损失值，将第一损失值、第二损失值和第三损失值加权求和得到总损失值(也即前述总损失函数值的简称)，基于总损失值调整神经网络模型的参数，直至达到第一训练结束条件。此时即可得到初步训练好的第一神经网络模型，第一神经网络模型具一定的实例分割能力。
步骤二，获取第一神经网络模型针对样本图像中每个待测点的输出结果，基于输出结果生成伪标签；伪标签包括样本图像中每个点所 属的语义类别和实例类别。在实际应用中，可以将样本图像中的每个点均作为待测点，也可以将样本图像中的部分点作为待测点，样本图像中的部分点可以是随机确定的多个点，也可以是指定的多个点；诸如可以随机确定样本图像中预设数量个点作为待测点，也可以根据需求而指定分布在样本图像中不同位置的多个点作为待测点。倘若待测点仅是样本图像中的部分点，还可以进一步设定数量阈值，令待测点的数量需要高于预设数量阈值，以此确保模型训练的可靠性。
输出结果包括实例特征值、语义概率和偏移向量；在此基础上，在基于输出结果生成伪标签时，可以基于样本图像中每个待测点的实例特征值、语义概率和偏移向量，以及目标点所属的语义类别和实例类别，采用预设聚类算法生成伪标签。可以理解的是，伪标签并非人为真正标记的标签，而是模型在训练过程中针对无标记数据的预测结果，由于不一定精准，所以可称为伪标签(或伪标记)。但是伪标签可帮助模型学习到无标注数据中隐藏的信息，通过将模型自生成的伪标签再应用于模型训练过程中进行监督，可较为可靠地达到基于自训练的弱监督学习效果。在本公开实施例中，可以基于神经网络模型输出的样本图像中每个待测点的实例特征值、语义概率和偏移向量按照聚类算法进行聚类，根据聚类结果和已经确知的目标点所属的语义类别和实例类别，便可便捷地得到每个待测点所属的语义类别和实例类别，生成的伪标签也相对准确。另外，本公开实施例对聚类算法可不进行限制，包括但不限于K-means聚类算法(K均值聚类)等。
为便于理解，本公开实施例进一步给出了采用预设聚类算法生成伪标签的具体实施方式，示例性地，可以参照如下步骤A～步骤B实现：
步骤A，将目标点作为预设聚类算法中的初始聚类中心，基于初始聚类中心和样本图像中每个待测点的实例特征值、语义概率和偏移向量进行聚类处理。
聚类中心也可称为种子，初始聚类中心也可称为初始种子，最初可将目标点作为初始聚类中心，然后进行多次迭代聚类，直至聚类中心不再发生变化。具体而言，可以参照如下实现:
在首次聚类处理时，基于样本图像中每个待测点的实例特征值、语义概率和偏移向量，得到每个待测点与初始聚类中心之间的第一相似度(也可称为第一距离)，根据第一相似度确定新的聚类中心。具体的，可以根据每个待测点与每个初始聚类中心对应的第一相似度进行类别划分，将每个待测点归到和其相似度最近的聚类中心所属的类，然后再基于初步得到的类簇重新计算每个聚类中心的位置(诸如将每个类内的所有点取平均值等)。在非首次聚类处理时，基于样本图像中每个待测点的实例特征值、语义概率和偏移向量，得到每个待测点与上一次聚类处理所得的聚类中心之间的第二相似度(也可称为第二距离)，根据第二相似度确定新的聚类中心(可参见上述确定新的聚类中心的具体实现方式)，直至聚类中心不再改变时结束聚类处理。其中，聚类中心的数量等于目标点的数量，也即等于样本图像中的物体数量。上述相似度在聚类算法中也可被称为距离。通过上述多次迭代处理，反复调整聚类中心，最终在聚类中心不再发生改变时停止聚类，便可以得到较为准确的聚类结果。而且由于本公开实施例将每个物体上标注的目标点作为初始聚类中心，不同目标点对应不同物体，这种方式有助于提升聚类效率，在较少的迭代次数内便可找到最终的聚类中心，可有效节约计算开销，较好地避免了最初选取不合适的初始聚类中心而导致较多的计算开销的问题；另外，上述方式在一定程度上也可保障聚类结果的可靠性，最终所得的每个类簇都可较为准确地对应一个物体。
步骤B，根据聚类结果以及目标点所属的语义类别和实例类别，得到样本图像中每个待测点所属的语义类别和实例类别。
在一些具体的实施示例中，可以根据聚类结果获取目标点的同簇点；可以理解的是，聚类结果包括多个类簇，同簇点为样本图像中与 目标点同属一个类簇的待测点，聚类结果中的类簇与样本图像中的物体一一对应(也即不同类簇对应的物体不同)；换言之，通过上述聚类结果可以找到与每个目标点同属于一个物体的其余待测点，之后便可直接将目标点所属的语义类别和实例类别分别作为同簇点的语义类别和实例类别。可以理解的是，在样本图像中的每个待测点都有其所属的物体，而每个物体都已标注有一个目标点，因此每个待测点都会对应有与其同属一个物体的目标点。对于每个物体而言，通过上述聚类方式得到该物体的类簇，而该物体对应的类簇中目标点的语义类别和实例类别是已知的，其余待测点的语义类别和实例类别理论上与目标点的语义类别和实例类别是一致的，此时便可根据该物体中目标点的语义类别和实例类别直接确定该物体中其余待测点的语义类别和实例类别，通过上述方式，样本图像中的所有待测点的语义类别和实例类别便均可获知。
通过上述步骤A和步骤B，将目标点作为初始聚类中心的方式可有效提升聚类效率，节约聚类所需的计算开销，并保障聚类结果的可靠性，在此基础上通过聚类方式可以较为准确地找到与目标点同属一个物体的其余待测点，并将每个物体中目标点的语义类别和实例类别直接作为该物体中其余待测点的语义类别和实例类别，从而可以高效且准确地对每个未标注的待测点进行自动标注，得到模型输出的伪标签。
步骤三，基于伪标签对第一神经网络模型进行第二训练，并将达到第二训练结束条件时的第一神经网络模型作为实例分割模型。
基于伪标签对第一神经网络模型进行第二训练的过程与前述基于初始标签和扩展标签对初始的神经网络模型进行第一训练的过程类似，区别在于第一训练为初始训练，采用的是初始标签和基于初始标签扩展得到的标签，第二训练为模型自训练，采用的是初步训练好的第一神经网络模型输出结果所得到的伪标签，也即训练时所采用的标签不同。上述第二训练结束条件可以包括以下中的一种或多种：总损失函 数值收敛至第二预设阈值，达到第二训练次数阈值、达到第二训练时长阈值。上述总损失函数值的计算方式也参照第一训练过程中计算总损失函数值的方式，诸如需要实例特征值对应的第一损失值、语义概率对应的第二损失值和偏移向量对应的第三损失值进行加权求和得到总损失函数值，在此不再赘述。另外应当说明的是，第一训练和第二训练都表示训练阶段，而不代表训练次数，无论是第一训练还是第二训练，都会训练多次(也即通过反向传播的方式多次调整模型参数)，每次训练都会输出伪标签，输出的伪标签用以监督模型在下一次训练时的输出结果，以此来调整模型参数，直至达到预设的训练结束条件时停止训练。
通过上述步骤一～步骤三，首先采用初始标签和扩展标签对神经网络模型进行初步训练，在神经网络模型具有一定实例分割能力后，再基于神经网络模型的输出结果生成伪标签，采用伪标签对神经网络模型进行自训练，上述方式最初只需要针对每个物体标注一个点，后续标签均可基于初始标注的点进行自动生成，极大降低了标注成本，而且上述无论是基于低级特征获得扩展标签还是基于聚类方式获得伪标签的方式较为可靠，可以有效保障用于训练模型的标签的准确性，也在一定程度上保障了训练所得的模型精度。
为了便于理解，可参照图2所示的一种模型训练示意图，在该示意图中重点示意出将样本图像输入至神经网络模型，神经网络模型可针对样本图像中的每个点分别输出实例特征值、语义概率和偏移向量，基于输出结果便可生成伪标签，伪标签包含样本图像中每个点所属的语义类别和实例类别，其中，语义类别基于语义概率得到，实例类别基于实例特征值和偏移向量得到；然后便可采用伪标签对神经网络模型进行监督，从而调整神经网络模型的参数。在图2中示意出每个物体都只需标注一个目标点，同时示意出基于模型的输出结果生成伪标签的方式，具体的，可基于模型针对样本图像输出的每个点的实例特征值、语义概率和偏移向量采用聚类算法进行聚类处理，其中，目标 点为初始聚类中心，分别为O1、O2和O3(以不同灰度的实心圈表示)，以随意一个点为例，该点与O1之间的相似度为0.8、与O2之间的相似度为0.6，与O3之间的相似度为0.2，通过计算每个点分别与O1、O2和O3之间的相似度后可以重新聚类，不同类簇对应的灰度不同；然后找到新的类簇的聚类中心O1’、O2’、O3’(以不同灰度的实心圈表示)，之后再次计算每个点与新的聚类中心O1’、O2’、O3’之间的相似度，再基于相似度重新聚类，经过多次聚类后最终得到聚类中心不再改变的类簇，每个类簇对应一个物体，从而基于每个点所属类簇中的目标点的语义类别和实例类别，得到每个点的语义类别和实例类别，较为便捷可靠地生成伪标签。另外应当说明的是，图2仅为简单示例，并未将所有类簇/所有点均示意出。
在前述基础上，本公开实施例提供了一种实例分割方法，参见图3所示的一种实例分割方法的流程示意图，主要包括如下步骤S302～步骤S306：
步骤S302，获取目标图像。目标图像即为待进行实例分割的图像。
步骤S304，将目标图像输入至预先训练得到的实例分割模型；其中，实例分割模型是采用上述任一项模型训练方法得到的，具体训练方式可以参照上述相关内容，在此不进行限定。
步骤S306，获取实例分割模型输出的实例分割结果；实例分割结果包括目标图像中每个点所属的语义类别和实例类别。具体实现时，实例分割模型可以针对目标图像输出每个点对应的实例特征值、语义概率和偏移向量，基于语义概率即可得到每个点所属的语义类别，基于实例特征值和语义概率即可计算不同点之间的相似度，从而通过诸如Mean-shift聚类方式得到每个点所属的实例类别。
由于实例分割方法所采用的实例分割模型所需的标注成本较低，也即获取实例分割模型更为方便，因此也有助于提升实例分割方法的普适性。进一步，基于前述方式无论是基于低级特征获得扩展标签还是基于聚类方式获得伪标签的方式均较为可靠，相比于常规的弱监督 训练方式而言，本公开实施例提供的上述方式可以有效保障用于训练模型的标签的准确性，因此训练所得的实例分割模型的精确度较高，也即本公开实施例只需极少的标注信息便可达到较高的分割精度。
对应于前述模型训练方法，本公开实施例提供了一种模型训练装置的结构示意图，图4为本公开实施例提供的一种模型训练装置的结构示意图，该装置可由软件和/或硬件实现，一般可集成在电子设备中，如图4所示，包括：
样本获取模块402，用于获取样本图像；
初始标签获取模块404，用于获取样本图像的初始标签；其中，初始标签包括目标点所属的语义类别和实例类别，样本图像中每个物体分别标注有一个目标点，且物体标注的目标点为物体中所包含的一个点；
扩展标签生成模块406，用于基于初始标签生成样本图像的扩展标签；其中，扩展标签包括目标点的关联点所属的语义类别和实例类别，且关联点所属的语义类别和实例类别是基于目标点所属的语义类别和实例类别确定的；
模型训练模块408，用于基于初始标签和扩展标签对初始的神经网络模型进行训练，并将训练结束时的神经网络模型作为实例分割模型。
本公开实施例提供的上述技术方案，在模型训练时可以只针对样本图像中每个物体分别标注一个目标点即可，之后便可基于初始标签(目标点所属的语义类别和实例类别)进行扩展，得到扩展标签(目标点的关联点所属的语义类别和实例类别)，从而基于初始标签和扩展标签进行模型训练，便可训练得到实例分割模型及进行后续实例分割。上述方式只需最初针对每个物体标注一个点即可，极大节约了标注成本。
在一些实施方式中，扩展标签生成模块406具体用于：获取所述样本图像的低级特征；根据所述低级特征对所述样本图像进行分割处理，基于分割结果得到所述目标点的关联点；将所述目标点所属的语 义类别和实例类别分别作为所述关联点所属的语义类别和实例类别，得到所述样本图像的扩展标签。
在一些实施方式中，扩展标签生成模块406具体用于下述至少一项：在所述样本图像为二维图像的情况下，根据所述低级特征对所述样本图像进行超像素分割处理，基于分割结果得到所述目标点的关联点；在所述样本图像为三维图像的情况下，根据所述低级特征对所述样本图像进行超体素分割处理，基于分割结果得到所述目标点的关联点。
在一些实施方式中，模型训练模块408具体用于：基于所述初始标签和所述扩展标签对初始的神经网络模型进行第一训练，将达到第一训练结束条件时的神经网络模型作为第一神经网络模型；获取所述第一神经网络模型针对所述样本图像中每个待测点的输出结果，基于所述输出结果生成伪标签；所述伪标签包括所述样本图像中每个所述待测点所属的语义类别和实例类别；基于所述伪标签对所述第一神经网络模型进行第二训练，并将达到第二训练结束条件时的第一神经网络模型作为实例分割模型。
在一些实施方式中，所述输出结果包括实例特征值、语义概率和偏移向量；模型训练模块408具体用于：基于所述样本图像中每个所述待测点的实例特征值、语义概率和偏移向量，以及所述目标点所属的语义类别和实例类别，采用预设聚类算法生成伪标签。
在一些实施方式中，模型训练模块408具体用于：将所述目标点作为预设聚类算法中的初始聚类中心，基于所述初始聚类中心和所述样本图像中每个所述待测点的实例特征值、语义概率和偏移向量进行聚类处理；根据聚类结果以及所述目标点所属的语义类别和实例类别，得到所述样本图像中每个所述待测点所属的语义类别和实例类别。
在一些实施方式中，模型训练模块408具体用于：在首次聚类处理时，基于所述样本图像中每个所述待测点的实例特征值、语义概率和偏移向量，得到每个所述待测点与所述初始聚类中心之间的第一相 似度，根据所述第一相似度确定新的聚类中心；在非首次聚类处理时，基于所述样本图像中每个所述待测点的实例特征值、语义概率和偏移向量，得到每个所述待测点与上一次聚类处理所得的聚类中心之间的第二相似度，根据所述第二相似度确定新的聚类中心，直至聚类中心不再改变时结束聚类处理。
在一些实施方式中，模型训练模块408具体用于：根据聚类结果获取所述目标点的同簇点；其中，所述同簇点为所述样本图像中与所述目标点同属一个类簇的所述待测点，所述聚类结果中的类簇与所述样本图像中的物体一一对应；将所述目标点所属的语义类别和实例类别分别作为所述同簇点的语义类别和实例类别。
本公开实施例所提供的模型训练装置可执行本公开任意实施例所提供的模型训练方法，具备执行方法相应的功能模块和有益效果。
对应于前述实例分割方法，图5为本公开实施例提供的一种实例分割装置的结构示意图，该装置可由软件和/或硬件实现，一般可集成在电子设备中。如图5所示，包括：
图像获取模块502，用于获取目标图像；
模型输入模块504，用于将所述目标图像输入至预先训练得到的实例分割模型；其中，所述实例分割模型是采用任一项所述的模型训练方法得到的；
结果获取模块506，用于获取所述实例分割模型输出的实例分割结果；所述实例分割结果包括所述目标图像中每个点所属的语义类别和实例类别。
由于实例分割装置所采用的实例分割模型所需的标注成本较低，也即获取实例分割模型更为方便，因此也有助于提升实例分割方法的普适性。
本公开实施例所提供的实例分割装置可执行本公开任意实施例所提供的实例分割方法，具备执行方法相应的功能模块和有益效果。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的装置实施例的具体工作过程，可以参考方法实施例中的对应过程，在此不再赘述。
本公开实施例提供一种电子设备，电子设备包括：处理器；用于存储处理器可执行指令的存储器；处理器，用于从存储器中读取可执行指令，并执行指令以实现上述任一模型训练方法或者实例分割方法。
图6为本公开实施例提供的一种电子设备的结构示意图。如图6所示，电子设备600包括一个或多个处理器601和存储器602。
处理器601可以是中央处理单元(CPU)或者具有数据处理能力和/或指令执行能力的其他形式的处理单元，并且可以控制电子设备600中的其他组件以执行期望的功能。
存储器602可以包括一个或多个计算机程序产品，所述计算机程序产品可以包括各种形式的计算机可读存储介质，例如易失性存储器和/或非易失性存储器。所述易失性存储器例如可以包括随机存取存储器(RAM)和/或高速缓冲存储器(cache)等。所述非易失性存储器例如可以包括只读存储器(ROM)、硬盘、闪存等。在所述计算机可读存储介质上可以存储一个或多个计算机程序指令，处理器601可以运行所述程序指令，以实现上文所述的本公开的实施例的模型训练方法或实例分割方法以及/或者其他期望的功能。在所述计算机可读存储介质中还可以存储诸如输入信号、信号分量、噪声分量等各种内容。
在一个示例中，电子设备600还可以包括：输入装置603和输出装置604，这些组件通过总线系统和/或其他形式的连接机构(未示出)互连。
此外，该输入装置603还可以包括例如键盘、鼠标等等。
该输出装置604可以向外部输出各种信息，包括确定出的距离信息、方向信息等。该输出装置604可以包括例如显示器、扬声器、打印机、以及通信网络及其所连接的远程输出设备等等。
当然，为了简化，图6中仅示出了该电子设备600中与本公开有关的组件中的一些，省略了诸如总线、输入/输出接口等等的组件。除此之外，根据具体应用情况，电子设备600还可以包括任何其他适当的组件。
除了上述方法和设备以外，本公开的实施例还可以是计算机程序产品，其包括计算机程序指令，所述计算机程序指令在被处理器运行时使得所述处理器执行本公开实施例所提供的模型训练方法或实例分割方法。
所述计算机程序产品可以以一种或多种程序设计语言的任意组合来编写用于执行本公开实施例操作的程序代码，所述程序设计语言包括面向对象的程序设计语言，诸如Java、C++等，还包括常规的过程式程序设计语言，诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算设备上执行、部分地在用户设备上执行、作为一个独立的软件包执行、部分在用户计算设备上部分在远程计算设备上执行、或者完全在远程计算设备或服务器上执行。
此外，本公开的实施例还可以是计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令在被处理器运行时使得所述处理器执行本公开实施例所提供的模型训练方法或实例分割方法。
所述计算机可读存储介质可以采用一个或多个可读介质的任意组合。可读介质可以是可读信号介质或者可读存储介质。可读存储介质例如可以包括但不限于电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。可读存储介质的更具体的例子(非穷举的列表)包括：具有一个或多个导线的电连接、便携式盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。
本公开实施例还提供了一种计算机程序产品，包括计算机程序/指令，该计算机程序/指令被处理器执行时实现本公开实施例中的模型训练方法或实例分割方法。
需要说明的是，在本文中，诸如“第一”和“第二”等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上所述仅是本公开的具体实施方式，使本领域技术人员能够理解或实现本公开。对这些实施例的多种修改对本领域的技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本公开的精神或范围的情况下，在其它实施例中实现。因此，本公开将不会被限制于本文所述的这些实施例，而是要符合与本文所公开的原理和新颖特点相一致的最宽的范围。

Claims (13)

  1. 一种模型训练方法，其特征在于，包括：
    获取样本图像；
    获取所述样本图像的初始标签；其中，所述初始标签包括目标点所属的语义类别和实例类别，所述样本图像中每个物体分别标注有一个目标点，且所述物体标注的目标点为所述物体中所包含的一个点；
    基于所述初始标签生成所述样本图像的扩展标签；其中，所述扩展标签包括所述目标点的关联点所属的语义类别和实例类别，且所述关联点所属的语义类别和实例类别是基于所述目标点所属的语义类别和实例类别确定的；
    基于所述初始标签和所述扩展标签对初始的神经网络模型进行训练，并将训练结束时的所述神经网络模型作为实例分割模型。
  2. 根据权利要求1所述的方法，其特征在于，基于所述初始标签生成所述样本图像的扩展标签的步骤，包括：
    获取所述样本图像的低级特征；
    根据所述低级特征对所述样本图像进行分割处理，基于分割结果得到所述目标点的关联点；
    将所述目标点所属的语义类别和实例类别分别作为所述关联点所属的语义类别和实例类别，得到所述样本图像的扩展标签。
  3. 根据权利要求2所述的方法，其特征在于，根据所述低级特征对所述样本图像进行分割处理，基于分割结果得到所述目标点的关联点的步骤，包括下述至少一项：
    在所述样本图像为二维图像的情况下，根据所述低级特征对所述样本图像进行超像素分割处理，基于分割结果得到所述目标点的关联点；
    在所述样本图像为三维图像的情况下，根据所述低级特征对所述样本图像进行超体素分割处理，基于分割结果得到所述目标点的关联点。
  4. 根据权利要求1-3任一项所述的方法，其特征在于，基于所述初始标签和所述扩展标签对预设的神经网络模型进行训练，并将训练结束时的所述神经网络模型作为实例分割模型的步骤，包括：
    基于所述初始标签和所述扩展标签对初始的神经网络模型进行第一训练，将达到第一训练结束条件时的神经网络模型作为第一神经网络模型；
    获取所述第一神经网络模型针对所述样本图像中每个待测点的输出结果，基于所述输出结果生成伪标签；所述伪标签包括所述样本图像中每个所述待测点所属的语义类别和实例类别；
    基于所述伪标签对所述第一神经网络模型进行第二训练，并将达到第二训练结束条件时的第一神经网络模型作为实例分割模型。
  5. 根据权利要求4所述的方法，其特征在于，所述输出结果包括实例特征值、语义概率和偏移向量；基于所述输出结果生成伪标签的步骤，包括：
    基于所述样本图像中每个所述待测点的实例特征值、语义概率和偏移向量，以及所述目标点所属的语义类别和实例类别，采用预设聚类算法生成伪标签。
  6. 根据权利要求5所述的方法，其特征在于，采用预设聚类算法生成伪标签的步骤，包括：
    将所述目标点作为预设聚类算法中的初始聚类中心，基于所述初始聚类中心和所述样本图像中每个所述待测点的实例特征值、语义概率和偏移向量进行聚类处理；
    根据聚类结果以及所述目标点所属的语义类别和实例类别，得到所述样本图像中每个所述待测点所属的语义类别和实例类别。
  7. 根据权利要求6所述的方法，其特征在于，基于所述初始聚类中心和所述样本图像中每个所述待测点的实例特征值、语义概率和偏移向量进行聚类处理的步骤，包括：
    在首次聚类处理时，基于所述样本图像中每个所述待测点的实例特征值、语义概率和偏移向量，得到每个所述待测点与所述初始聚类中心之间的第一相似度，根据所述第一相似度确定新的聚类中心；
    在非首次聚类处理时，基于所述样本图像中每个所述待测点的实例特征值、语义概率和偏移向量，得到每个所述待测点与上一次聚类处理所得的聚类中心之间的第二相似度，根据所述第二相似度确定新的聚类中心，直至聚类中心不再改变时结束聚类处理。
  8. 根据权利要求6所述的方法，其特征在于，根据聚类结果以及所述目标点所属的语义类别和实例类别，得到所述样本图像中每个所述待测点所属的语义类别和实例类别的步骤，包括：
    根据聚类结果获取所述目标点的同簇点；其中，所述同簇点为所述样本图像中与所述目标点同属一个类簇的所述待测点，所述聚类结果中的类簇与所述样本图像中的物体一一对应；
    将所述目标点所属的语义类别和实例类别分别作为所述同簇点的语义类别和实例类别。
  9. 一种实例分割方法，其特征在于，包括：
    获取目标图像；
    将所述目标图像输入至预先训练得到的实例分割模型；其中，所述实例分割模型是采用权利要求1至8任一项所述的模型训练方法得到的；
    获取所述实例分割模型输出的实例分割结果；所述实例分割结果包括所述目标图像中每个点所属的语义类别和实例类别。
  10. 一种模型训练装置，其特征在于，包括：
    样本获取模块，用于获取样本图像；
    初始标签获取模块，用于获取所述样本图像的初始标签；其中，所述初始标签包括目标点所属的语义类别和实例类别，所述样本图像中每个物体分别标注有一个目标点，且所述物体标注的目标点为所述物体中所包含的一个点；
    扩展标签生成模块，用于基于所述初始标签生成所述样本图像的扩展标签；其中，所述扩展标签包括所述目标点的关联点所属的语义类别和实例类别，且所述关联点所属的语义类别和实例类别是基于所述目标点所属的语义类别和实例类别确定的；
    模型训练模块，用于基于所述初始标签和所述扩展标签对初始的神经网络模型进行训练，并将训练结束时的所述神经网络模型作为实例分割模型。
  11. 一种实例分割装置，其特征在于，包括：
    图像获取模块，用于获取目标图像；
    模型输入模块，用于将所述目标图像输入至预先训练得到的实例分割模型；其中，所述实例分割模型是采用任一项所述的模型训练方法得到的；
    结果获取模块，用于获取所述实例分割模型输出的实例分割结果；所述实例分割结果包括所述目标图像中每个点所属的语义类别和实例类别。
  12. 一种电子设备，其特征在于，所述电子设备包括：
    处理器；
    用于存储所述处理器可执行指令的存储器；
    所述处理器，用于从所述存储器中读取所述可执行指令，并执行所述指令以实现上述权利要求1-8中任一所述的模型训练方法或者权利要求9所述的实例分割方法。
  13. 一种计算机可读存储介质，其特征在于，所述存储介质存储有计算机程序，所述计算机程序用于执行上述权利要求1-8中任一所述的模型训练方法或者权利要求9所述的实例分割方法。
PCT/CN2023/107725 2022-07-18 2023-07-17 模型训练方法、实例分割方法、装置、设备及介质 Ceased WO2024017199A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210843130.6A CN115115923B (zh) 2022-07-18 2022-07-18 模型训练方法、实例分割方法、装置、设备及介质
CN202210843130.6 2022-07-18

Publications (1)

Publication Number Publication Date
WO2024017199A1 true WO2024017199A1 (zh) 2024-01-25

Family

ID=83331876

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/107725 Ceased WO2024017199A1 (zh) 2022-07-18 2023-07-17 模型训练方法、实例分割方法、装置、设备及介质

Country Status (2)

Country Link
CN (1) CN115115923B (zh)
WO (1) WO2024017199A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118279648A (zh) * 2024-03-26 2024-07-02 北京积加科技有限公司 图像分类标签生成方法、装置和电子设备

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115115923B (zh) * 2022-07-18 2024-04-09 北京有竹居网络技术有限公司 模型训练方法、实例分割方法、装置、设备及介质
CN115690465A (zh) * 2022-10-17 2023-02-03 西安闻泰信息技术有限公司 图像多标签分类方法、装置、计算机设备和存储介质
CN116433704A (zh) * 2022-12-29 2023-07-14 鹏城实验室 一种基于中心点的细胞核分割方法及相关设备
CN115953778A (zh) * 2022-12-30 2023-04-11 东软睿驰汽车技术（沈阳）有限公司 场景语义分割模型训练方法、装置和电子设备
CN119785030B (zh) * 2024-12-25 2025-11-18 温州电力设计有限公司普华招标咨询分公司 超像素语义基元生成方法、系统、电子设备和存储介质
CN120765946B (zh) * 2025-09-09 2025-11-28 宁波博登智能科技有限公司 点云序列语义分割模型的训练方法及应用

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112927244A (zh) * 2021-03-31 2021-06-08 清华大学 弱监督下的三维场景分割方法和装置
US20220207861A1 (en) * 2020-12-25 2022-06-30 Nec Corporation Methods, devices, and computer readable storage media for image processing
CN115115923A (zh) * 2022-07-18 2022-09-27 北京有竹居网络技术有限公司 模型训练方法、实例分割方法、装置、设备及介质

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11244504B2 (en) * 2019-05-03 2022-02-08 Facebook Technologies, Llc Semantic fusion
US11367188B2 (en) * 2019-10-18 2022-06-21 Retrace Labs Dental image synthesis using generative adversarial networks with semantic activation blocks
CN111488923B (zh) * 2020-04-03 2023-02-07 陕西师范大学 增强的锚点图半监督分类方法
CN113628224B (zh) * 2021-08-09 2023-12-19 南通大学 一种基于三维欧式距离变换的房间分割方法
CN114092487A (zh) * 2021-10-13 2022-02-25 山东师范大学 目标果实实例分割方法及系统
CN113971727A (zh) * 2021-10-21 2022-01-25 京东鲲鹏(江苏)科技有限公司 一种语义分割模型的训练方法、装置、设备和介质
CN113989305B (zh) * 2021-12-27 2022-04-22 城云科技（中国）有限公司 目标语义分割方法及应用其的街道目标异常检测方法
CN114723949A (zh) * 2022-04-18 2022-07-08 北京百度网讯科技有限公司 三维场景分割方法和用于训练分割模型的方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220207861A1 (en) * 2020-12-25 2022-06-30 Nec Corporation Methods, devices, and computer readable storage media for image processing
CN112927244A (zh) * 2021-03-31 2021-06-08 清华大学 弱监督下的三维场景分割方法和装置
CN115115923A (zh) * 2022-07-18 2022-09-27 北京有竹居网络技术有限公司 模型训练方法、实例分割方法、装置、设备及介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118279648A (zh) * 2024-03-26 2024-07-02 北京积加科技有限公司 图像分类标签生成方法、装置和电子设备

Also Published As

Publication number Publication date
CN115115923A (zh) 2022-09-27
CN115115923B (zh) 2024-04-09

Similar Documents

Publication Publication Date Title
WO2024017199A1 (zh) 模型训练方法、实例分割方法、装置、设备及介质
JP7128022B2 (ja) 完全教師あり学習用のデータセットの形成
CN108038474B (zh) 人脸检测方法、卷积神经网络参数的训练方法、装置及介质
Levinshtein et al. Turbopixels: Fast superpixels using geometric flows
US20220156992A1 (en) Image segmentation using text embedding
CN113743426A (zh) 一种训练方法、装置、设备以及计算机可读存储介质
CN113204988B (zh) 小样本视点估计
CN116342887B (zh) 用于图像分割的方法、装置、设备和存储介质
CN110363049A (zh) 图形元素检测识别和类别确定的方法及装置
CN107330908B (zh) 一种基于卷积受限玻尔兹曼机的形状建模方法及相关装置
CN110929802A (zh) 基于信息熵的细分类识别模型训练、图像识别方法及装置
Zhang et al. Adaptive NormalHedge for robust visual tracking
CN115424267A (zh) 基于高斯分布的旋转目标检测方法和装置
CN113822144A (zh) 一种目标检测方法、装置、计算机设备和存储介质
CN119516285A (zh) 基于预训练扩散模型处理受限数据的高精度姿态估计方法
US20240185578A1 (en) Image encoding learning and application
CN109189941B (zh) 用于更新模型参数的方法、装置、设备和介质
CN111177811A (zh) 一种应用于云平台的消防点位自动布图的方法
CN117152527B (zh) 一种基于图组合优化的稀疏标记遥感图像目标检测方法
Chabot et al. Lapnet: Automatic balanced loss and optimal assignment for real-time dense object detection
CN112348161B (zh) 神经网络的训练方法、神经网络的训练装置和电子设备
US20220215228A1 (en) Detection method, computer-readable recording medium storing detection program, and detection device
CN112949590B (zh) 一种跨域行人重识别模型构建方法及构建系统
CN114565017A (zh) 基于标签到标签的多属性预测方法、装置、设备及介质
CN114842251A (zh) 图像分类模型的训练及图像处理方法、装置和计算设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23842262

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23842262

Country of ref document: EP

Kind code of ref document: A1