CN117934488A - Construction and optimization method of three-dimensional shape segmentation framework based on semi-supervision and electronic equipment - Google Patents

Abstract

The invention relates to a method for constructing and optimizing a three-dimensional shape segmentation framework based on semi-supervision and electronic equipment; training the auxiliary segmentation network by inputting the third data set into the auxiliary segmentation network module; inputting the second data set into the trained auxiliary segmentation network module to output a three-dimensional shape containing complete labels, namely a pseudo tag, and obtaining a fifth data set; training a primary segmentation network through the first data set and the fifth data set; fusing the pseudo tag with the patch characteristics output by the main segmentation network through the self-refinement module; calculating a cross entropy loss value by using the real label, the fused pseudo label and a prediction label generated by the main segmentation network through a calculation module, and adjusting network parameters of the main segmentation network according to the cross entropy loss value; the method and the device generate the corresponding complete label for the sparse label in the second data set through the trained auxiliary segmentation network, amplify the training data for training the main segmentation network, and avoid the cost of expensive full-label training data.

Description

Construction and optimization method of three-dimensional shape segmentation framework based on semi-supervision and electronic device

Technical Field

The present application relates to the field of three-dimensional shape segmentation, and in particular to a construction and optimization method of a semi-supervised three-dimensional shape segmentation framework and an electronic device.

Background technique

The 3D shape segmentation task involves dividing a 3D shape into meaningful parts and is crucial for efficient processing of 3D shapes. It enables us to understand the intrinsic properties of a shape, such as its topological structure, more easily. Therefore, various tasks, including mesh editing, reconstruction, modeling, deformation, and shape retrieval, rely on 3D shape segmentation to obtain satisfactory results. Currently, shape segmentation has become one of the most popular and challenging research areas.

Traditional 3D shape segmentation methods usually consist of three main steps. First, each face on the shape is mapped to a corresponding feature vector using hand-crafted shape descriptors. Subsequently, clustering or classification methods are applied in the feature space to assign labels to each feature vector. Finally, each face in the 3D shape is labeled according to the label of the corresponding feature vector. However, recent advances in machine learning have led to the development of learning-based segmentation methods, especially those based on deep learning architectures. These methods have shown significant improvements in performance compared to traditional geometric optimization methods.

Although learning-based segmentation methods, especially deep learning methods, have achieved impressive results, they have a major drawback, which is the need for a large amount of fully labeled training data with similar shapes to the target, which may bring significant time and cost burden in terms of manual labeling. In addition, these methods usually require preparing different training shape sets for each 3D shape of the target, which increases the complexity of the training process.

Summary of the invention

In order to overcome the limitations of the current learning-based segmentation method, reduce the complexity of the training process, and improve the accuracy of segmentation, according to one aspect of the embodiment of the present application, the present invention proposes a method for constructing a semi-supervised three-dimensional shape segmentation framework, wherein the three-dimensional shape segmentation framework includes an auxiliary segmentation network module, a self-refinement module, and a main segmentation network module including a main segmentation network and a calculation module; the auxiliary segmentation network module includes an auxiliary segmentation network for predicting pseudo labels, a projection module for projecting three-dimensional shapes, and a back-projection module;

The construction method comprises:

Acquire a first data set and a second data set; the data in the first data set are completely annotated three-dimensional shapes; the data in the second data set are three-dimensional shapes with sparse scribble annotations;

Sampling the reference label of each three-dimensional shape in the first data set to generate a corresponding three-dimensional shape with sparse scribble annotations, thereby obtaining a third data set;

The third data set is input into the auxiliary segmentation network module to train the auxiliary segmentation network, wherein the projection module obtains the multi-view projection images corresponding to each three-dimensional shape in the third data set to obtain a fourth data set, and establishes a reference matrix; the data in the reference matrix includes the vertex coordinates of each pixel corresponding to the original three-dimensional shape in the process of obtaining the multi-view projection images; the auxiliary segmentation network is trained based on the first objective function and the multi-view projection images in the fourth data set to predict the predicted labels corresponding to each two-dimensional image in the multi-view projection images; the back-projection module uses the reference matrix to map the predicted labels back to the corresponding three-dimensional shape patches to generate corresponding complete annotations for each patch of the three-dimensional shape in the third data set;

Inputting the second data set into the trained auxiliary segmentation network module to output a three-dimensional shape containing complete annotations, i.e., a pseudo label, to obtain a fifth data set;

The main segmentation network is trained by using the first data set and the fifth data set to output corresponding patch features for each patch of the three-dimensional shape and predict corresponding prediction labels;

The pseudo labels output by the auxiliary segmentation network module are fused with the patch features output by the main segmentation network through the self-refinement module to obtain the fused pseudo labels.

The calculation module uses the true label, the fused pseudo label and the predicted label generated by the main segmentation network to calculate the cross entropy loss value, and adjusts the network parameters of the main segmentation network according to the cross entropy loss value.

Further, the two-dimensional image of the multi-view projection image includes a depth map and a rendering image;

The back-projection module is specifically used to find the facets of corresponding coordinates in the three-dimensional shape according to the vertex coordinates recorded by each pixel in the reference matrix (corresponding position), obtain the correspondence between the predicted label in the two-dimensional image and the semantic label of each facet on the three-dimensional shape, and map the predicted label to the corresponding three-dimensional shape facet based on the correspondence.

Furthermore, the auxiliary segmentation network includes:

The Encoder module is used to receive the multi-view projection image and extract the image feature information corresponding to each two-dimensional image in the multi-view projection image;

The Decoder module is used to predict the label of the corresponding two-dimensional image according to the image feature information to obtain the predicted label corresponding to each two-dimensional image.

Furthermore, the main segmentation network includes:

The main subdivision module consists of four fully connected layers to extract patch features corresponding to each patch in the 3D shape;

Softmax module, used to predict the corresponding prediction label according to the patch features corresponding to each patch;

The calculation module is specifically used for:

For the first data set, the cross entropy loss value is calculated by the real label and the predicted label generated by the Softmax module. For the fifth data set, the cross entropy loss value is calculated by the fused pseudo label and the predicted label generated by the Softmax module.

Furthermore, the self-refinement module includes two convolutional layers for dynamically learning the weight distribution of the prediction results of the main segmentation network and the auxiliary segmentation network for the three-dimensional shape with sparse Scribble annotations during the prediction process.

Furthermore, the formula expression of the first objective function is:

Where F represents the first data set, s ^(f) represents the patch of the three-dimensional shape in the first data set; x ^(f) represents the feature vector of the patch s ^(f) , y ^(f) represents the predicted label of x ^(f) by the auxiliary segmentation network, θ represents the network parameters of the auxiliary segmentation network, and _paux represents the prediction function corresponding to the auxiliary segmentation network.

According to another aspect of the embodiment of the present application, an optimization method based on a semi-supervised three-dimensional shape segmentation framework is also proposed, including:

Get the target function;

The three-dimensional shape segmentation framework is optimized through the objective function; wherein the three-dimensional shape segmentation framework is obtained through the construction method of the three-dimensional shape segmentation framework based on semi-supervision described above.

Furthermore, the objective function is expressed as follows:

In the formula, and φ both represent the network parameters of the main segmentation network, λ represent the network parameters of the self-refinement module, F represents the first dataset, p _pri represents the prediction of the main segmentation network, x ^(f) represents the feature vector of the 3D shape patch in the first dataset F, y ^(f) represents the predicted label of x ^(f) by the main segmentation network, S represents the second dataset or the third dataset, q _conv represents the dynamic weight generated by the self-refinement module, x ^(s) represents the feature vector of the 3D shape patch in the second dataset or the third dataset, s ^(s) represents the 3D shape patch in the second dataset or the third dataset, Y represents the prediction of x ^(s) by the auxiliary segmentation network, and s ^(f) represents the 3D shape patch in the first dataset.

According to another aspect of the embodiment of the present application, a semi-supervised 3D shape segmentation method is also proposed, comprising:

Obtain an unlabeled three-dimensional shape to be segmented;

The unlabeled three-dimensional shape to be segmented is input into the main segmentation network for prediction to obtain a three-dimensional shape containing complete annotations; wherein the main segmentation network is trained by the construction method based on the semi-supervised three-dimensional shape segmentation framework described above.

According to another aspect of the embodiments of the present application, an electronic device is proposed, including: a processor, and a memory for storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the method in any of the above embodiments.

Compared with the prior art, the technical solution proposed in this application has the following technical effects:

(1) The present invention samples the reference labels of each three-dimensional shape in the first data set to generate corresponding three-dimensional shapes with sparse Scribble annotations to obtain a third data set; inputs the third data set into the auxiliary segmentation network module to train the auxiliary segmentation network; inputs the trained second data set into the auxiliary segmentation network module to output three-dimensional shapes containing complete annotations, namely pseudo labels, to obtain a fifth data set; trains the main segmentation network through the first data set and the fifth data set to output corresponding patch features for each facet of the three-dimensional shape and predict corresponding predicted labels; fuses the pseudo labels output by the auxiliary segmentation network module with the facet features output by the main segmentation network through the self-refinement module to obtain fused pseudo labels; calculates the cross entropy loss value using the real label, the fused pseudo label and the predicted label generated by the main segmentation network through the calculation module, and adjusts the network parameters of the main segmentation network according to the cross entropy loss value; that is, the present invention generates corresponding complete labels for the sparse annotations in the second data set through the trained auxiliary segmentation network, thereby expanding the training data used to train the main segmentation network, and effectively avoiding the cost of expensive fully labeled training data;

(2) The present invention fuses the pseudo-labels output by the auxiliary segmentation network module with the patch features output by the main segmentation network through the self-refinement module to obtain the fused pseudo-labels, and calculates the cross-entropy loss value using the real labels, the fused pseudo-labels and the predicted labels generated by the main segmentation network through the calculation module, and adjusts the network parameters of the main segmentation network according to the cross-entropy loss value, thereby greatly improving the prediction accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

FIG1 is a flow chart of a method for constructing a semi-supervised 3D shape segmentation framework;

FIG2 is a flow chart of an optimization method based on a semi-supervised 3D shape segmentation framework;

FIG3 is a flow chart of a semi-supervised three-dimensional shape segmentation method;

FIG4 is a three-dimensional shape graph with sparse Scribble annotations;

FIG5 is a three-dimensional shape diagram with complete annotations;

FIG6 is a structural diagram of the auxiliary segmentation network module;

FIG7 is a block diagram of a main segmentation network module;

FIG8 is a diagram of a semi-supervised 3D shape segmentation framework;

FIG. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

Embodiments of the present embodiment will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present embodiment are shown in the accompanying drawings, it should be understood that the present embodiment can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein, which are instead provided for a more thorough and complete understanding of the present embodiment. It should be understood that the drawings and embodiments of the present embodiment are only for exemplary purposes and are not intended to limit the scope of protection of the present embodiment.

In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

Semi-supervised learning is a key issue in the field of pattern recognition and machine learning. It is a learning method that combines supervised learning with unsupervised learning. Semi-supervised learning uses a large amount of unlabeled data and labeled data to perform pattern recognition. When using semi-supervised learning, it will require as few people as possible to do the work, while at the same time achieving relatively high accuracy.

In order to overcome the limitations of the current learning-based segmentation method, reduce the complexity of the training process, and improve the accuracy of segmentation, as shown in FIG1, the present invention proposes a method for constructing a three-dimensional shape segmentation framework based on semi-supervision, as shown in FIG8, the three-dimensional shape segmentation framework includes an auxiliary segmentation network module as shown in FIG6, a self-refinement module and a main segmentation network module including a main segmentation network and a calculation module as shown in FIG7; the auxiliary segmentation network module includes an auxiliary segmentation network for predicting pseudo labels, a projection module for projecting three-dimensional shapes, and a back-projection module;

The construction method comprises:

Acquire a first data set and a second data set; the data in the first data set is a completely annotated three-dimensional shape as shown in FIG5 ; the data in the second data set is a three-dimensional shape with sparse Scribble annotations as shown in FIG4 ;

It should be explained that this embodiment obtains the first data set and the second data set through some public benchmark data sets, including the PSB data set, the COSEG data set, and the ShapeNetCore data set. These public data sets provide a patch-level reference label for each three-dimensional shape. It should also be noted that the label of each three-dimensional shape in the first data set is a real label.

This embodiment adopts different data set division strategies ("1+2+2", "2+1+2") for subsequent training. Specifically, "1+2+2" means that 20% of the fully labeled shapes (becoming the first data set) and 40% of the shapes with sparse Scribble marks (becoming the second data set) in the benchmark data set are selected as the training data set, and the remaining 40% of the shapes are retained as the test data set. At the same time, "2+1+2" means that 40% of the fully labeled shapes (the first data set) and 20% of the shapes with sparse Scribble marks (the second data set) in the benchmark data set are selected as the training data set, and similarly, the remaining 40% of the shapes are retained as the test data set.

For each three-dimensional shape in the first data set, the reference label thereof is sampled according to a preset ratio to generate a corresponding three-dimensional shape with sparse Scribble annotations, thereby obtaining a third data set;

The third data set is input into the auxiliary segmentation network module to train the auxiliary segmentation network, wherein the projection module obtains the multi-view projection images corresponding to each three-dimensional shape in the third data set to obtain a fourth data set, and establishes a reference matrix; the data in the reference matrix includes the vertex coordinates of each pixel corresponding to the original three-dimensional shape in the process of obtaining the multi-view projection images; the auxiliary segmentation network is trained based on the first objective function and the multi-view projection images in the fourth data set to predict the predicted labels corresponding to each two-dimensional image in the multi-view projection images; the back-projection module uses the reference matrix to map the predicted labels back to the corresponding three-dimensional shape patches to generate corresponding complete annotations for each patch of the three-dimensional shape in the third data set;

Specifically, this embodiment performs a projection operation on each three-dimensional shape in the third data set: 32 virtual cameras are placed at preset positions of the spherical bounding box of the model (three-dimensional shape), and each camera is rotated four times at intervals of 90 degrees, thereby generating a total of 128 sets of depth maps and rendering images, which cover almost all vertices and faces of each three-dimensional shape in the data set.

Specifically, the method for establishing the reference matrix is: in the process of rendering the three-dimensional shape according to the camera posture, the vertex coordinates of each pixel corresponding to the original three-dimensional shape are recorded by using an additional rendering buffer and saved as a reference matrix.

The two-dimensional image of the multi-view projection image includes a depth map and a rendering image;

The auxiliary segmentation network comprises:

The Encoder module is used to receive the multi-view projection image and extract the image feature information corresponding to each two-dimensional image in the multi-view projection image;

The Decoder module is used to predict the label of the corresponding two-dimensional image according to the image feature information to obtain the predicted label corresponding to each two-dimensional image.

It should be noted that the Encoder module and the Decoder module are the main modules in the DeepLabv3+ image semantic segmentation network. This implementation generates a semantic label for each pixel of the input image by combining the Encoder module and the Decoder module.

The formula expression of the first objective function is:

Where F represents the first data set, s ^(f) represents the patch of the three-dimensional shape in the first data set; x ^(f) represents the feature vector of the patch s ^(f) , y ^(f) represents the predicted label of x ^(f) by the auxiliary segmentation network, θ represents the network parameters of the auxiliary segmentation network, and _paux represents the prediction function corresponding to the auxiliary segmentation network.

The back-projection module is specifically used to find the facets of corresponding coordinates in the three-dimensional shape according to the vertex coordinates recorded by each pixel in the reference matrix, obtain the correspondence between the predicted label in the two-dimensional image and the semantic label of each facet on the three-dimensional shape, and map the predicted label to the corresponding three-dimensional shape facet based on the correspondence.

Inputting the second data set into the trained auxiliary segmentation network module to output a three-dimensional shape containing complete annotations, i.e., a pseudo label, to obtain a fifth data set;

The main segmentation network is trained by using the first data set and the fifth data set to output corresponding patch features for each patch of the three-dimensional shape and predict corresponding prediction labels;

The main segmentation network includes:

The main subdivision module consists of four fully connected layers to extract patch features corresponding to each patch in the 3D shape;

Softmax module, used to predict the corresponding prediction label according to the patch features corresponding to each patch;

The calculation module is specifically used for:

For the first data set, the cross entropy loss value is calculated by the real label and the predicted label generated by the Softmax module. For the fifth data set, the cross entropy loss value is calculated by the fused pseudo label and the predicted label generated by the Softmax module.

The pseudo labels output by the auxiliary segmentation network module are fused with the patch features output by the main segmentation network through the self-refinement module to obtain the fused pseudo labels.

Since only the trained auxiliary segmentation network module is applied to the second data set, the accuracy of the pseudo-label obtained is very limited. Therefore, in order to improve the accuracy of the generated complete annotated data set, the present invention introduces a self-refinement module based on CNN convolution. This module fuses the prediction results of the main segmentation network and the auxiliary segmentation network modules to dynamically generate more accurate patch-level complete annotated data, thereby improving the prediction accuracy of the main segmentation network.

The self-refinement module includes two convolutional layers, which are used to dynamically learn the weight allocation of the prediction results of the main segmentation network and the auxiliary segmentation network for the three-dimensional shape with sparse Scribble annotations during the prediction process. For example, in the early stages of training, the prediction of the auxiliary segmentation network may be more accurate, so the self-refinement module will assign higher weights to the predicted labels generated by the auxiliary segmentation network. However, in the later stages of training, the prediction of the main segmentation network may be more accurate, so the self-refinement module will assign higher weights to the predicted labels generated by the main segmentation network. Through this dynamic adjustment mechanism, the self-refinement module is able to generate more accurate label prediction results for the second data set.

The calculation module uses the true label, the fused pseudo label and the predicted label generated by the main segmentation network to calculate the cross entropy loss value, and adjusts the network parameters of the main segmentation network according to the cross entropy loss value.

The present invention samples the reference labels of each three-dimensional shape in the first data set to generate corresponding three-dimensional shapes with sparse Scribble annotations to obtain a third data set; inputs the third data set into an auxiliary segmentation network module to train the auxiliary segmentation network; inputs the second data set into the trained auxiliary segmentation network module to output a three-dimensional shape containing complete annotations, namely a pseudo label, to obtain a fifth data set; trains the main segmentation network through the first data set and the fifth data set to output corresponding patch features for each facet of the three-dimensional shape and predict corresponding predicted labels; fuses the pseudo labels output by the auxiliary segmentation network module with the facet features output by the main segmentation network through a self-refinement module to obtain fused pseudo labels; calculates a cross entropy loss value using the real label, the fused pseudo label and the predicted label generated by the main segmentation network through a calculation module, and adjusts the network parameters of the main segmentation network according to the cross entropy loss value; that is, the present invention generates corresponding complete labels for the sparse annotations in the second data set through the trained auxiliary segmentation network, thereby expanding the training data used to train the main segmentation network and effectively avoiding the cost of expensive fully labeled training data.

As shown in FIG. 2 , an embodiment of the present invention further provides an optimization method based on a semi-supervised three-dimensional shape segmentation framework, comprising:

Get the target function;

Specifically, the objective function of the 3D shape segmentation framework is based on:

The objective function of the auxiliary segmentation network is trained, the objective function of the main segmentation network needs to be optimized when only the first data set is used, and the objective function of the three-dimensional shape segmentation framework is constructed when only the auxiliary segmentation network module and the main segmentation network module are used; wherein:

The objective function of training the auxiliary segmentation network is expressed as:

Wherein, F represents the first data set, s ^(f) represents the patch of the three-dimensional shape in the first data set; x ^(f) represents the feature vector of the patch s ^(f) , y ^(f) in the _paux function represents the predicted label of the auxiliary segmentation network for x ^(f) , θ represents the network parameters of the auxiliary segmentation network, and _paux represents the prediction function corresponding to the auxiliary segmentation network;

The formula of the objective function that the main segmentation network needs to optimize when only the first data set is used is:

In the formula, represents the network parameters of the main segmentation network, p _pri represents the prediction of the main segmentation network, x ^(f) represents the feature vector of the patch of the three-dimensional shape in the first data set F, and y ^(f) in the p _pri function represents the predicted label of the main segmentation network for x ^(f) ;

The objective function of the 3D shape segmentation framework using only the auxiliary segmentation network module and the main segmentation network module is expressed as:

Therefore, the objective function of the 3D shape segmentation framework is expressed as:

In the formula, and φ both represent the network parameters of the main segmentation network, λ represent the network parameters of the self-refinement module, F represents the first dataset, p _pri represents the prediction of the main segmentation network, x ^(f) represents the feature vector of the 3D shape patch in the first dataset F, y ^(f) represents the predicted label of x ^(f) by the main segmentation network, S represents the second dataset or the third dataset, q _conv represents the dynamic weight generated by the self-refinement module, x ^(s) represents the feature vector of the 3D shape patch in the second dataset or the third dataset, s ^(s) represents the 3D shape patch in the second dataset or the third dataset, y represents the prediction of x ^(s) by the auxiliary segmentation network, and s ^(f) represents the 3D shape patch in the first dataset.

The three-dimensional shape segmentation framework is optimized through the objective function; wherein the three-dimensional shape segmentation framework is obtained through the construction method of the three-dimensional shape segmentation framework based on semi-supervision described above.

As shown in FIG3 , the embodiment of the present invention further provides a semi-supervised 3D shape segmentation method, comprising:

Obtain an unlabeled three-dimensional shape to be segmented;

The unlabeled three-dimensional shape to be segmented is input into the main segmentation network for prediction to obtain a three-dimensional shape containing complete annotations; wherein the main segmentation network is trained by the construction method based on the semi-supervised three-dimensional shape segmentation framework described above.

An embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory in communication with the at least one processor. The memory stores a computer program executable by the at least one processor, and when the at least one processor executes the computer program, the electronic device executes the method of the embodiment of the present invention.

An embodiment of the present invention further provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiment of the present invention.

An embodiment of the present invention further provides a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to enable the computer to execute the method of the embodiment of the present invention.

With reference to Figure 4, the structural block diagram of the electronic device that can be used as the server or client of an embodiment of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described herein and/or required.

As shown in Figure 4, the electronic device includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In RAM 403, various programs and data required for the operation of the electronic device can also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to the bus 404.

Multiple components in the electronic device are connected to the I/O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 can be any type of device that can input information to the electronic device, and the input unit 406 can receive input digital or character information, and generate key signal input related to user settings and/or function control of the electronic device. The output unit 407 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video/audio output terminal, a vibrator, and/or a printer. The storage unit 408 can include but is not limited to a disk, an optical disk. The communication unit 409 allows the electronic device to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and/or a chipset, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and/or the like.

The computing unit 401 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and/or installed on an electronic device via ROM 402 and/or a communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the above-described method in any other appropriate manner (e.g., by means of firmware).

The computer programs for implementing the methods of the embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor or controller, the functions/operations specified in the flow chart and/or block diagram are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

In the context of an embodiment of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

The various steps described in the method implementation methods provided in the embodiments of the present invention may be performed in different orders and/or in parallel. In addition, the method implementation methods may include additional steps and/or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims (10)

1. The method for constructing the three-dimensional shape segmentation framework based on semi-supervision is characterized in that the three-dimensional shape segmentation framework comprises an auxiliary segmentation network module, a self-refinement module and a main segmentation network module comprising a main segmentation network and a calculation module; the auxiliary segmentation network module comprises an auxiliary segmentation network for predicting pseudo labels, a projection module for projecting a three-dimensional shape and a back projection module;

the construction method comprises the following steps:

Acquiring a first data set and a second data set; the data in the first data set is a completely marked three-dimensional shape; the data in the second data set are three-dimensional shapes with sparse Scribble labels;

sampling the reference labels of the three-dimensional shapes in the first data set to generate corresponding three-dimensional shapes with sparse scale labels, and obtaining a third data set;

inputting the third data set into an auxiliary segmentation network module to train an auxiliary segmentation network, wherein the projection module acquires multi-view projection images corresponding to three-dimensional shapes in the third data set to obtain a fourth data set, and establishes a reference matrix; the data in the reference matrix comprises vertex coordinates of each pixel on the original three-dimensional shape in the process of acquiring the multi-view projection image; the auxiliary segmentation network trains based on the first objective function and the multi-view projection images in the fourth dataset to predict prediction labels corresponding to two-dimensional images in the multi-view projection images; the back projection module maps the prediction tag back to the corresponding three-dimensional shape patch by utilizing the reference matrix so as to generate a corresponding complete label for each three-dimensional shape patch in the third data set;

Inputting the second data set into the trained auxiliary segmentation network module to output a three-dimensional shape containing complete labels, namely a pseudo tag, and obtaining a fifth data set;

Training a main segmentation network through the first data set and the fifth data set to output corresponding patch characteristics for each patch of the three-dimensional shape and predict corresponding prediction labels;

the pseudo tag output by the auxiliary segmentation network module is fused with the patch characteristic output by the main segmentation network through the self-thinning module, so that the fused pseudo tag is obtained;

And calculating a cross entropy loss value by using the real label, the fused pseudo label and a prediction label generated by the main segmentation network through a calculation module, and adjusting network parameters of the main segmentation network according to the cross entropy loss value.

2. The method for constructing a semi-supervised three dimensional shape segmentation framework according to claim 1, wherein the two-dimensional image of the multi-view projection image comprises a depth map and a rendering map;

The back projection module is specifically configured to find a patch with a corresponding coordinate in the three-dimensional shape according to the vertex coordinate recorded in the reference matrix (corresponding position) of each pixel, obtain a corresponding relationship between a prediction label in the two-dimensional image and a semantic label of each patch on the three-dimensional shape, and map the prediction label onto the corresponding patch with the three-dimensional shape based on the corresponding relationship.

3. A method of constructing a semi-supervised three dimensional shape segmentation framework as defined in claim 2, wherein the auxiliary segmentation network comprises:

encoder module for receiving the multi-view projection image and extracting the image characteristic information corresponding to each two-dimensional image in the multi-view projection image;

And the Decoder module is used for carrying out label prediction on the corresponding two-dimensional images according to the image characteristic information to obtain prediction labels corresponding to the two-dimensional images.

4. A method of constructing a semi-supervised based three dimensional shape segmentation framework as defined in claim 3, wherein the primary segmentation network comprises:

the main subdivision module consists of four full-connection layers and is used for extracting the surface patch characteristics corresponding to each surface patch in the three-dimensional shape;

the Softmax module is used for predicting a corresponding prediction tag according to the corresponding patch characteristics of each patch;

The computing module is specifically configured to:

For the first data set, a cross entropy loss value is calculated through a real label and a prediction label generated by the Softmax module, and for the fifth data set, a cross entropy loss value is calculated through a fused pseudo label and a prediction label generated by the Softmax module.

5. The method for constructing a three-dimensional shape segmentation framework based on semi-supervision according to claim 4, wherein the self-refinement module comprises two layers of convolution layers for dynamically learning the weight distribution of the main segmentation network and the auxiliary segmentation network to the prediction result of the three-dimensional shape with sparse scale labels in the prediction process.

6. The method for constructing a three-dimensional shape segmentation framework based on semi-supervision according to claim 5, wherein the formula expression of the first objective function is:

Wherein F represents a first dataset and S ^(f) represents a patch of three-dimensional shape in the first dataset; x ^(f) represents the feature vector of the S ^(f) patch, y ^(f) represents the prediction label of the auxiliary partition network to x ^(f), θ represents the network parameter of the auxiliary partition network, and p _aux represents the prediction function corresponding to the auxiliary partition network.

7. An optimization method of a three-dimensional shape segmentation framework based on semi-supervision is characterized by comprising the following steps:

Acquiring an objective function;

Optimizing the three-dimensional shape segmentation frame through an objective function; wherein the three-dimensional shape segmentation framework is obtained by the method of construction of a semi-supervised based three-dimensional shape segmentation framework as defined in any one of claims 1 to 6.

8. The method of optimizing a semi-supervised based three dimensional shape segmentation framework of claim 7, wherein the objective function has a formula:

In the method, in the process of the invention, And phi represents network parameters of the main segmentation network, lambda represents network parameters of the self-refinement module, F represents a first data set, p _pri represents prediction of the main segmentation network, x ^(f) represents a characteristic vector of a three-dimensional shape patch in the first data set F, y ^(f) represents a prediction label of the main segmentation network on x ^(f), S represents a second data set or a third data set, q _conv represents dynamic weight generated by the self-refinement module, x ^(s) represents a characteristic vector of a three-dimensional shape patch in the second data set or the third data set, S ^(s) represents a prediction of x ^(s) by the auxiliary segmentation network, and S ^(f) represents a three-dimensional shape patch in the first data set.

9. A semi-supervised three dimensional shape segmentation method, comprising:

Obtaining an unlabeled three-dimensional shape to be segmented;

Inputting the unlabeled three-dimensional shape to be segmented into a main segmentation network for prediction to obtain a three-dimensional shape containing complete labeling; wherein the main segmentation network is trained by the method for constructing a semi-supervised three dimensional shape segmentation framework of any of claims 1-6.

10. An electronic device, comprising: a processor, and a memory storing a program, characterized in that the program comprises instructions which, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 9.