CN116433903A - Instance segmentation model construction method, system, electronic equipment and storage medium - Google Patents
Instance segmentation model construction method, system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116433903A CN116433903A CN202310347568.XA CN202310347568A CN116433903A CN 116433903 A CN116433903 A CN 116433903A CN 202310347568 A CN202310347568 A CN 202310347568A CN 116433903 A CN116433903 A CN 116433903A
- Authority
- CN
- China
- Prior art keywords
- network
- depth estimation
- backbone network
- monocular
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000011218 segmentation Effects 0.000 title claims abstract description 113
- 238000010276 construction Methods 0.000 title claims abstract description 36
- 238000012549 training Methods 0.000 claims abstract description 46
- 238000000034 method Methods 0.000 claims abstract description 32
- 238000010606 normalization Methods 0.000 claims abstract description 21
- 238000007781 pre-processing Methods 0.000 claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 14
- 230000004927 fusion Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 11
- 238000005070 sampling Methods 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 4
- 238000001994 activation Methods 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000001514 detection method Methods 0.000 abstract description 20
- 230000008447 perception Effects 0.000 abstract description 5
- 238000005516 engineering process Methods 0.000 abstract description 3
- 230000002194 synthesizing effect Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 9
- 230000000694 effects Effects 0.000 description 8
- 238000012360 testing method Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000009499 grossing Methods 0.000 description 5
- 238000013459 approach Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000032683 aging Effects 0.000 description 2
- 230000003321 amplification Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000004438 eyesight Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- BASFCYQUMIYNBI-UHFFFAOYSA-N platinum Chemical compound [Pt] BASFCYQUMIYNBI-UHFFFAOYSA-N 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 239000012633 leachable Substances 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 229910052697 platinum Inorganic materials 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Multimedia (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an example segmentation model construction method, an example segmentation model construction system, electronic equipment and a storage medium, and belongs to the field of perception technology of automatic driving image processing. The method comprises the following steps: training a monocular depth estimation model by adopting a training framework of a self-supervision monocular depth estimation network; extracting a ResNeXt50 backbone network based on FPN characteristics of a monocular depth estimation model, and carrying out group normalization and weight normalization processing on the backbone network to obtain a preprocessed backbone network; adding the feature pyramid network into a monocular depth estimation model; constructing a depth information guide module based on the improved spatial attention; loading the information between the characteristic pyramid network and the preprocessing backbone network through a depth information guiding module; and correcting the region of interest according to the pyramid characteristics and synthesizing the fixed region so as to construct the target instance segmentation model. The method and the device can effectively solve the problems of wrong segmentation or failure caused by low accuracy and unclear segmentation boundary of the detection and segmentation of the long-distance target and/or the large-size target.
Description
Technical Field
The invention belongs to the technical field of perception of automatic driving image processing, and particularly relates to an example segmentation model construction method, an example segmentation model construction system, electronic equipment and a storage medium.
Background
Visual perception systems for automatically driving automobiles involve a number of visual tasks including object detection, classification, segmentation, localization, object tracking, depth estimation, and the like. Target motion prediction and gesture estimation realized based on target detection results are the precondition of realizing functions such as collision avoidance control, motion track planning and the like of an automatic driving automobile; the object detection and the instance segmentation in the road scene mainly judge the object category attribute and accurately position the object position. General road scenes can be mainly divided into two categories, namely, a taking category containing specific countable objects such as pedestrians, vehicles of different types and the like, and a stuff category containing texture similar areas and difficult countable areas. The example segmentation can be regarded as the expansion of target detection, mainly aims at the taking object, and realizes the pixel-by-pixel segmentation of different types of targets with different numbers on the basis of detecting targets. Unlike semantic segmentation, the objects of the same class in the instance segmentation need to be treated differently, and the number of objects of the same class is variable, so that the instance segmentation problem cannot be solved simply through a pixel-by-pixel classification concept similar to the semantic segmentation, such as the perception task comparison of different scenes shown in fig. 1, fig. 1 (a) original image, fig. 1 (b) image processed by semantic segmentation, fig. 1 (c) image processed by object detection, and fig. 1 (d) image processed by instance segmentation.
Taking the common road scene of automatic driving as an example, the case segmentation of objects such as densely distributed long-distance and/or large-size crowd, vehicles and the like sometimes has unclear segmentation boundaries, even segmentation errors and failures due to the fact that local textures of images are close or occur, as in the case of case segmentation aging shown in fig. 2, the original image and the local amplification thereof are shown in fig. 2 (a), and the case segmentation result and the local amplification thereof are shown in fig. 2 (b). Compared with the method for learning the positions and the sizes of the mask and the detection frame of the region where the target is located from single two-dimensional image information, along with the depth characteristics introduced by the image processing based on the deep learning, the positions of different objects in the three-dimensional scene are distinguishable, and the regions with similar textures and larger depth value difference are not necessarily the same object. In particular, depth estimation information of scenes such as vehicle queues, pedestrians and the like with shielding relation in a road environment can further guide and assist an instance segmentation task. However, few studies in the prior art focus on the feasibility and effectiveness of depth estimation information for example segmentation model performance improvement.
Therefore, how to apply depth estimation information guidance to perform segmentation to overcome the problems of detection of a long-distance target and/or a large-size target, segmentation errors or failures caused by low accuracy and unclear segmentation boundaries is a problem to be solved by those skilled in the art.
Disclosure of Invention
In order to solve the technical problems, the invention provides an example segmentation model construction method, an example segmentation model construction system, electronic equipment and a storage medium.
In a first aspect, the invention provides a method for constructing an instance segmentation model, which comprises the following steps:
training a monocular depth estimation model by adopting a training framework of a self-supervision monocular depth estimation network;
extracting a ResNeXt50 backbone network based on FPN characteristics of the monocular depth estimation model, and performing group normalization and weight normalization processing on the backbone network to obtain a preprocessing backbone network;
adding a feature pyramid network into the monocular depth estimation model;
constructing a depth information guide module based on the improved spatial attention;
loading the depth information between the feature pyramid network and the preprocessing backbone network through the depth information guiding module so as to enable the depth information extracted by the preprocessing backbone network to be reflected into pyramid features with different scales along an up-sampling and fusion channel of the feature pyramid network;
and correcting the interested region and the synthesized fixed region according to the pyramid characteristics with the depth information, and combining the monocular depth estimation model to construct a target instance segmentation model.
Preferably, the self-supervision monocular depth estimation network comprises a monocular depth estimation network and a pose estimation network, wherein a backbone network of the monocular depth estimation network adopts a standard ResNet-50 network.
Preferably, the monocular depth estimation network generates a multi-scale relative depth feature map through a depth estimation prediction head in stages 4, 3, 2 and 1 respectively, wherein the relative depth ranges from 0 to 1.
Preferably, the decoder of the standard ResNet-50 network comprises a fused convolution of a series of upsampling and fused skip connections; the pose estimation network carries out a plurality of convolutions and activations on the feature map output by the last layer of the encoder of the standard ResNet-50 network so as to predict transformation matrix parameters between two frames of feature maps.
Preferably, the feature pyramid network outputs F at four layers layer1, layer2, layer3, layer4 of the ResNeXt50 backbone network 1 、F 2 、F 3 、F 4 Based on the channel conversion, up-sampling fusion generates the channel number C out And adding an output P based on the last layer output of the ResNeXt50 backbone network 5 。
Preferably, the depth information guiding module is used for guiding the characteristics from the ResNeXt50 backbone network
And features from the monocular depth estimation network ∈>
And performing linear transformation to obtain respective subspace representations.
Preferably, a learnable weight parameter α is added to the depth information guiding module, and α is initialized to 0 when the self-supervision monocular depth estimation network training starts, and the depth information guiding module is equivalent to the transverse convolution of the ResNeXt50 backbone network.
In a second aspect, the invention provides an example segmentation model construction system, comprising:
the training module is used for training the monocular depth estimation model by adopting a training framework of the self-supervision monocular depth estimation network;
the preprocessing module is used for extracting a ResNeXt50 backbone network based on the FPN characteristics of the monocular depth estimation model, and carrying out group normalization and weight normalization processing on the backbone network to obtain a preprocessed backbone network;
the adding module is used for adding the feature pyramid network into the monocular depth estimation model;
a construction module for constructing a depth information guide module based on the improved spatial attention;
the fusion module is used for loading the depth information between the characteristic pyramid network and the preprocessing backbone network through the depth information guiding module so as to enable the depth information extracted by the preprocessing backbone network to be reflected into pyramid characteristics with different scales along an up-sampling and fusion channel of the characteristic pyramid network;
and the construction module is used for correcting the region of interest and the synthesized fixed region according to the pyramid characteristics with the depth information and combining the monocular depth estimation model so as to construct a target instance segmentation model.
In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the method for building an instance segmentation model according to the first aspect when executing the computer program.
In a fourth aspect, embodiments of the present application provide a storage medium having stored thereon a computer program which, when executed by a processor, implements the instance segmentation model construction method as set forth in the first aspect.
Compared with the prior art, the method, the system, the electronic equipment and the storage medium for constructing the example segmentation model have the following beneficial effects:
1. the method comprises the steps of adding a multi-scale spatial attention module at an input end of feature pyramid generation of an instance segmentation backbone network, selectively weighting original instance segmentation features by means of depth information by calculating similarity relation between the depth features and features extracted by the instance segmentation backbone network, so that scene structure information is highlighted in pyramid features extracted by the instance segmentation backbone network, and more assistance is provided for instance segmentation large object detection, remote target detection and segmentation.
2. Compared with the existing model, the built instance segmentation model remarkably improves the detection and segmentation accuracy of a long-distance target and/or a large-size target by utilizing visual depth information, and simultaneously improves the accuracy of the instance segmentation boundary of a low-texture region; better support is provided for accurate perception of remote target tracking and target position in an automatic driving application scene.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a diagram of a perceived task comparison effect for different scenarios in the prior art;
FIG. 2 is a schematic diagram of an example split aging case of the prior art;
FIG. 3 is a flowchart of an example segmentation model construction method provided in embodiment 1 of the present invention;
FIG. 4 is a diagram of a self-monitoring monocular depth estimation network training framework provided in embodiment 1 of the present invention;
fig. 5 is a detail structure diagram of a ResNeXt50 backbone network provided in embodiment 1 of the present invention;
fig. 6 is a schematic diagram of a DGM module according to embodiment 1 of the present invention;
fig. 7 is a diagram of a SA-FPN backbone network according to embodiment 1 of the present invention;
FIG. 8 is a diagram of the overall architecture of the object example segmentation model provided in embodiment 1 of the present invention;
fig. 9 is a graph showing the lifting effect of depth information on an example segmentation boundary in experimental verification provided in embodiment 1 of the present invention;
FIG. 10 is a block diagram of an example segmentation model construction system corresponding to the method of embodiment 1 provided in embodiment 2 of the present invention;
fig. 11 is a schematic hardware structure of an electronic device according to embodiment 3 of the present invention.
Reference numerals illustrate:
10-a training module;
20-a pretreatment module;
30-adding a module;
40-constructing a module;
a 50-fusion module;
60-building a module;
70-bus, 71-processor, 72-memory, 73-communication interface.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the disclosed aspects may be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the disclosure.
The block diagrams depicted in the figures are merely functional entities and do not necessarily correspond to physically separate entities. I.e. the functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
The flow diagrams depicted in the figures are exemplary only, and do not necessarily include all of the elements and operations/steps, nor must they be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.
Taking the common road scene of automatic driving as an example, the example segmentation of objects such as densely distributed crowd, vehicles and the like sometimes has unclear segmentation boundaries, even segmentation errors and failures due to the approach of local textures of images. In response to the above difficulties, there are currently methods that combine depth estimation with instance segmentation.
Among them, the current mainstream example segmentation can be mainly divided into two methods, namely bottom-up and top-down. The top-down method follows the idea of detecting first and then dividing, namely, the boundary box of the object is obtained through target detection first, and the in-box area is divided through a division network to obtain an example division mask. The bottom-up method is to acquire the instance segmentation mask by using a method of segmentation-first clustering-second clustering, which often needs an operation of grouping pixels to generate an instance segmentation result, and the performance of the method is weaker than that of the common instance segmentation.
The monocular depth estimation is one of the basic problems of computer vision, and the task of monocular depth estimation can be mainly divided into a supervised mode and an unsupervised mode at present. The supervised monocular depth estimation learns the mapping relation of the two-dimensional image to the relative depth value in an end-to-end mode, is limited by the natural deficiency of 2D to 3D mapping, and recent methods attempt to improve the performance of the supervised monocular depth by adding geometric information prior. Unsupervised monocular depth estimation is one of the most successful examples of unsupervised learning at present. Under the unsupervised framework, the 3D packing-unpacking operation is used for replacing the pooling (downsampling) and upsampling operation, so that more image detail information representation is reserved in the encoding-decoding process, and the effect of unsupervised monocular depth estimation is further improved.
In order to better utilize the relevance among the multitasking and information assistance, many scene understanding deep learning methods integrate multiple subtask modules such as semantic segmentation, depth estimation and the like. However, for example segmentation of objects such as densely distributed long-distance and/or large-size crowd, vehicles and the like of road scenes, the existing method for combining depth estimation and example segmentation has the problems of insufficient ability of accurately detecting the objects and predicting fine-grained example masks and difficulty in capturing scene structures and contexts. The present application is presented based on this.
Example 1
Specifically, fig. 3 is a schematic flow chart of an example segmentation model construction method according to the present embodiment.
As shown in fig. 3, the example segmentation model construction method of the present embodiment includes the steps of:
s101, training a monocular depth estimation model by adopting a training framework of a self-supervision monocular depth estimation network.
The self-supervision monocular depth estimation network comprises a monocular depth estimation network and a pose estimation network. Preferably, the backbone network of the monocular depth estimation network employs a standard ResNet-50 network, and the corresponding decoder mainly comprises a series of upsampling (UpConv i ) And fusion jumping connection (SkiFusion convolution of p Connection (FConv) i ). The decoder of the monocular depth estimation network generates multi-scale relative depth maps through the depth estimation pre-header (Depth Prediction Head) at stages 4, 3, 2, 1, respectively, with relative depths ranging from [0,1 ]]The final activation function for each pre-header is thus Sigmoid.
Further, the pose estimation network needs to regress parameters of a pose transformation matrix with high abstraction, and a plurality of convolutions and activation operations are directly carried out on the last layer of output feature diagram of the standard ResNet-50 network to reduce the number of feature channels, so that transformation matrix parameters between two frames are finally predicted. It should be noted that the input of the pose estimation network is the result of combining two image channels, and when the res net pre-training network is used, the problem that the sizes of the first layer convolution parameters are not matched occurs. The convolution kernel size of the ResNet first layer is Cm×Cout×K×K (K is the convolution kernel size), and when two pictures are combined and input, C is obtained in Doubling is required, here the idea of copying parameters proposed by Godard et al, i.e. according to C in The dimension replicates the parameter by a factor of 2 and then divides the weight of the entire convolution kernel by 2.
S102, extracting a ResNeXt50 backbone network based on FPN characteristics of the monocular depth estimation model, and performing group normalization and weight normalization processing on the backbone network to obtain a preprocessing backbone network.
The data size of the adopted data set Cityscapes is larger, the picture width and height are 2048 and 1024 respectively, and after a data enhancement method such as random clipping is adopted, the clipping size cannot be too small so as not to lose excessive picture global information, so that when distributed training is adopted, the number of pictures which can be processed once on each GPU is small. While the Batch normalization (Batch Normalization, BN) employed in the original ResNeXt model works well in many computer vision applications, it has very limited effectiveness for small Batch (Mini Batch) training, especially where only 1-2 images per GPU are used for training. This is mainly due to the fact that when training batches get smaller, it is difficult to make an accurate estimate of the distribution based on a small number of samples that follow a particular distribution, and the effect of efficient batch normalization is lost. Therefore, the embodiment adopts group normalization processing, wherein the group normalization is to group the characteristic channel dimension (C) on the basis of original batch normalization, the normalization operation is carried out in the group, all batches are not normalized on each channel, the operation is not dependent on batch dimension (N) any more, and the method is helpful for small batch training.
Further, self-supervised depth estimation requires constraining training with a priori features of the depth map and reconstructed similarities. The self-supervision monocular depth estimation framework adopts two Loss function constraints, namely luminosity Loss (Photometric Loss) and depth smoothing Loss (smoothloss), and particularly as shown in fig. 4, the self-supervision monocular depth estimation framework adopts the following formula to characterize: l=0.15l Smooth +L Photometric The method comprises the steps of carrying out a first treatment on the surface of the Wherein, luminosity loss L Smooth By combining the luminance constancy and spatial smoothness prior knowledge used in classical dense matching algorithms, the photometric difference between the resampled reference frame and the real target frame is used as part of an unsupervised loss function. Depth smoothing loss L Photometric Is a depth continuity constraint on non-boundary regions. The smoothness constraint utilizes the continuity feature priori of the depth distribution of the non-boundary region to constrain the depth estimation result.
And S103, adding a feature pyramid network into the monocular depth estimation model.
Wherein the characteristic pyramid network outputs F at four layers layer1, layer2, layer3 and layer4 of the ResNeXt50 backbone network 1 、F 2 、F 3 、F 4 Based on the channel conversion, up-sampling fusion generates the channel number C out And adding an output P based on the last layer output of the ResNeXt50 backbone network 5 The method comprises the steps of carrying out a first treatment on the surface of the As shown in particular in fig. 5.
S104, constructing a depth information guiding module based on the improved spatial attention.
Wherein the depth information guiding module is used for guiding the characteristics from the ResNeXt50 backbone network
And features from the monocular depth estimation network ∈>
And performing linear transformation to obtain respective subspace representations, and simultaneously, reducing the dimension of the channel, so as to avoid excessive calculation amount consumption in calculating the similarity matrix. Specifically, in the depth guidance module of the present embodiment, in order to enable the example split backbone network to sufficiently capture the long-range relationship between the depth feature and the example split feature during training, the depth information guidance module DGM is designed by improving the spatial attention (Spatial Attention) module, and the specific structure is shown in fig. 6. In addition, a leachable weight parameter alpha is added in the depth information guiding module, and alpha is initialized to 0 when the self-supervision monocular depth estimation network training starts, and the depth information guiding module is equivalent to the transverse convolution of the ResNeXt50 backbone network. The addition of the learnable weight parameter alpha gives the DGM module a certain self-adaptability, especially at the beginning of training; on the other hand, the automatic parameter optimization search can be performed without manually adjusting network parameters.
It should be noted that the Dual attention variant of the self-attention mechanism shows excellent performance in semantic segmentation applications, and by introducing spatial attention modules and channel attention modules, the network can encode more extensive context information as local features, thereby enhancing its representation capability. In addition to the cross-task spatial attention depth information guiding module, a backbone network is proposed that improves the instance segmentation performance by using spatial self-attention: spatial self-attention FPN (SA-FPN) is shown in particular in fig. 7. Considering that the attention mechanism has higher demand on computing resources and the resolution of the picture data itself for instance segmentation on the Cityscapes is higher, adding the self-attention module on a larger scale can further compress the training batch, and the hardware demand cannot be satisfied. Meanwhile, the high-level semantic information contained in the feature map output at the last layer of the backbone network is more abundant, and the self-attention module is added on the basis, so that the relevance between the high-level semantic information can be better mined, and the method is helpful for improving the performance of target detection and instance segmentation.
S105, loading depth information extracted by the preprocessing backbone network between the feature pyramid network and the preprocessing backbone network through the depth information guiding module so as to enable the depth information extracted by the preprocessing backbone network to be reflected into pyramid features with different scales along an up-sampling and fusion channel of the feature pyramid network.
S106, correcting the interested region and the synthesized fixed region according to the pyramid features with the depth information, and combining the monocular depth estimation model to construct a target instance segmentation model.
The example segmentation model mainly comprises two parts, namely RPN (Region Proposal Network) and RoISH (RoISegmentation Head). Wherein the RPN mainly uses the concept of an "Anchor" (Anchor) proposed in the fast R-CNN, which is a certain number of predefined bounding boxes generated at each location on the pyramid feature, the RPN network achieves a correction of the predefined "Anchor" by predicting the offset of the predefined bounding box to the real bounding box and the likelihood of a detection target within the box, giving a final region of interest proposal (Rol RegionProposal). The RoISH is a region with a fixed size which is segmented in the corresponding pyramid feature of the FPN backbone network according to the central position and the wide and high prediction results of all the regions of interest obtained in the first stage, so as to better process and unify the target detection and segmentation of different scales. Then, the classification prediction of the target and the segmentation result prediction of the target are respectively carried out on the basis of the proposal of the region of interest with the uniform size. Specifically, the overall architecture of the target instance segmentation model of the present embodiment is shown in fig. 8, in which DGM (Depth Guide Module) is a depth information guiding module, DGFPN (Depth GuidedFeature Pyramid Network) is a Feature Pyramid (FPN) network based on depth information guiding, and depth features of different scales led out by the depth estimation network are respectively led into the instance segmentation backbone network FPN of corresponding scales via the DGM module.
The following experimental verification was performed with respect to the example segmentation model constructed in the above steps.
1.1, data set and evaluation index
And selecting the automatic driving scene perception data sets Cityscapes and Kitti as experimental data. The Cityscapes dataset provides true annotations including semantic segmentation, instance segmentation, etc., and the dataset contains 5000 finely annotated Cityscapes fine and 20000 coarsely annotated Cityscapes coarse. The Kitti dataset contains semantic segmentation labels on the three-dimensional point cloud, depth estimation labels synthesized by the laser radar data, and a small number of optical flow estimation labels. All experiments adopted richer image enhancement during training, and the method comprises random clipping, random overturning, image luminosity adjustment, chromaticity increment and the like according to a IoU minimum threshold value.
In the example segmentation task, the Cityscapes data set is calculated by using a segmentation mask region Mpred and a corresponding label Mgt as an evaluation index; specific evaluation indexes are shown in Table 1.
TABLE 1 example segmentation and target detection index specification
1.2 experimental parameters
All experiments referred to herein were based on Python 3.8 implementation, version 1.8.2 for the deep learning tool pythoch. The experimental hardware platform is Ubuntu18.04 LTS server provided with 8 RTX2070 Ti graphic cards, and the model of the server CPU is Intel (R) Xeon (R) Platinum 7163CPU@2.50GHz.
The self-supervision learning training super-parameter setting of the monocular depth estimation network mainly refers to Monodepth2. The optimizer adopts Adam, the learning rate is set to be 0.0001, the training Batch Size (Batch Size) is 12, the training time length is 20, the step length of the adopted step-type attenuation learning rate planner (StepLR Scheduler) is set to be 15, and the attenuation factor y=0.1; the smoothing loss weight in the total depth estimation loss function is set to α=0.001.
The depth information guided example segmentation network employs an AdamW optimizer in which the weight decay coefficients are empirically set to 5 x 10 -4 . Meanwhile, a step-type attenuation learning rate planner (StepLR Scheduler) is adopted in training, the attenuation factor y=0.2, and the attenuation step sizes are set to be [16, 34, 40, 43 ] according to experience]。
1.3 analysis of Experimental results
Depth estimation pre-training model depth estimation effects were tested on the Kitti eigen test set after training using monocular Kitti data, as shown in table 2, the pre-training model was comparable to the classical model Monodepth2 in terms of self-supervised depth estimation with similar training.
TABLE 2 Kittieigen test set self-supervised monocular depth estimation contrast
For the instance segmentation model, after training on the Cityscapes fine training set is completed, the performance of the proposed Multi-scale depth information guided instance segmentation model Multi-DG-CMR (Multi-scaleDepth-guided Cascade Mask R-CNN) and the corresponding classical instance segmentation baseline model CMR (cascademask R-CNN) is first tested on the Cityscapes fine test test set, see table 3.
TABLE 3 comparison of the effects of Cityscapes Fine test set instance segmentation
Table 3 contains the baseline model CMR and the proposed depth information guided example segmentation model Multi-DG-CMR, and also compares the effect of the classical example segmentation model MR (Mask R-CNN) based on the "top-down" approach when there is pre-training of the COCO dataset, and the effect of the test set based on the ERFNet example segmentation model based on the "bottom-up" approach. From table 3, it can be seen that the depth guidance mechanism proposed herein improves various indexes including mAP and mAP100m in the example segmentation task, and also improves the classical Mask R-CNN model pre-trained by the COCO dataset to a certain extent.
It is particularly pointed out that compared with the detection precision of the Multi-DG-CMR in the range of 100m and the detection precision of the Multi-DG-CMR in the range of 50m, the detection precision of the remote target is improved after the depth estimation guide module (DGM) is introduced. This is mainly understood to be that, in the improved spatial attention module in the DGM, the relative depth information in the depth features gives a greater weight to the remote target, so as to improve the attention of the remote target in the network training process.
In addition to the average precision index, the precision of the above model over each instance segmentation class is also compared herein, as in table 4.
TABLE 4 comparison of average accuracy of segmentation (Mask mAP) for each class of examples of the Cityscape fine test set
It can be seen from Table 4 that the proposed Multi-DG-CMR example segmentation model provides a greater boost over CMR in the larger size categories of cars (Car), trucks (Truck), buses (Bus), etc., and performs quite well in the other categories. The overall average improvement of the models in table 3 above comes primarily from the improvement in detection accuracy for these classes of targets.
One major constraint of the self-supervised training of the depth estimation network is the smoothing loss of the original gradient attention, which only emphasizes the smoothing that constrains the non-boundary region, while the boundary region can only rely on another loss function constraint. It is also true that most depth estimation result detail information cannot be as fine as the original RGB image, and that the boundary information for large-sized objects remains more, and that the assistance for instance segmentation is more pronounced.
As shown in fig. 9, in the CMR method, since the top area of the car is close to the background, a boundary error is generated, and after depth information is introduced, the boundary information of the depth estimation information in the top area of the car gives a certain constraint to the example segmentation network, so that the result of the Multi-DG-CMR method completely segments the top of the car.
Example 2
This embodiment provides a block diagram of a system corresponding to the method described in embodiment 1. FIG. 2 is a block diagram of an example segmentation model construction system, as shown in FIG. 2, according to an embodiment of the present application, including:
a training module 10 for training a monocular depth estimation model using a training framework of a self-supervising monocular depth estimation network;
the preprocessing module 20 is configured to extract a ResNeXt50 backbone network based on FPN features of the monocular depth estimation model, and perform group normalization and weight normalization processing on the backbone network to obtain a preprocessed backbone network;
an adding module 30 for adding a feature pyramid network to the monocular depth estimation model;
a construction module 40 for constructing a depth information guide module based on the improved spatial attention;
the fusion module 50 is configured to load depth information extracted by the preprocessing backbone network between the feature pyramid network and the preprocessing backbone network through the depth information guiding module, so that depth information extracted by the preprocessing backbone network is reflected into pyramid features with different scales along an upsampling and fusion channel of the feature pyramid network;
a construction module 60, configured to modify a region of interest and a synthesized fixed region according to the pyramid feature with the depth information, and combine the monocular depth estimation model, so as to construct a target instance segmentation model.
The above-described respective modules may be functional modules or program modules, and may be implemented by software or hardware. For modules implemented in hardware, the various modules described above may be located in the same processor; or the above modules may be located in different processors in any combination.
Example 3
The example segmentation model construction method described in connection with fig. 3 may be implemented by an electronic device. Fig. 11 is a schematic diagram of the hardware structure of the electronic device according to the present embodiment.
The electronic device may include a processor 71 and a memory 72 storing computer program instructions.
In particular, the processor 71 may include a Central Processing Unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, abbreviated as ASIC), or may be configured to implement one or more integrated circuits of embodiments of the present application.
The processor 71 implements the example division model construction method of embodiment 1 described above by reading and executing the computer program instructions stored in the memory 72.
In some of these embodiments, the electronic device may also include a communication interface 73 and a bus 70. As shown in fig. 11, the processor 71, the memory 72, and the communication interface 73 are connected to each other via the bus 70 and perform communication with each other.
The communication interface 73 is used to enable communication between various modules, devices, units and/or units in embodiments of the application. Communication interface 73 may also enable communication with other components such as: and the external equipment, the image/data acquisition equipment, the database, the external storage, the image/data processing workstation and the like are used for data communication.
The electronic device may execute the example segmentation model construction method of embodiment 1 of the present application based on the obtained example segmentation model construction identification system.
In addition, in combination with the example segmentation model construction method of the above embodiment 1, the embodiment of the present application may provide a storage medium for implementation. The storage medium having stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement the example segmentation model construction method of embodiment 1 described above.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.
Claims (10)
1. An instance segmentation model construction method is characterized by comprising the following steps:
training a monocular depth estimation model by adopting a training framework of a self-supervision monocular depth estimation network;
extracting a ResNeXt50 backbone network based on FPN characteristics of the monocular depth estimation model, and performing group normalization and weight normalization processing on the backbone network to obtain a preprocessing backbone network;
adding a feature pyramid network into the monocular depth estimation model;
constructing a depth information guide module based on the improved spatial attention;
loading the depth information between the feature pyramid network and the preprocessing backbone network through the depth information guiding module so as to enable the depth information extracted by the preprocessing backbone network to be reflected into pyramid features with different scales along an up-sampling and fusion channel of the feature pyramid network;
and correcting the interested region and the synthesized fixed region according to the pyramid characteristics with the depth information, and combining the monocular depth estimation model to construct a target instance segmentation model.
2. The instance segmentation model construction method according to claim 1, wherein the self-supervising monocular depth estimation network comprises a monocular depth estimation network and a pose estimation network, wherein a backbone network of the monocular depth estimation network employs a standard res net-50 network.
3. The instance segmentation model construction method according to claim 2, wherein the monocular depth estimation network generates a multi-scale relative depth feature map through depth estimation prediction heads at stages 4, 3, 2, 1, respectively, wherein the relative depth ranges from 0 to 1.
4. The instance segmentation model construction method according to claim 2, wherein the decoder of the standard res net-50 network comprises a fused convolution of a series up-sampling and a fused skip connection; the pose estimation network carries out a plurality of convolutions and activations on the feature map output by the last layer of the encoder of the standard ResNet-50 network so as to predict transformation matrix parameters between two frames of feature maps.
5. The instance division model construction method according to claim 1, wherein the feature pyramid network outputs F at four layers layer1, layer2, layer3, layer4 of the ResNeXt50 backbone network 1 、F 2 、F 3 、F 4 Based on the channel conversion, up-sampling fusion generates the channel number C out And adding an output P based on the last layer output of the ResNeXt50 backbone network 5 。
6. The instance segmentation model construction method according to claim 1, wherein the depth information guide module is configured to guide features from the ResNeXt50 backbone network
And features from the monocular depth estimation network ∈>
And performing linear transformation to obtain respective subspace representations.
7. The method of claim 1, wherein a learnable weight parameter α is added to the depth information guiding module, and α is initialized to 0 at the beginning of the self-supervised monocular depth estimation network training, the depth information guiding module being equivalent to a lateral convolution of the ResNeXt50 backbone network.
8. An instance segmentation model construction system, comprising:
the training module is used for training the monocular depth estimation model by adopting a training framework of the self-supervision monocular depth estimation network;
the preprocessing module is used for extracting a ResNeXt50 backbone network based on the FPN characteristics of the monocular depth estimation model, and carrying out group normalization and weight normalization processing on the backbone network to obtain a preprocessed backbone network;
the adding module is used for adding the feature pyramid network into the monocular depth estimation model;
a construction module for constructing a depth information guide module based on the improved spatial attention;
the fusion module is used for loading the depth information between the characteristic pyramid network and the preprocessing backbone network through the depth information guiding module so as to enable the depth information extracted by the preprocessing backbone network to be reflected into pyramid characteristics with different scales along an up-sampling and fusion channel of the characteristic pyramid network;
and the construction module is used for correcting the region of interest and the synthesized fixed region according to the pyramid characteristics with the depth information and combining the monocular depth estimation model so as to construct a target instance segmentation model.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the instance segmentation model construction method of any one of claims 1-7 when the computer program is executed.
10. A storage medium having stored thereon a computer program, which when executed by a processor implements the instance segmentation model construction method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310347568.XA CN116433903A (en) | 2023-04-03 | 2023-04-03 | Instance segmentation model construction method, system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310347568.XA CN116433903A (en) | 2023-04-03 | 2023-04-03 | Instance segmentation model construction method, system, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116433903A true CN116433903A (en) | 2023-07-14 |
Family
ID=87092017
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310347568.XA Pending CN116433903A (en) | 2023-04-03 | 2023-04-03 | Instance segmentation model construction method, system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116433903A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117292120A (en) * | 2023-11-27 | 2023-12-26 | 南昌工程学院 | Light-weight visible light insulator target detection method and system |
| CN117314932A (en) * | 2023-09-12 | 2023-12-29 | 四川大学华西第四医院(四川大学华西职业病医院) | Token pyramid-based pancreatic bile duct segmentation method, model and storage medium |
-
2023
- 2023-04-03 CN CN202310347568.XA patent/CN116433903A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117314932A (en) * | 2023-09-12 | 2023-12-29 | 四川大学华西第四医院(四川大学华西职业病医院) | Token pyramid-based pancreatic bile duct segmentation method, model and storage medium |
| CN117314932B (en) * | 2023-09-12 | 2024-06-07 | 四川大学华西第四医院(四川大学华西职业病医院) | Token pyramid-based pancreatic bile duct segmentation method, model and storage medium |
| CN117292120A (en) * | 2023-11-27 | 2023-12-26 | 南昌工程学院 | Light-weight visible light insulator target detection method and system |
| CN117292120B (en) * | 2023-11-27 | 2024-02-09 | 南昌工程学院 | Light-weight visible light insulator target detection method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200250436A1 (en) | Video object segmentation by reference-guided mask propagation | |
| US10489913B2 (en) | Methods and apparatuses, and computing devices for segmenting object | |
| Žbontar et al. | Stereo matching by training a convolutional neural network to compare image patches | |
| US20190042888A1 (en) | Training method, training apparatus, region classifier, and non-transitory computer readable medium | |
| EP3211596A1 (en) | Generating a virtual world to assess real-world video analysis performance | |
| CN111126359B (en) | High-definition image small target detection method based on self-encoder and YOLO algorithm | |
| US11687773B2 (en) | Learning method and recording medium | |
| CN116433903A (en) | Instance segmentation model construction method, system, electronic equipment and storage medium | |
| CN111914698B (en) | Human body segmentation method, segmentation system, electronic equipment and storage medium in image | |
| CN114022799A (en) | Self-supervision monocular depth estimation method and device | |
| CN111461212A (en) | Compression method for point cloud target detection model | |
| CN110879960B (en) | Method and computing device for generating image data set for convolutional neural network learning | |
| CN110781850A (en) | Semantic segmentation system and method for road recognition, and computer storage medium | |
| CN111768415A (en) | Image instance segmentation method without quantization pooling | |
| CN111553414A (en) | In-vehicle lost object detection method based on improved Faster R-CNN | |
| US20230281974A1 (en) | Method and system for adaptation of a trained object detection model to account for domain shift | |
| CN112613387A (en) | Traffic sign detection method based on YOLOv3 | |
| CN112183649A (en) | Algorithm for predicting pyramid feature map | |
| CN112396657A (en) | Neural network-based depth pose estimation method and device and terminal equipment | |
| CN114359361A (en) | Depth estimation method, depth estimation device, electronic equipment and computer-readable storage medium | |
| Wang et al. | Cross-domain learning using optimized pseudo labels: toward adaptive car detection in different weather conditions and urban cities | |
| US20230298335A1 (en) | Computer-implemented method, data processing apparatus and computer program for object detection | |
| CN111709377A (en) | Feature extraction method, target re-identification method and device and electronic equipment | |
| CN111738069A (en) | Face detection method and device, electronic equipment and storage medium | |
| CN111079634A (en) | Method, device and system for detecting obstacle in vehicle running and vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |