CN111242954B - Panorama segmentation method with bidirectional connection and shielding processing - Google Patents

Panorama segmentation method with bidirectional connection and shielding processing Download PDF

Info

Publication number
CN111242954B
CN111242954B CN202010067124.7A CN202010067124A CN111242954B CN 111242954 B CN111242954 B CN 111242954B CN 202010067124 A CN202010067124 A CN 202010067124A CN 111242954 B CN111242954 B CN 111242954B
Authority
CN
China
Prior art keywords
segmentation
semantic
features
feature
instance
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010067124.7A
Other languages
Chinese (zh)
Other versions
CN111242954A (en
Inventor
李玺
陈怡峰
蔺广琛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University ZJU
Original Assignee
Zhejiang University ZJU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University ZJU filed Critical Zhejiang University ZJU
Priority to CN202010067124.7A priority Critical patent/CN111242954B/en
Publication of CN111242954A publication Critical patent/CN111242954A/en
Application granted granted Critical
Publication of CN111242954B publication Critical patent/CN111242954B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/254—Fusion techniques of classification results, e.g. of results related to same input data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Abstract

The invention discloses a panorama segmentation method with bidirectional connection and shielding processing. The method comprises the following steps: 1) acquiring a data set for training panoramic segmentation, and defining an algorithm target; 2) carrying out feature learning on the images in the group by using a full convolution network; 3) extracting semantic features from the feature map through a semantic feature extraction branch; 4) extracting example features from the feature map through an example feature extraction branch; 5) establishing connection from instance segmentation to semantic segmentation, and aggregating semantic features and instance features to perform semantic segmentation; 6) establishing connection from semantic segmentation to instance segmentation, and aggregating instance features and semantic features to perform instance segmentation; 7) and fusing the results of semantic segmentation and instance segmentation by using an occlusion processing algorithm, and outputting a panoramic segmentation result. The method fully utilizes the complementarity between semantic segmentation and example segmentation, and simultaneously applies the occlusion processing algorithm proposed by the bottom layer characteristic appearance information to efficiently complete the panoramic segmentation of the image.

Description

Panorama segmentation method with bidirectional connection and shielding processing
Technical Field
The invention belongs to the field of computer vision, and particularly relates to a panoramic segmentation method with bidirectional connection and shielding processing.
Background
The panorama segmentation task is a set of semantic segmentation tasks and instance segmentation tasks that require not only prediction of semantic classes at the pixel level, but also differentiation of instances to foreground classes. The task is an important basic task for scene understanding, and has wide application value in the fields of automatic driving and the like. The current mainstream technical route is divided into a top-down mode and a bottom-up mode. The top-down approach finds the bounding box of the instance first and then determines whether the pixel-by-pixel within the box belongs to the instance. The bottom-up approach first predicts the pixel-by-pixel instance attribution and generates a bounding box based thereon. From empirical results, top-down technical routes tend to perform better than bottom-up ones.
However, the top-down approach suffers from two major problems. First, in this scheme, two sub-networks exist for the two tasks of semantic segmentation and instance segmentation, but there is no way of information propagation between the two sub-networks. Thus, the complementarity between these two tasks is not well exploited. Second, for detected instances, there may be instances of mutual occlusion. Past approaches rely on the class score of the target to handle the occlusion relationship, but this is clearly not optimal due to the relevance of the class score to other factors such as data distribution. How to solve the two problems becomes the key of the top-down panorama segmentation method.
Disclosure of Invention
In order to solve the two problems, the invention provides a panorama segmentation method with bidirectional connection and occlusion processing. The method is based on a deep learning network, and features between the two tasks can be mutually strengthened by establishing bidirectional connection between semantic segmentation and instance segmentation. In addition, we propose an occlusion handling algorithm to deal exclusively with occlusion problems between instances. Through the two points, the method of the invention obtains excellent panorama segmentation performance.
The technical scheme of the invention comprises the following steps:
a panorama segmentation method with bidirectional connection and shielding processing comprises the following steps:
s1, acquiring a data set for training panoramic segmentation, and defining an algorithm target;
s2, extracting the features of the images in the data set by using a full convolution network to obtain a feature map of the images;
s3, extracting semantic features from the feature map by using a semantic feature extraction network;
s4, extracting example features from the feature graph by using an example feature extraction network;
s5, establishing connection from instance segmentation to semantic segmentation, and aggregating semantic features and instance features to perform semantic segmentation;
s6, establishing connection from semantic segmentation to instance segmentation, and aggregating instance features and semantic features to perform instance segmentation;
and S7, fusing the results of semantic segmentation and example segmentation by using an occlusion processing algorithm, and outputting a panoramic segmentation result.
On the basis of the scheme, the steps can be further realized in the following preferred mode.
Preferably, the algorithm goal of step S1 is: for each picture I in the data set for panoramic segmentation, identifying the semantic category to which the background pixel appears in the picture I; for foreground pixels appearing in I, the semantic class and the belonging instance to which they belong are identified.
Preferably, in step S2, a full convolution neural network is used to extract features for each pixel in the image, and a feature map F ═ Φ (I) of the image is obtained.
Preferably, in step S3, the feature map F extracted in S2 is input to extract semantic features from the feature map using a full convolutional neural network ψ, and the semantic features S ψ (F) ψ (Φ (I)) are extracted.
Preferably, the extracting example features described in step S4 specifically includes the following sub-steps:
s41, detecting an example set O in the image by using an area extraction network, and obtaining O ═ { O ═ O1，...，OkIn which O isiDenotes the ith instance detected, i ∈ [1, k ]]K is the total number of detected instances;
s42, for each detected example OiCalculate its bounding box Bi；
S43, extracting example features by using the example feature extraction network zeta, namely inputting the feature graph F extracted in S2 and the example OiSurrounding frame BiExtract its instance features
Figure BDA0002376314970000031
Figure BDA0002376314970000032
Preferably, the specific steps of establishing the connection from instance segmentation to semantic segmentation described in step S5 are as follows:
s51, restoring spatial information F of the example features extracted in S4 by using RoIInlay with differentiable operationI：
FI＝RoIInlay(I1，...，Ik，B1，...，Bk)；
The specific operation of the differentiable operation roinlay is as follows: for an example with coordinates (a, b) at the top left corner and a size h × w, assuming that it is clipped and deformed to obtain a feature map of m × m, each point (u, v) on the feature map is sampled from a position (x, y) of the original feature map:
Figure BDA0002376314970000033
that is, the value at the point (u, v) on the m × m feature map corresponds to the value v (x, y) at the point (x, y) on the original feature map, and thus, for any point (x) located within the target regionp，yp) Finding four sampling points surrounding it, marked as set C, and obtaining (x) by bilinear interpolationp，yp) Value at a point
Figure BDA0002376314970000041
Figure BDA0002376314970000042
Wherein G iswAnd GhIs an interpolation function under the relative coordinate system of the sample points:
Figure BDA0002376314970000043
Figure BDA0002376314970000044
in the formula: parameter(s)
Figure BDA0002376314970000045
Parameter(s)
Figure BDA0002376314970000046
For values within the target region but beyond the boundaries of the sample points, the sample points are pulled to the target boundaries;
s52, FIThe feature obtained by aggregation with the semantic feature S extracted in the step S3 is used for predicting a semantic segmentation result; the specific operation of characteristic polymerization is as follows: firstly F is put inISplicing the signal S with the channel dimension to form a new feature, and then carrying out 1-layer 3 × 3 convolution processing on the new feature to eliminate deformation caused by RoIInlay; then, after multi-scale pooling operation is carried out on the feature, descriptions of scenes with the sizes of 8 × 8, 4 × 4, 2 × 2 and 1 × 1 are obtained respectively; and finally, flattening and splicing the descriptions, splicing the spliced features and the original features on each pixel point, and performing 1 × 1 convolution processing to obtain the aggregated features.
Preferably, the specific operation of establishing the connection from semantic segmentation to instance segmentation in step S6 is as follows: first get instance O from semantic feature S using RoIAlign operationiCorresponding semantic feature SOi＝RoIAlign(S，Bi) (ii) a Then the S isOiAfter a 3 x 3 convolution processing, the data is compared with the example characteristic IiAnd adding the aggregated features to predict the segmentation result of the example, wherein the segmentation result comprises the position, the category and the segmentation map of the example.
Preferably, the step S7 includes the following steps: firstly, judging the covering relation between mutually shielded examples through a shielding processing algorithm, and then fusing the example segmentation result and the semantic segmentation result to obtain a panoramic segmentation result; wherein for target instance OiAnd OjAssuming that the overlapping region is P, the appearance of the region is defined as the average of RGB values of pixels in the region, and the attribution of the P region, i.e. O, is judged by comparing the appearance of the P region with the similarity of the appearance of two targetsiAnd OjThe occlusion relationship of (1).
The invention can fully utilize the complementarity between the semantic segmentation and the instance segmentation to establish the connection between the two tasks, so that the two tasks are mutually beneficial, and the panoramic segmentation performance is finally improved. Meanwhile, the invention utilizes the shielding processing algorithm provided by the bottom layer characteristic appearance information to effectively process the shielding problem, so that the model performance is further improved.
Drawings
FIG. 1 is a schematic flow chart of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
On the contrary, the invention is intended to cover alternatives, modifications, equivalents and alternatives which may be included within the spirit and scope of the invention as defined by the appended claims. Furthermore, in the following detailed description of the present invention, certain specific details are set forth in order to provide a better understanding of the present invention. It will be apparent to one skilled in the art that the present invention may be practiced without these specific details.
As shown in fig. 1, a panorama segmentation method with bidirectional connection and occlusion processing includes the following steps:
s1, acquiring a data set for training panoramic segmentation, and defining an algorithm target. In this step, the algorithm targets are: for each picture I in the data set for panoramic segmentation, identifying the semantic category to which the background pixel appears in the picture I; for foreground pixels appearing in I, the semantic class and the belonging instance to which they belong are identified.
And S2, performing feature extraction on the image in the data set by using a full convolution network to obtain a feature map of the image. In this step, a full convolution neural network is used to extract features for each pixel in the image, and a feature map F ═ Φ (I) of the image is obtained.
And S3, extracting semantic features from the feature map by using the semantic feature extraction network branches. In this step, a feature map F extracted in S2, which is a semantic feature extracted from the feature map, is extracted using one full convolution neural network ψ, and its semantic feature S ψ (F) ψ (Φ (I)) is extracted.
And S4, extracting example features from the feature graph by using the example feature extraction network branches. In this step, extracting example features specifically includes the following substeps:
s41, detecting an example set O in the image by using an area extraction network (RPN), and obtaining O ═ O1，...，OkIn which O isiDenotes the ith instance detected, i ∈ [1, k ]]K is the total number of detected instances;
s42, for each detected example OiCalculate its bounding box Bi；
S43, extracting example features by using the example feature extraction network zeta, namely inputting the feature graph F extracted in S2 and the example OiSurrounding frame BiExtracting its example feature Ii＝ζ(F，Bi)＝ζ(φ(I)，Bi)。
And S5, establishing connection from instance segmentation to semantic segmentation through a first feature aggregation module, and aggregating semantic features and instance features to perform semantic segmentation. In this step, the specific steps of establishing the connection from instance segmentation to semantic segmentation are as follows:
s51, restoring spatial information F of the example features extracted in S4 by using RoIInlay with differentiable operationI：
FI＝RoIInlay(I1，...，Ik，B1，...，Bk)；
The specific operation of the differentiable operation roinlay is as follows: for an example with coordinates of (a, b) at the upper left corner and a size of h × w (the upper left coordinate of the example refers to the upper left coordinate of the bounding box of the target object corresponding to the example), assuming that the example is subjected to clipping and deformation to obtain a feature map of m × m, each point (u, v) on the feature map is sampled from a position (x, y) of the original feature map:
Figure BDA0002376314970000071
that is, the value at the point (u, v) on the m × m feature map corresponds to the value v (x, y) at the point (x, y) on the original feature map, and thus, for any point (x) located within the target regionp，yp) Finding four sampling points surrounding it, marked as set C, and obtaining (x) by bilinear interpolationp，yp) Value at a point
Figure BDA0002376314970000072
Figure BDA0002376314970000073
Wherein G iswAnd GhIs an interpolation function under the relative coordinate system of the sample points:
Figure BDA0002376314970000074
Figure BDA0002376314970000075
in the formula: parameter(s)
Figure BDA0002376314970000076
Parameter(s)
Figure BDA0002376314970000077
For values within the target region but beyond the boundaries of the sample points, the sample points are pulled to the target boundaries;
s52, mixing FIThe feature obtained by aggregating with the semantic feature S extracted in step S3 is used to predict the semantic segmentation result. The feature aggregation is performed in the first feature aggregation module, and the specific operations are as follows: firstly F is put inISplicing with S in channel dimension (concatenate) to form a new feature, and then performing 1-layer 3 × 3 convolution on the new featureProcessing to eliminate distortion caused by RoIInlay; then, after the feature is subjected to multiscale Pooling (Avg Pooling), descriptions of scenes with sizes of 8 × 8, 4 × 4, 2 × 2 and 1 × 1 are obtained respectively; and finally, flattening (flatten) and splicing the descriptions, splicing the spliced features and the original features on each pixel point, and performing convolution processing of 1 × 1 to obtain the aggregated features. And (4) performing convolution of 1 multiplied by 1 according to the characteristics obtained after aggregation to obtain a prediction partitioning result.
And S6, establishing connection from semantic segmentation to instance segmentation through a second feature aggregation module, and aggregating instance features and semantic features to perform instance segmentation. The specific operation of establishing the connection from semantic segmentation to instance segmentation is as follows: first get instance O from semantic feature S using RoIAlign operationiCorresponding semantic feature SOi＝RoIAlign(S，Bi) (ii) a Then in a second feature aggregation module, SOiAfter a 3 x 3 convolution processing, the data is compared with the example characteristic IiAnd adding the aggregated features to predict the segmentation result of the example, wherein the segmentation result comprises the position, the category and the segmentation map of the example. According to the characteristics obtained after aggregation, the prediction segmentation result of the example can be obtained by performing 1 × 1 convolution on the characteristics.
And S7, judging the coverage relation between mutually shielded examples by using a shielding processing algorithm, fusing the results of semantic segmentation and example segmentation, and outputting a panoramic segmentation result. In this step, the specific steps of outputting the panorama segmentation result are as follows: firstly, judging the covering relation between mutually shielded examples through a shielding processing algorithm, and then fusing the example segmentation result and the semantic segmentation result to obtain a panoramic segmentation result; wherein for target instance OiAnd OjAssuming that the overlapping region is P, the appearance of the region is defined as the average of RGB values of pixels in the region, and the attribution of the P region, i.e. O, is judged by comparing the appearance of the P region with the similarity of the appearance of two targetsiAnd OjThe occlusion relationship of (1).
The panorama segmentation algorithm firstly uses a common full convolution neural network, semantically segments branches and extracts features by instance segmentation. Then, by means of original RoIInlay operation, bidirectional connection of feature levels is established between the semantic segmentation task and the instance segmentation task, and the complementary relation between the two tasks is fully utilized. For the occlusion problem between the possible instances, the invention designs a simple and effective occlusion processing algorithm. The algorithm utilizes the appearance information of the bottom layer, and can deduce the occlusion relation between the instances without training. Through the two points, the method achieves excellent panorama segmentation performance.
Examples
The following simulation experiment is performed based on the above method, and the implementation method of this embodiment is as described above, and the specific steps are not described in detail, and only the experimental results are shown below.
This embodiment uses ResNet-50 and FPN (feature Pyramid network) as the base network (backbone) to extract features. The semantic feature extraction network is formed by stacking three layers of Deformable Convolution (Deformable Convolution). The example feature extraction network is stacked from three layers of conventional convolutions. The model of the invention is trained on the training set of the COCO data set and is subjected to performance test on the corresponding verification set. The performance is shown in table 1, compared to the model without bi-directional connection and without occlusion inference.
TABLE 1 comparison of the Performance of the different models
Bidirectional connection Occlusion handling PQ(％)
× × 41.3
√ × 41.8
√ √ 43.0
Note: in the table, x represents use, and v represents non-use.
Therefore, through the technical scheme, the panoramic segmentation method is developed based on the deep learning technology. The invention can fully utilize the complementarity between the semantic segmentation and the instance segmentation to establish the connection between the two tasks, so that the two tasks are mutually beneficial, and the panoramic segmentation performance is finally improved. Meanwhile, the invention utilizes the shielding processing algorithm provided by the bottom layer characteristic appearance information to effectively process the shielding problem, so that the model performance is further improved.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.

Claims (5)

1. A panorama segmentation method with bidirectional connection and shielding processing is characterized by comprising the following steps:
s1, acquiring a data set for training panoramic segmentation, and defining an algorithm target;
s2, extracting the features of the images in the data set by using a full convolution network to obtain a feature map of the images;
s3, extracting semantic features from the feature map by using a semantic feature extraction network;
s4, extracting example features from the feature graph by using an example feature extraction network;
s5, establishing connection from instance segmentation to semantic segmentation, and aggregating semantic features and instance features to perform semantic segmentation;
s6, establishing connection from semantic segmentation to instance segmentation, and aggregating instance features and semantic features to perform instance segmentation;
s7, fusing semantic segmentation and instance segmentation results by using an occlusion processing algorithm, and outputting a panoramic segmentation result;
the extracting of the example features described in step S4 specifically includes the following sub-steps:
s41, detecting an example set O in the image by using an area extraction network, and obtaining O ═ { O ═ O1，...，OkIn which O isiDenotes the ith instance detected, i ∈ [1, k ]]K is the total number of detected instances;
s42, for each detected example OiCalculate its bounding box Bi；
S43, extracting example features by using the example feature extraction network zeta, namely inputting the feature graph F extracted in S2 and the example OiSurrounding frame BiExtracting its example feature Ii＝ζ(F，Bi)＝ζ(φ(I)，Bi)；
The specific steps of establishing the connection from instance segmentation to semantic segmentation described in step S5 are as follows:
s51, restoring spatial information F of the example features extracted in S4 by using RoIInlay with differentiable operationI：
FI＝RoIInlay(I1，...，Ik，B1，...，Bk)；
The specific operation of the differentiable operation roinlay is as follows: for an example with coordinates (a, b) at the upper left corner and a size h × w, assuming that the example is subjected to clipping and deformation to obtain an m × m feature map, each point (u, v) on the feature map is sampled from a position (x, y) of the original feature map:
Figure FDA0003559345730000021
i.e., the value at point (u, v) on the m × m feature map corresponds to the value v (x, y) at point (x, y) on the original feature map, and thus for any one located within the target regionMean one point (x)p，yp) Finding four sampling points surrounding it, marked as set C, and obtaining (x) by bilinear interpolationp，yp) Value at a point
Figure FDA0003559345730000022
Figure FDA0003559345730000023
Wherein G iswAnd GhIs an interpolation function in the relative coordinate system of the sample points:
Figure FDA0003559345730000024
Figure FDA0003559345730000025
in the formula: parameter(s)
Figure FDA0003559345730000026
Parameter(s)
Figure FDA0003559345730000027
For values within the target region but beyond the boundaries of the sample points, the sample points are pulled to the target boundaries;
s52, FIThe feature obtained by aggregating the semantic features S extracted in the step S3 is used for predicting semantic segmentation results; the specific operation of characteristic polymerization is as follows: firstly F is put inISplicing the signal S with the channel dimension to form a new feature, and then carrying out 1-layer 3 × 3 convolution processing on the new feature to eliminate deformation caused by RoIInlay; then, after multi-scale pooling operation is carried out on the feature, descriptions of scenes with the sizes of 8 × 8, 4 × 4, 2 × 2 and 1 × 1 are obtained respectively; finally, the descriptions are leveled and spliced, and the splicing is carried outSplicing the post-feature and the original feature on each pixel point, and obtaining the aggregated feature after 1 × 1 convolution processing;
the specific operation of establishing the connection from semantic segmentation to instance segmentation described in step S6 is as follows: first get instance O from semantic feature S using RoIAlign operationiCorresponding semantic features
Figure FDA0003559345730000031
Then will be
Figure FDA0003559345730000032
After a 3 x 3 convolution processing, the data is compared with the example characteristic IiAnd adding the aggregated features to predict the segmentation result of the example, wherein the segmentation result comprises the position, the category and the segmentation map of the example.
2. A method of panorama segmentation with bidirectional join and occlusion processing as claimed in claim 1, characterized in that the algorithm goal of step S1 is: for each picture I in the data set for panoramic segmentation, identifying the semantic category to which the background pixel appears in the picture I; for foreground pixels appearing in I, the semantic class and the belonging instance to which they belong are identified.
3. A panorama segmentation method with bidirectional occlusion processing as claimed in claim 2, characterized in that in step S2, a fully convolutional neural network is used to extract features for each pixel in the image, and a feature map F ═ Φ (I) of the image is obtained.
4. A panorama segmentation method with bidirectional connection and occlusion processing as claimed in claim 3, characterized in that in step S3, a fully convolutional neural network ψ is used to extract semantic features from the feature map, i.e. the feature map F extracted in S2 is input, and its semantic features S ψ (F) ψ (Φ (I)) are extracted.
5. A method according to claim 1 with bidirectional connectionAnd a panorama segmentation method for occlusion processing, characterized in that step S7 specifically includes the steps of: firstly, judging the covering relation between mutually shielded examples through a shielding processing algorithm, and then fusing the example segmentation result and the semantic segmentation result to obtain a panoramic segmentation result; wherein for target instance OiAnd OjAssuming that the overlapping region is P, the appearance of the region is defined as the average of RGB values of pixels in the region, and the attribution of the P region, i.e. O, is judged by comparing the appearance of the P region with the similarity of the appearance of two targetsiAnd OjThe occlusion relationship of (1).
CN202010067124.7A 2020-01-20 2020-01-20 Panorama segmentation method with bidirectional connection and shielding processing Active CN111242954B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010067124.7A CN111242954B (en) 2020-01-20 2020-01-20 Panorama segmentation method with bidirectional connection and shielding processing

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010067124.7A CN111242954B (en) 2020-01-20 2020-01-20 Panorama segmentation method with bidirectional connection and shielding processing

Publications (2)

Publication Number Publication Date
CN111242954A CN111242954A (en) 2020-06-05
CN111242954B true CN111242954B (en) 2022-05-13

Family

ID=70879738

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010067124.7A Active CN111242954B (en) 2020-01-20 2020-01-20 Panorama segmentation method with bidirectional connection and shielding processing

Country Status (1)

Country Link
CN (1) CN111242954B (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111524150B (en) * 2020-07-03 2021-06-11 支付宝(杭州)信息技术有限公司 Image processing method and device
CN112489064B (en) * 2020-12-14 2022-03-25 桂林电子科技大学 Panorama segmentation method based on edge scaling correction
CN112699888A (en) * 2020-12-31 2021-04-23 上海肇观电子科技有限公司 Image recognition method, target object extraction method, device, medium and equipment
CN112699894A (en) * 2021-01-13 2021-04-23 上海微亿智造科技有限公司 Method, system and medium for improving segmentation precision and speed of industrial quality inspection example
CN113052858B (en) * 2021-03-23 2023-02-14 电子科技大学 Panorama segmentation method based on semantic stream
CN113096136A (en) * 2021-03-30 2021-07-09 电子科技大学 Panoramic segmentation method based on deep learning
CN116468889B (en) * 2023-04-04 2023-11-07 中国航天员科研训练中心 Panorama segmentation method and system based on multi-branch feature extraction

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109801297A (en) * 2019-01-14 2019-05-24 浙江大学 A kind of image panorama segmentation prediction optimization method realized based on convolution
CN109886272A (en) * 2019-02-25 2019-06-14 腾讯科技（深圳）有限公司 Point cloud segmentation method, apparatus, computer readable storage medium and computer equipment
CN110008808A (en) * 2018-12-29 2019-07-12 北京迈格威科技有限公司 Panorama dividing method, device and system and storage medium
WO2019185170A1 (en) * 2018-03-30 2019-10-03 Toyota Motor Europe Electronic device, robotic system and method for localizing a robotic system
CN110533720A (en) * 2019-08-20 2019-12-03 西安电子科技大学 Semantic SLAM system and method based on joint constraint
CN110533048A (en) * 2018-05-23 2019-12-03 上海交通大学 The realization method and system of combination semantic hierarchies link model based on panoramic field scene perception

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019185170A1 (en) * 2018-03-30 2019-10-03 Toyota Motor Europe Electronic device, robotic system and method for localizing a robotic system
CN110533048A (en) * 2018-05-23 2019-12-03 上海交通大学 The realization method and system of combination semantic hierarchies link model based on panoramic field scene perception
CN110008808A (en) * 2018-12-29 2019-07-12 北京迈格威科技有限公司 Panorama dividing method, device and system and storage medium
CN109801297A (en) * 2019-01-14 2019-05-24 浙江大学 A kind of image panorama segmentation prediction optimization method realized based on convolution
CN109886272A (en) * 2019-02-25 2019-06-14 腾讯科技（深圳）有限公司 Point cloud segmentation method, apparatus, computer readable storage medium and computer equipment
CN110533720A (en) * 2019-08-20 2019-12-03 西安电子科技大学 Semantic SLAM system and method based on joint constraint

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于深度学习的点云分割方法综述;俞斌等;《计算机工程与应用》;20191211(第01期);全文 *

Also Published As

Publication number Publication date
CN111242954A (en) 2020-06-05

Similar Documents

Publication Publication Date Title
CN111242954B (en) Panorama segmentation method with bidirectional connection and shielding processing
CN107274419B (en) Deep learning significance detection method based on global prior and local context
CN110222787B (en) Multi-scale target detection method and device, computer equipment and storage medium
Xu et al. Inter/intra-category discriminative features for aerial image classification: A quality-aware selection model
CN104751142B (en) A kind of natural scene Method for text detection based on stroke feature
WO2018103608A1 (en) Text detection method, device and storage medium
Wang et al. Background-driven salient object detection
CN109086777B (en) Saliency map refining method based on global pixel characteristics
CN109214403B (en) Image recognition method, device and equipment and readable medium
CN110852349A (en) Image processing method, detection method, related equipment and storage medium
WO2020233397A1 (en) Method and apparatus for detecting target in video, and computing device and storage medium
US11887346B2 (en) Systems and methods for image feature extraction
CN111768415A (en) Image instance segmentation method without quantization pooling
CN111640089A (en) Defect detection method and device based on feature map center point
CN110852327A (en) Image processing method, image processing device, electronic equipment and storage medium
CN108345835B (en) Target identification method based on compound eye imitation perception
Dong et al. Learning regional purity for instance segmentation on 3d point clouds
CN115375917A (en) Target edge feature extraction method, device, terminal and storage medium
CN113628181A (en) Image processing method, image processing device, electronic equipment and storage medium
CN111047614A (en) Feature extraction-based method for extracting target corner of complex scene image
Dadgostar et al. Gesture-based human–machine interfaces: a novel approach for robust hand and face tracking
CN112446230B (en) Lane line image recognition method and device
CN114724175A (en) Pedestrian image detection network, detection method, training method, electronic device, and medium
US20220237932A1 (en) Computer implemented method for segmenting a binarized document
CN114155273A (en) Video image single-target tracking method combined with historical track information

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant