KR102167808B1 - Semantic segmentation method and system applicable to AR - Google Patents

Abstract

The present invention relates to a method and a system for semantic segmentation applicable to augmented reality (AR). The present invention is to improve the performance rate and accuracy of video analysis in order to apply the video analysis to AR. The method includes: a semantic segmentation image acquisition step (S10) of acquiring a semantic segmentation image by classifying an object from an input image and labeling it; a modified dilated residual network (DRN) step (S20) of extracting a feature map from the acquired image by using atrous convolution; and an atrous pyramid pooling module step (S30) of selectively applying various atrous convolutions in accordance with the extracted feature map, extracting a feature by arranging the atrous convolutions in parallel, and then forming a pyramid-shaped feature map in order to effectively extract an object occupying a small region in the image.

Description

Semantic segmentation method and system applicable to AR

The present invention relates to a semantic segmentation method and system applicable to AR, and more specifically, semantic segmentation applicable to AR that improves the speed and accuracy of image analysis in order to apply image analysis to AR (Augmented Reality). It relates to a method and system.

As the development of artificial intelligence and robots accelerates after the 4th industrial revolution, research in the field of thinking like a human is expanding. Therefore, there is increasing interest in research that moves, judges and performs actions in real time, such as virtual and augmented reality systems, autonomous driving, medical robots, and drones.

The basis of this research is a study that analyzes the image coming through a camera that replaces the human eye. In the study of image analysis, a study on semantic segmentation, which labels each pixel to which class, is a basic task.

Semantic segmentation is a technique of dividing images by pixels for a class that has been learned in advance, and not simply classifying photos by class, but a high-level technique that fully understands the scene in the image, and fully understands the visual environment. It is one of the key computer vision skills required for this.

In addition, the semantic segmentation algorithm requires high efficient execution speed for quick interaction or response and accuracy for accurate determination. For example, the speed of performing semantic division and the accuracy for accurate judgment are essential factors for technologies such as safety control and driving decisions in autonomous driving, and collision avoidance.

However, it is difficult to perform accurate semantic segmentation in real time in an image actually photographed. First, visual objects are often affected by deformation, occlusion and scale deformation. Second, background noise makes it difficult to separate objects from the background.

Therefore, to deal with these problems, we need an algorithm that is resilient and robust to appearance changes. At the same time, it is necessary to consider various context-specific information to distinguish objects from complex backgrounds.

Republic of Korea Patent Publication No. 10-2019-0033933 (published on April 1, 2019)

Accordingly, an object of the present invention is to solve the conventional shortcomings and to improve the accuracy of image analysis by effectively extracting a small object from an input image. In addition, the purpose of this is to improve the speed of performing image analysis in order to apply image analysis to AR (Augmented Reality).

A semantic segmentation method applicable to AR according to an aspect of the present invention for achieving such a technical problem is semantic segmentation in which an object is classified from an input image and is labeled to obtain a semantic segmented image. Including an image acquisition step (S10) and a modified extended residual network (DRN, Dilated Residual Network) step (S20) of extracting a feature map from an image acquired using Atrous convolution. do.

In addition, in order to effectively extract an object with a small area occupied in the image, various Atrous convolutions are selectively applied according to the extracted feature point map, and feature points are extracted by arranging the Atrous convolutions in parallel. The result extracted through the Atrous Pyramid Pooling Module step (S30) and the Atrous Pyramid Pooling Module step (S30) forming a feature point map in a pyramid shape and the standard database And a modified extended residual network backpropagation step (S40) of comparing the provided result images and correcting weights to reduce an error rate based on the comparison result.

In this case, in the Atros pyramid pulling module step (S30), the feature point map extracted in the Atros pyramid pulling module step (S30) is stacked in a pyramid shape, and the feature point map formed in the pyramid shape is applied to a 1x1 convolution. It includes the process of forming a feature point map of.

In addition, a semantic segmentation system applicable to an AR according to another aspect of the present invention includes an image input unit, a segmented image acquisition unit, a feature point extraction unit, a determination unit, and a storage unit. The image input unit may receive image information captured through an imaging device such as a camera or image information through a standard database.

In addition, the segmented image acquisition unit classifies an object from image information received through an image input unit, and labels the classified object to obtain a semantic segmentation image.

In addition, the feature point extraction unit includes an extended residual network module and an Atros pyramid pooling module. The extended residual network module extracts a feature map from the image acquired through the segmented image acquisition unit using Atrous convolution.

In addition, the Atros pyramid pooling module selectively applies various Atrous convolutions according to the feature point map extracted from the extended residual network module in order to effectively extract objects with a small area occupied in the image. The feature point map is extracted by arranging convolutions in parallel.

In addition, the determination unit compares a result extracted from the Atros pyramid pooling module with a result image provided from a preset standard database, determines based on the comparison result, and corrects the weight to reduce an error rate.

As described above, the semantic segmentation method and system applicable to AR according to the present invention can effectively extract small objects and improve the accuracy of image analysis by applying Atros convolution to the feature point map extracted from the image. There is an effect.

In addition, it is possible to improve the speed of semantic segmentation to apply image analysis to AR (Augmented Reality) by forming a feature point map in a pyramid shape after extracting feature points by arranging the Atros convolution in parallel. There is an effect. In addition, by using Dilated Residual Network Backpropagation, there is an effect of reducing an error rate of semantic division and improving accuracy.

1 is a conceptual diagram illustrating a semantic segmentation method applicable to an AR according to an embodiment of the present invention.
2 is a flowchart illustrating a semantic segmentation method applicable to AR according to an embodiment of the present invention.
3 is a diagram illustrating a semantic segmentation image acquisition process.
4 is a diagram showing an image of the PASCAL VOC 2012 database.
5 is a diagram showing an image of a Cityscape database.
6 is a diagram illustrating an Atrous convolution according to an embodiment of the present invention.
FIG. 7 is a diagram illustrating a result of performing semantic division between a conventional convolution and an Atros convolution according to an embodiment of the present invention.
8 is a diagram illustrating an extended residual network (DRN) according to an embodiment of the present invention.
9 is a diagram illustrating an Atros pyramid pooling module according to an embodiment of the present invention.
10 is a diagram showing a structure of convolution.
11A and 11B are diagrams illustrating a graph of Karpathy calculation in a backpropagation process.
12 is a diagram illustrating an image of a backpropagation process according to an embodiment of the present invention.
13 is a flowchart illustrating an evaluation procedure of a semantic segmentation method applicable to an AR according to an embodiment of the present invention.
14 is a block diagram illustrating a semantic segmentation system applicable to an AR according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, the present invention may be implemented in various forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless otherwise stated. In addition, terms such as "... unit", "... group", and "... module" described in the specification mean a unit that processes at least one function or operation, which is implemented by hardware or software or a combination of hardware and software. Can be.

Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the accompanying drawings.

The same reference numerals in each drawing indicate the same members.

FIG. 1 is a conceptual diagram illustrating a semantic segmentation method applicable to an Augmented Reality (AR) according to an embodiment of the present invention, and FIG. 2 is a flowchart illustrating a semantic segmentation method applicable to an AR according to an embodiment of the present invention.

Semantic segmentation method applicable to AR according to an embodiment of the present invention is to obtain a semantic segmentation image that classifies objects from an input image and obtains a semantic segmented image by labeling. Step S10 and a modified extended residual network (DRN, Dilated Residual Network) step S20 of extracting a feature map from the image acquired using Atrous convolution.

3 is a diagram showing a semantic segmentation image acquisition process, FIG. 4 is a diagram showing an image of a PASCAL VOC 2012 database, and FIG. 5 is a diagram showing an image of a Cityscape database .

As shown in FIG. 3, the semantic segmented image acquisition step (S10) includes taking an image through an imaging device such as a camera (S11), detecting an object from the captured image (S12), and detecting the detected object. Based on the labeling, a step of obtaining a segmentation image (S13) may be included.

Meanwhile, a normalized standard database may be used for objective evaluation of a semantic segmentation method applicable to AR according to an embodiment of the present invention. That is, a normalized standard database is used to describe a semantic segmentation method applicable to AR according to an embodiment of the present invention.

For example, semantic segmentation images can be obtained by classifying and labeling objects of an image using the PASCAL VOC 2012 database of FIG. 4 and the Cityscape database of FIG. 5. have.

The PASCAL VOC 2012 database of FIG. 4 includes an airplane, a bicycle, a bird, a boat, a bottle, a bus, a car, and a cat. , Chair, cow, dining table, dog, horse, motorbike, person, potted plant, sheep, sofa It represents a database consisting of a total of 20 classes, such as sofa), train, and TV/monitor.

In addition, the Cityscape database of FIG. 5 is a standard database constructed from urban street scene images and published. That is, the Cityscape database is a set of image data taken at various days and times in 50 cities, and is composed of 5000 images having 30 classes.

In this case, it is preferable to adjust the semantic segmentation image to the same size to maintain consistency. For example, the semantic divided images may be adjusted to a size of 513x513.

6 is a diagram illustrating an Atrous convolution according to an embodiment of the present invention. That is, the drawing (a) of FIG. 6 is a diagram showing the Atrous convolution when the size of the rate r=1, and the drawing (b) is the size of the rate r=2 days. It is a diagram showing atrous convolution at the time, and FIG. (c) is a diagram showing atrous convolution when the size r = 3 of the rate.

6, the modified extended residual network (DRN, Dilated Residual Network) step (S20) maintains spatial information as much as possible by transforming convolution and extracts feature points at Atrous convolution. ) To extract a feature map from the original image.

The Atrous convolution structure increases the size of the window without increasing the number of weights by adding zero padding inside the filter under the influence of wavelet analysis. .

This Atrous convolution can capture large features with the same amount of computation compared to the conventional convolution, and more spatial features can be extracted by parallel use of the Atrous convolution with various expansion ratios. .

이 1인 경우로 종래의 컨볼루션을 나타내는 수식이다. 또한, 아래의 [수학식 2]는

이 1보다 큰 경우로 아트로스 컨볼루션을 나타내는 수식이다.[Equation 1] below is

In the case of 1, this is an equation representing a conventional convolution. Also, [Equation 2] below

If this is greater than 1, it is an equation representing the atros convolution.

[Equation 1]

[Equation 2]

Here, k represents a kernel, and s represents a stride.

FIG. 7 is a diagram illustrating a result of performing semantic division between a conventional convolution and an Atros convolution according to an embodiment of the present invention. When semantic division is performed with a small feature point map obtained using a conventional convolution network, accuracy is reduced.

As shown in FIG. 7, the upper image performing semantic segmentation through down-sampling, convolution, and up-sampling, and Atrous convolution. Through this, it is possible to check the difference between the lower image that performs semantic division.

7, it can be seen that the resolution of semantic segmentation is degraded while up-sampling is performed in a state in which there is a loss of spatial information in a state in which spatial information is lost.

However, when looking at the image below performing Atrous convolution in FIG. 7, convolution is performed while taking a large receptive field, thereby minimizing loss of spatial information and obtaining a high resolution result. .

Therefore, the semantic segmentation method applicable to AR according to an embodiment of the present invention extends the kernel with empty weights using Atrous convolution, so that the network does not rely on a pooling function, and provides long distance features. Learning, and without pooling, the network can maintain more high spatial frequency details.

FIG. 8 is a diagram showing a Dilated Residual Network (DRN) according to an embodiment of the present invention, and FIG. 9 is a diagram showing an Atrous pyramid pooling module according to an embodiment of the present invention. .

That is, FIG. 8 is a diagram illustrating a modification of 101 stages of a residual network by applying the principle of DRN, and extracting a feature point map using the modified DRN. [Table 1] below shows the structure of a DRN according to an embodiment of the present invention.

[Table 1] Structure table of DRN (Dilated Residual Network)

The DRN according to an embodiment of the present invention generates a feature point map by extracting only important features while preserving the features of the spatial domain for the entire image from the input image as much as possible.

In addition, the semantic segmentation method applicable to AR according to an embodiment of the present invention selectively uses various atrous convolutions according to the extracted feature point map in order to effectively extract an object with a small area occupied by an image. convolution), extracting feature points by arranging the Atros convolution in parallel, and then forming a feature point map in a pyramid shape (S30), and the Atrous Pyramid Pooling Modified extended residual network backpropagation (Dilated Residual) that compares the result extracted through the Atrous Pyramid Pooling Module step (S30) with the result image provided from the standard database, and corrects the weight to reduce the error rate based on the comparison result. Network Backpropagation) step (S40).

In order to increase accuracy in semantic segmentation, it is important to accurately extract even small objects. However, in the conventional semantic segmentation field, when small objects are complexly arranged, there is a lot of difficulty in segmentation.

Therefore, the semantic segmentation method applicable to AR according to an embodiment of the present invention is applied by applying a modification in Pyramid Sense Pooling of a Pyramid Scene Parsing Network (PSPNet) in order to solve this problem.

For example, as shown in FIG. 9, a feature point map may be extracted by applying five types of atros convolutions in parallel to a feature point map having a size of 28x28 obtained through DRN. The Atros convolution applied here is a general convolution with a rate of 1, an Atros convolution with a rate of 3, 6, and 9, and the modified extended residual network (DRN) step. Image pooling is applied to the feature point map extracted in (S20).

After that, the five feature point maps extracted as shown in FIG. 9 are stacked in a pyramid shape, and a 1-channel feature point map is extracted by applying 1x1 convolution.

In this way, the Atrous Pyramid Pooling Module step (S30) is performed in the modified extended residual network (DRN, Dilated Residual Network) step (S20) to effectively extract an object with a small area occupied in the image. Various Atrous convolutions are applied to the extracted feature map.

In addition, in the Atrous Pyramid Pooling Module step (S30), feature points are extracted by arranging atrous convolutions in parallel to reduce the required time, and then a feature point map is stacked in a pyramid shape.

In this case, it is preferable to stack the feature point maps having a size smaller than a preset reference value to match the size of the feature point maps through up-sampling. In addition, when various feature point maps are stacked in a pyramid shape, a 1-channel feature point map is generated by applying a 1x1 convolution.

In addition, in the modified extended residual network backpropagation step (S40), semantic segmentation obtained through convolution from the final feature point map extracted from the Atrous Pyramid Pooling Module is provided in the database. When an error rate of more than a predetermined value is generated compared to the result image, the weight is corrected by applying it to the convolution performed in the modified extended residual network (DRN) step (S20), and the error rate is reduced.

In general, in a convolutional neural network (CNN), a filter extracts a regional feature while sliding input data, compresses it into a maximum value (max pooling) or an average value (average pooling), and transmits it to the next layer. Repeating this process is the general structure of CNN.

에 대한 역전파 과정을 나타내는 도면이고, 도 11b는 도 10의 컨볼루션 구조를 바탕으로

에 대한 역전파 과정을 나타내는 도면이다.10 is a diagram showing a structure of a convolution, and FIGS. 11A and 11B are diagrams illustrating a Karpathy calculation graph in a backpropagation process. That is, Figure 11a is based on the convolution structure of Figure 10

It is a diagram showing a backpropagation process for, and FIG. 11B is based on the convolutional structure of FIG.

It is a diagram showing a backpropagation process for.

는 각각 입력값의 i번째 행, j번째 열의 요소를 나타낸다. 이때, 해당 입력값에 필터의 크기가 3x3인 컨볼루션을 수행하면, 출력값은 2x2의 크기를 갖는다. 예를 들어, 도 10에서

의 값은

의 합성곱으로 출력된다.In Fig. 10, the input value is a 5x5 matrix,

Represents the elements of the i-th row and j-th column of the input value, respectively. At this time, when convolution with a filter size of 3x3 is performed on the input value, the output value has a size of 2x2. For example, in Figure 10

Is the value of

Is output as a convolution of

은 포워드(forward) 과정에서 3x3 필터의

가중치하고만 합성곱이 수행되기 때문에 역전파도 한 번만 진행된다. 이때,

의 기울기는 흘러들어온 기울기

에 상대방의 변화량을 나타내는 로컬 그래디언트(

)을 곱해서 구할 수 있다.In addition, as shown in Figure 11a

Is the 3x3 filter in the forward process.

Since convolution is performed only with the weights, backpropagation is performed only once. At this time,

The slope of the flow in

A local gradient (

It can be found by multiplying by ).

의 그래디언트는 흘러들어온 그래디언트

에 로컬 그래디언트(

)를 곱해 계산할 수 있다. 또한, 도 11b에서 도시된 바와 같이 상기

과 동일한 방식으로

에 대한 역전파를 계산할 수 있다.Likewise,

The gradient of is the flowing gradient

To the local gradient(

It can be calculated by multiplying by ). In addition, as shown in Figure 11b

In the same way

You can calculate the backpropagation for

12 is a diagram illustrating an image of a backpropagation process according to an embodiment of the present invention. As shown in FIGS. 11A and 11B, it is difficult to substitute the backpropagation method individually.

Therefore, the gradient can be simply obtained by using the backpropagation process image of FIG. 12. That is, if the elements of the filter used to create the convolution layer are reversed and convolution is performed on the gradient matrix, the gradient for the input vector can be obtained.

의 그래디언트는 도 12의 좌측 상단을 참고로 아래의 [수학식 1]을 이용하여 구할 수 있다.E.g,

The gradient of can be obtained using [Equation 1] below with reference to the upper left of FIG. 12.

[Equation 1]

이

와 연결되어 있기에 필터의 그래디언트는 흘러들어온 그래디언트(

)에 로컬 그래디언트를 곱해서 구할 수 있다.Also, the gradient of the filter is the first element of the gradient matrix

this

Because it is connected to, the gradient of the filter is the gradient (

) By the local gradient.

의 그래디언트는 아래의 [수학식 2]와 같이 구할 수 있다.therefore,

The gradient of can be obtained as [Equation 2] below.

[Equation 2]

Backpropagation is performed by comparing the result obtained by applying the input image and the modified DRN (Dilated Residual Network), but the result obtained through the Atros pyramid pooling module 320 may affect the actual accuracy.

Therefore, the modified extended residual network backpropagation step (S40) compares the result obtained through the Atros pyramid pooling module 320 as shown in FIG. 1 with the true value provided from the standard database, and is preset. Deformed extended residual network backpropagation is performed using the error rate.

13 is a flowchart illustrating an evaluation procedure of a semantic segmentation method applicable to an AR according to an embodiment of the present invention. Through the process shown in FIG. 13, the time taken for recognition according to the number of recognitions of the semantic segmentation method applicable to the AR of the present invention (the time required) and the intersection rate (accuracy) of the predicted bounding box and the actual ground truth bounding box Can be evaluated.

As shown in Fig. 13, normalization such as PASCAL VOC 2012 database and Cityscape database by applying DRN (Dilated Residual Network) and Atrous pyramid pooling module modified in the learning process Train the standard database.

In addition, in the execution process, semantic segmentation is performed based on previously learned data.

At this time, the image of the PASCAL VOC 2012 database used in the experiment is adjusted to a size of 513x513 to evaluate the accuracy and time required. In addition, "Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation" published by Liang-Chieh Chen and others in 2018 to evaluate the objective reliability of the semantic segmentation method applicable to AR according to an embodiment of the present invention Compare accuracy.

[Table 2] below shows the results of comparing the semantic segmentation method applicable to AR according to an embodiment of the present invention and the accuracy of the PASCAL VOC 2012 database for other papers.

[Table 2] Comparison of Semantic Segmentation Techniques and Other Papers with PASCAL VOC 2012 Database Accuracy

That is, the above [Table 2] is based on the accuracy results of "Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation" published by Liang-Chieh Chen et al. compared with techniques published in other papers in the same environment. , Shows the accuracy result compared with the semantic segmentation method applicable to AR according to an embodiment of the present invention.

As shown in [Table 2], the result of the semantic segmentation method applicable to AR according to an embodiment of the present invention exhibits higher accuracy than techniques published in other papers. The DeepLabv3+ technique presented in "Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation" by Liang-Chieh Chen and 4 others only considers the forward direction while performing a semantic segmentation technique. Backpropagation) is not performed, resulting in poor accuracy.

However, the semantic segmentation method applicable to AR according to an embodiment of the present invention performs backpropagation when mIOU, which is a measurement index, does not appear above a predetermined value in the final image.

[Table 3] below shows the time required for semantic segmentation for the entire image of the PASCAL VOC 2012 database.

[Table 3] Duration of semantic segmentation for images in PASCAL VOC 2012 database

As shown in [Table 3], the time taken by semantic segmentation is performed by inputting the image of the PASCAL VOC 2012 database. As shown in [Table 3], the semantic segmentation method applicable to AR according to an embodiment of the present invention is the number of images that semantic segmentation is performed in 1 second for the entire image of the PASCAL VOC 2012 database. Frames per second measured the time required shows a result of 64.3 (FPS).

Therefore, it can be confirmed that the frame per second (FPS) is 60 (FPS) or more, so it can be applied to the AR field that follows the human movement speed.

Meanwhile, in order to evaluate the objective reliability of the semantic segmentation method applicable to AR according to an embodiment of the present invention, "Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation", published by Liang-Chieh Chen and others in 2018, and Compare.

At this time, in order to evaluate the accuracy, the image of the Cityscape database used in the experiment is adjusted to a size of 513x513 as the image of the PASCAL VOC 2012 database and used.

[Table 4] below shows a comparison result of the semantic segmentation method applicable to AR according to an embodiment of the present invention and the accuracy of the Cityscape database for other papers.

[Table 4] Comparison of Semantic Segmentation Techniques and Other Papers with Cityscapes Database Accuracy

That is, the above [Table 4] is based on the results of comparing and evaluating "Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation" published by Liang-Chieh Chen and other papers in the same environment. A result of comparison with a semantic segmentation method applicable to AR according to an embodiment of the present invention is shown.

As shown in [Table 4], the result of the semantic segmentation method applicable to the AR according to the embodiment of the present invention exhibits higher accuracy than the results of the techniques published in other papers. The DeepLabv3+ technique of "Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation" published by Liang-Chieh Chen and 4 others only considers the forward0 direction and does not perform backpropagation. The accuracy is poor.

However, the semantic segmentation method applicable to AR according to an embodiment of the present invention exhibits better accuracy than techniques published in other papers by performing backpropagation when mIOU, which is a measurement index, does not appear above a certain value in the final image. .

[Table 5] below shows the time required for semantic segmentation for the entire image of the Cityscape database.

[Table 5] Duration of semantic segmentation for images in Cityscapes database

As shown in [Table 5], the time taken by semantic segmentation is performed by inputting an image from the Cityscape database. As shown in [Table 5], the semantic segmentation method applicable to the AR according to an embodiment of the present invention measures the time required by the number of images that semantic segmentation is performed in one second for a Cityscape database. One frame per second represents a result of 61 (FPS).

Therefore, it can be confirmed that the frame per second (FPS) is 60 (FPS) or more, so it can be applied to the AR field that follows the human movement speed.

14 is a block diagram illustrating a semantic segmentation system applicable to an AR according to an embodiment of the present invention. The semantic segmentation system 10 applicable to AR according to an embodiment of the present invention includes an image input unit 100, a segmented image acquisition unit 200, a feature point extraction unit 300, a determination unit 400, and a storage unit 500. ) Can be included.

The image input unit 100 may receive image information. That is, the image input unit 100 may receive image information captured through an imaging device such as a camera or image information through a standard database.

The standard database may include a PASCAL VOC 2012 database and a Cityscape database.

The segmented image acquisition unit 200 classifies an object from image information input through the image input unit 100 and labels the classified object to obtain a semantic segmentation image.

In addition, the feature point extraction unit 300 includes an extended residual network module 310 and an Atros pyramid pooling module 320. The extended residual network module 310 extracts a feature map from the image acquired through the segmented image acquisition unit 200 using Atrous convolution.

In addition, the Atros pyramid pooling module 320 selectively applies various Atrous convolutions according to the feature point map extracted from the extended residual network module 310 in order to effectively extract an object with a small area occupied in the image. And, by arranging the Atros convolution in parallel, the feature point map is extracted.

In addition, the determination unit 400 compares the result extracted from the Atros pyramid pooling module 320 with the result image provided from a preset standard database. In addition, the determination unit 400 determines based on the comparison result and corrects the weight to reduce the error rate.

That is, if an error rate equal to or greater than a preset error reference value occurs as a result of comparing the images, the determination unit 400 modifies the weight of the Atrous convolution performed by the extended residual network module 310 to provide a feature map of the image. map) again.

Further, the storage unit 500 may store image information or a standard database input through the image input unit 100. In addition, the storage unit 500 stores the feature point map extracted through the feature point extraction unit 300.

As described above, the semantic segmentation method and system applicable to the AR according to an embodiment of the present invention can effectively extract a small object by applying the Atros convolution to the extracted feature point map. In addition, it is possible to reduce the time required by arranging the Atros convolutions in parallel to extract feature points and then form a feature point map in a pyramid shape.

In addition, the error rate can be reduced and the accuracy can be improved by using the extended residual network backpropagation.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and is easily changed from the embodiments of the present invention by those of ordinary skill in the art to which the present invention belongs. It includes all changes to the extent deemed acceptable.

10: semantic division system
100: image input unit 200: segmented image acquisition unit
300: feature point extraction unit 310: extended residual network module
320: Atros Pyramid Pooling Module
400: judgment unit 500: storage unit

Claims (7)

A semantic segmentation image acquisition step (S10) of classifying and labeling objects from the input image to obtain a semantic segmented image;
Atrous convolution that maintains spatial information through various expansion ratios and extracts feature points with the same amount of computation, and feature point maps by selectively extracting only important features while preserving the features of the spatial domain for the entire image from the input image. A modified extended residual network (DRN, Dilated Residual Network) step of extracting a feature map from the image acquired from the image acquisition step S10 using the generated extended residual network (DRN, Dilated Residual Network) ( S20); And
In order to effectively extract an object with a small area occupied in the image, various Atrous convolutions are selectively applied according to the extracted feature point map, and finally, the feature point map extracted in the modified extended residual network step (S20) Atros Pyramid, which applies image pooling to the Atros convolutions in parallel, stacks the feature point maps extracted from the paralleled Atros convolutions in a pyramid shape, and extracts a 1-channel feature point map by applying 1x1 convolution. Semantic segmentation method applicable to AR including the Atrous Pyramid Pooling Module step (S30).

The method of claim 1,
A modified extended residual network backpropagation that compares the result extracted through the Atros pyramid pooling module step (S30) with the result image provided from a preset standard database, and corrects the weight to reduce the error rate by determining based on the comparison result. (Dilated Residual Network Backpropagation) Semantic segmentation method applicable to AR further comprising a step (S40).
The method of claim 1,
In the atros pyramid pulling module step (S30), the feature point map extracted in the atros pyramid pulling module step (S30) is stacked in a pyramid shape,
A semantic segmentation method applicable to AR, wherein the feature point map formed in a pyramid shape is formed as a one-channel feature point map by applying a 1x1 convolution.
The method of claim 3,
A semantic segmentation method applicable to AR, characterized in that the size of a feature point map having a size smaller than a preset reference value is equally adjusted through up-sampling.
delete An image input unit receiving image information;
A segmented image acquisition unit that classifies an object from the image information input through the image input unit and obtains a semantic segmentation image by labeling the classified object;
Atrous convolution that maintains spatial information through various expansion ratios and extracts feature points with the same amount of computation, and feature point maps by selectively extracting only important features while preserving the features of the spatial domain for the entire image from the input image. An extended residual network module for extracting a feature map from the image acquired through the segmented image acquisition unit using a generated extended residual network (DRN);
In order to effectively extract objects with a small area occupied in the image, various Atrous convolutions are selectively applied according to the feature point map extracted from the extended residual network module, and finally feature points extracted from the extended residual network module Atros convolution is placed in parallel by applying image pooling to the map, and the feature point maps extracted from the parallel placed Atros convolution are stacked in a pyramid shape, and a 1-channel feature point map is extracted by applying 1x1 convolution. Pyramid pooling module; And
A semantic segmentation system applicable to AR including a determination unit that compares the result extracted from the Atros pyramid pooling module with the result image provided from a preset standard database, and modifies the weight to reduce the error rate by determining based on the comparison result .
The method of claim 6,
The determination unit re-extracts a feature map of the image by modifying the weight of the Atrous convolution performed by the extended residual network module when an error rate greater than or equal to a preset error reference value occurs as a result of comparing the images. A semantic segmentation system applicable to the featured AR.

Images