KR102607748B1 - Apparatus and method for image analysis applying multi-task adaptation - Google Patents

Abstract

An image analysis device and method for extracting depth information and semantic segmentation information from unlabeled image data are disclosed. An image analysis device according to an embodiment includes an input unit that receives input image data; and an analysis unit that generates a depth map and a semantic segmentation map from input image data.

Description

Image analysis apparatus and method applying multi-task adaptation {Apparatus and method for image analysis applying multi-task adaptation}

It relates to an image analysis device and method for extracting depth information and semantic segmentation information from unlabeled image data.

Recently, as interest in autonomous driving has increased, various data sets have emerged that include different types of tasks and capture environments. However, collecting real-world data for autonomous driving requires expensive capture vehicles as well as high labeling costs. Additionally, there is a problem in covering large-scale real autonomous driving environments due to limitations in the size of the data set.

To overcome these problems, research is being conducted on Unsupervised Domain Adaptation (UDA) to achieve satisfactory performance over unlabeled domains by using labeled domains for the same task.

Korean Patent Publication No. 10-2169243 (2020.10.23)

The purpose is to provide an image analysis device and method for extracting depth information and semantic segmentation information from unlabeled image data.

According to one aspect, an image analysis device includes an input unit that receives input image data; and an analysis unit that generates a depth map and a semantic segmentation map from input image data.

The analysis unit includes an encoder unit that extracts feature information from input image data; A first bottleneck module that generates depth feature information based on the feature information of the encoder, a second bottleneck module that generates semantic segmentation feature information based on the feature information of the encoder, and reconstruction based on the feature information of the encoder. a bottleneck unit including a third bottleneck module that generates feature information; A first attention module for calculating the correlation (task correlation) between the depth feature information received from the bottleneck and the semantic segmentation feature information, and a second attention module for calculating the correlation between the depth feature information and the reconstruction feature information. an attention unit including an attention module and a third attention module for calculating a correlation between semantic segmentation feature information and reconstruction feature information; and a depth map decoder that generates a depth map using a combination of at least two of the feature information received from the bottleneck and the task correlations received from the attention section, and a semantic segmentation map decoder and depth that generate a semantic segmentation map. It may include a reconstruction unit including reconstruction decoders for reconstruction of each of the domain, semantic partition domain, and input image domain.

The depth map decoder can generate a depth map using the depth feature information received from the bottleneck, the correlation between the depth feature information and semantic segmentation feature information received from the attention section, and the correlation between depth feature information and reconstruction feature information. there is.

The semantic segmentation map decoder uses the semantic segmentation feature information received from the bottleneck, the correlation between the semantic segmentation feature information and depth feature information received from the attention unit, and the correlation between the semantic segmentation feature information and the reconstruction feature information. A semantic segmentation map can be created.

The analysis unit may be supervised learning using labeled depth domain training data, labeled semantic segmentation domain training data, and unlabeled input image domain training data. Here, supervised learning can be performed on the unlabeled input image domain through a reconstruction task.

The depth map decoder is supervised learning based on the labels of the generated depth map and depth domain training data, and the semantic segmentation map decoder is supervised learning based on the labels of the generated semantic segmentation map and semantic segmentation domain training data. Each of the reconstruction decoders for reconstructing the depth domain, semantic split domain, and input image domain may be supervised based on the reconstructed data, depth domain training data, semantic split domain training data, and input image domain training data. Here, the input image domain is unlabeled.

The analysis unit may be subjected to unsupervised learning based on unlabeled mixed learning data generated by combining depth domain learning data, semantic segmentation domain learning data, and input image domain learning data after supervised learning.

The mixed learning data is obtained by combining the depth mixed learning data generated by applying a depth mixing mask to the depth domain learning data and the semantic segmentation mixed learning data generated by applying the semantic segmentation mixing mask to the semantic segmentation domain learning data, and the remaining It can be created by combining input image domain learning data with a region.

In the case of areas where depth mixing training data generated by applying a depth mixing mask to depth domain training data and semantic segmentation mixing learning data generated by applying a semantic segmentation mixing mask to semantic segmentation domain training data overlap, depth mixing Among the depth of learning data and semantic segmentation mixed learning data, data with a closer depth can be selected and combined.

The analysis unit generates pseudo depth labels and pseudo-depth labels by inputting domain learning data, semantic segmentation domain learning data, and input image domain learning data into an Exponential Moving Average (EMA) model composed of the same model as the analysis unit. Unsupervised learning can be done based on the difference between the depth map and the semantic segmentation map generated by inputting mixed learning data into the semantic segmentation label and analysis unit.

According to one aspect, an image analysis method includes an input step of receiving input image data; And it may include an analysis step of generating a depth map and a semantic segmentation map from the input image data.

In the analysis stage, feature information is extracted from input image data using an encoder, depth feature information, semantic segmentation feature information, and reconstruction feature information are generated based on the encoder's features using a bottleneck module, and depth feature information is generated using an attention module. The correlation between feature information and semantic segmentation feature information (task correlation), the correlation between depth feature information and reconstruction feature information, and the correlation between semantic segmentation feature information and reconstruction feature information are calculated, and the feature information is collected using a decoder. and a combination of at least two of the task correlations can be used to generate a depth map, generate a semantic segmentation map, and reconstruct each of the depth domain, semantic segmentation domain, and input image domain.

A decoder for generating a depth map can generate a depth map using depth feature information, the correlation between depth feature information and semantic segmentation feature information, and the correlation between depth feature information and reconstruction feature information.

The decoder for generating a semantic segmentation map uses semantic segmentation feature information, the correlation between semantic segmentation feature information and depth feature information received from the attention unit, and the correlation between semantic segmentation feature information and reconstruction feature information. A segmentation map can be created.

The analysis step may be supervised learning using labeled depth domain training data, labeled semantic segmentation domain training data, and unlabeled input image domain training data.

The decoder for generating a depth map is supervised based on the labels of the generated depth map and depth domain training data, and the decoder for generating a semantic segmentation map is trained based on the labels of the generated semantic segmentation map and semantic segmentation domain training data. Based on the supervised learning, each reconstruction decoder for reconstruction of the depth domain, semantic segmentation domain, and input image domain is based on the reconstructed data, depth domain training data, semantic segmentation domain training data, and input image domain training data. It can be supervised learning.

The analysis unit stage may be unsupervised learning based on unlabeled mixed learning data generated by combining depth domain learning data, semantic segmentation domain learning data, and input image domain learning data after supervised learning.

The mixed learning data is obtained by combining the depth mixed learning data generated by applying a depth mixing mask to the depth domain learning data and the semantic segmentation mixed learning data generated by applying the semantic segmentation mixing mask to the semantic segmentation domain learning data, and the remaining It can be created by combining input image domain learning data with a region.

In the case of areas where depth mixing training data generated by applying a depth mixing mask to depth domain training data and semantic segmentation mixing learning data generated by applying a semantic segmentation mixing mask to semantic segmentation domain training data overlap, depth mixing Among the depth of learning data and semantic segmentation mixed learning data, data with a closer depth can be selected and combined.

The analysis step is a pseudo depth label and It can be unsupervised based on the difference between the depth map and the semantic segmentation map generated by inputting mixed learning data into the pseudo-semantic segmentation label and analysis unit.

According to one embodiment, depth information and semantic segmentation information can be extracted from unlabeled image data.

1 is a configuration diagram of an image analysis device according to an embodiment.
Figure 2 is a configuration diagram of an analysis unit according to an embodiment.
Figure 3 is an example diagram for explaining a learning method of an image analysis device according to an embodiment.
Figure 4 is an example diagram for explaining a method of generating mixed learning data according to an example.
Figure 5 is a flowchart illustrating an image analysis method according to an embodiment.
6 is a block diagram illustrating and illustrating a computing environment including a computing device suitable for use in example embodiments.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings. In describing the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the content throughout this specification.

Hereinafter, embodiments of the image analysis device and method will be described in detail with reference to the drawings.

1 is a configuration diagram of an image analysis device according to an embodiment.

According to one embodiment, the image analysis device 100 includes an input unit 110 that receives input image data and an analysis unit that generates a depth map and a semantic segmentation map from the input image data ( 120) may be included.

As an example, the image analysis device 100 may be trained using three types of domains consisting of two virtual source domains and one actual target domain. For example, the two virtual source domains could be labeled image data for semantic segmentation (Ds) and labeled image data for depth (Dd), respectively, and the real target domain could be the real environment without label information. It may be image data (Dt) captured from . Accordingly, the image analysis device 100 will be trained to perform multi-task prediction on unlabeled actual target data using labeled data for semantic segmentation and data labeled for depth. You can. For example, the image analysis device 100 may generate a depth map and a semantic segmentation map through multi-task prediction.

Figure 2 is a configuration diagram of an analysis unit according to an embodiment.

According to one embodiment, the analysis unit 120 may include an encoder unit 121 that includes an encoder that extracts feature information from input image data.

According to one embodiment, the analysis unit 120 includes a first bottleneck module (bottleneck 1) that generates depth feature information based on the feature information of the encoder, and a second bottleneck module that generates semantic segmentation feature information based on the feature information of the encoder. It may include a bottleneck unit 123 that includes two bottleneck modules (bottleneck 2) and a third bottleneck module (bottleneck 3) that generates reconstructed feature information based on feature information of the encoder.

According to one example, the bottleneck module may generate task-specific features for depth estimation, semantic segmentation, and reconstruction.

As an example, the first bottleneck module may receive feature information from an encoder and generate depth feature information.

According to one example, depth feature information that is the output of the first bottleneck module may be transmitted to an initial depth decoder, and the initial depth decoder may generate an initial depth map. As an example, the depth map decoder included in the reconstruction unit 127 may be trained using a loss function calculated based on the depth map generated by the depth map decoder and the initial depth map generated by the initial depth decoder.

As an example, the second bottleneck module may receive feature information from the encoder and generate semantic segmentation feature information.

According to one example, the semantic segmentation feature information that is the output of the second bottleneck module may be transmitted to an initial semantic segmentation map decoder (init semantic decoder), and the initial semantic segmentation decoder may generate an initial semantic segmentation map. there is. As an example, the semantic segmentation map decoder included in the reconstruction unit 127 calculates a loss based on the semantic segmentation map generated by the semantic segmentation map decoder and the initial semantic segmentation map generated by the initial semantic segmentation map decoder. It can be learned using functions.

As an example, the third bottleneck module may receive feature information from the encoder and generate reconstructed feature information.

According to one example, the depth feature information that is the output of the third bottleneck module is an initial depth domain reconstruction decoder (DD Init Reconstruction Decoder), an initial semantic segmentation domain reconstruction decoder (SD Init Reconstruction Decoder), and an initial input image domain reconstruction decoder (UD). Init Reconstruction Decoder).

For example, the configuration and architecture of the initial decoder may be the same as the decoder included in the reconstruction unit 127.

According to one embodiment, the analysis unit 120 includes a first attention module (attention 1) for calculating a correlation (task correlation) between the depth feature information received from the bottleneck 123 and the semantic segmentation feature information, a depth Attention including a second attention module (attention 2) for calculating the correlation between feature information and reconstructed feature information and a third attention module (attention 3) for calculating the correlation between semantic segmentation feature information and reconstructed feature information. It may include unit 125.

According to one example, task-specific features of the bottleneck module are provided to the attention module, which can be used as input to the decoder of the reconstruction unit. As an example, the attention module may be constructed using a self-attention block in which two vectors are multiplied together.

According to one embodiment, the analysis unit 120 generates a depth map using a combination of at least two of the feature information received from the bottleneck unit 123 and the task correlations received from the attention unit 125. A depth map decoder, a semantic segmentation map decoder that generates a semantic segmentation map, and a depth domain, a semantic segment domain, and an input image domain, respectively. It may include a reconstruction unit 127 including reconstruction decoders for reconstruction.

According to one example, the five decoders included in the reconstruction unit 127 may be composed of one depth map decoder, one split decoder, and three domain-unit reconstruction decoders for input data Dd, Ds, and Dt, respectively. As an example, the depth and semantic segmentation map decoder can be built with four expansion convolutional layers with expansion and padding series set to [6, 12, 18, 24]. For the input image data (I), the outputs of the depth map decoder, semantic segmentation map decoder, and reconstruction decoder for Dd, Ds, and Dt, respectively, are expressed as D(I), S(I), Rd(I), and Rs(I). and Rt(I).

According to one embodiment, the depth map decoder uses the depth feature information received from the bottleneck unit, the correlation between the depth feature information received from the attention unit and the semantic segmentation feature information, and the correlation between the depth feature information and the reconstruction feature information. A depth map can be created.

Referring to Figure 2, the depth map decoder can receive depth feature information from the first bottleneck module of the bottleneck unit 123, and the correlation and depth between depth feature information and semantic segmentation feature information from the attention unit 125. The correlation between feature information and reconstructed feature information can be received. Afterwards, the depth map decoder can generate a depth map using the input information.

According to one embodiment, the semantic segmentation map decoder determines the semantic segmentation feature information received from the bottleneck unit, the correlation between the semantic segmentation feature information and the depth feature information received from the attention unit, and the semantic segmentation feature information and reconstruction feature information. A semantic segmentation map can be created using the correlation of .

Referring to FIG. 2, the semantic segmentation map decoder may receive semantic segmentation feature information from the second bottleneck module of the bottleneck unit 123, and may receive semantic segmentation feature information and depth feature information from the attention unit 125. Correlation and correlation between semantic segmentation feature information and reconstruction feature information may be received. Afterwards, the semantic segmentation map decoder can generate a semantic segmentation map using the input information.

According to one example, the training loss for the analysis unit 120 may be the sum of two training losses divided into supervision loss and adaptation loss. The supervised loss is designed to train the network using the original input images and the corresponding answer labels, and the adaptive loss can be designed to perform integrated training to reduce the domain gap using mixed learning data (TripleMix).

According to one embodiment, the analysis unit 120 may be supervised using labeled depth domain training data, labeled semantic segmentation domain training data, and unlabeled input image domain training data. .

For example, the analysis unit 120 may supervised learning of a depth map decoder and a depth domain reconstruction decoder using labeled depth domain learning data (Dd).

As an example, a depth map decoder may be supervised based on the labels of the generated depth map and depth domain learning data. For example, a depth map decoder can generate a depth map for depth domain training data and can be trained by comparing the generated depth map with the labels of the depth domain training data.

As an example, the depth domain reconstruction decoder can be learned using depth domain learning data (Dd). For example, a depth domain reconstruction decoder may generate depth domain reconstruction data for depth domain training data, and may be trained by comparing the generated depth domain reconstruction data with the depth domain training data itself.

For example, the analysis unit 120 may learn a semantic segmentation map decoder and a semantic segmentation domain reconstruction decoder using labeled semantic segmentation domain learning data.

As an example, a semantic segmentation map decoder may be supervised based on the labels of the generated semantic segmentation map and semantic segmentation domain learning data. For example, if a semantic segmentation map decoder can generate a semantic segmentation map for semantic segmentation domain training data, it can be trained by comparing the generated semantic segmentation map with the labels of the semantic segmentation domain training data. .

As an example, a semantic split domain reconstruction decoder may be supervised learning based on semantic split domain learning data. For example, a semantic split domain reconstruction decoder can generate semantic split reconstruction data for semantic split domain training data, and can be learned by comparing the generated semantic split domain training data with the semantic split domain training data itself. You can.

According to one example, each of the reconstruction decoders for reconstruction of the depth domain, semantic segmentation domain, and input image domain is guided based on the reconstructed data and the depth domain training data, semantic segmentation domain training data, and input image domain training data. It can be learned. For example, each reconstruction decoder may generate reconstruction data corresponding to each input domain data, and may be learned by comparing the generated reconstruction data and the input domain data, respectively.

According to one embodiment, the analysis unit 120 performs unsupervised training based on unlabeled mixed learning data generated by combining depth domain learning data, semantic segmentation domain learning data, and input image domain learning data after supervised learning. It can be learned.

According to one embodiment, the mixed learning data includes depth mixed learning data generated by applying a depth mixing mask to depth domain learning data, and semantic segmentation mixed learning generated by applying a semantic segmentation mixing mask to semantic segmentation domain learning data. After combining the data, it can be generated by combining the input image domain learning data with the remaining areas.

As an example, the mixed learning data 315 may be created by mixing three images, Is, Id, and It, respectively extracted from the training sets 311 of Ds, Dd, and Dt. For this purpose, masks for Is and Id that determine the area to be extracted can be estimated, and the remaining areas that do not correspond to the masks for Is and Id can be filled with pixels of It. Masks for Is and Id can be denoted as Ms and Md, respectively.

Referring to FIG. 4, mixed learning data 420 is depth mixed learning data generated by applying a depth domain mixing mask 411 to depth domain learning data, and a semantic segmentation mixing mask 415 to semantic segmentation domain learning data. It can be generated by combining the semantic segmentation mixed learning data generated by applying and the input image mixed learning data generated by applying the input image mixing mask 413 to the input image domain learning data.

As an example, to handle the long tail problem, the number of pixels for each class of Is can be divided by the total number of pixels as shown in the equation below.

[Equation 1]

Here, c ∈ {1, ..., C} is the class index for category C, n _c refers to the number of pixels labeled with the c-th class, and N refers to the total number of pixels.

As an example, in order to estimate the initial mixing mask (M _o ^d ) for depth, continuous depth values can be separated into designated ranges to create individual pixel-specific categories for depth estimation. For example, the threshold can be fixed to [θ1,θ2,θ3,...,θNd] to determine depth categories without overlapping ranges. Here, Nd represents the number of depth categories, and a threshold can be determined to ensure that pixels are distributed uniformly across all categories. Afterwards, half of the Nd depth categories can be randomly sampled, assuming a uniform random distribution, and then M _o ^d can be set to have a value of 1 only in pixels of the selected depth category, and 0 otherwise.

According to one embodiment, depth mixed learning data generated by applying a depth mixing mask to depth domain learning data and semantic segmentation mixed learning data generated by applying a semantic segmentation mixing mask to semantic segmentation domain learning data are overlapping. In the case of a region, data with a closer depth among the depth of the depth mixed learning data and the depth of the semantic segmentation mixed learning data can be selected and combined.

According to one example, the overlapping region between M _o ^s and M _o ^d must be removed before mixing I d and I s. For example, a depth map predicted from an Exponential Moving Average (320) model can be used to consider geometric consistency between Is and Id. For example, close objects may need to obscure distant areas, so nearby image pixels may be selected for overlapping pixels of M _o ^s and M _o ^d . Therefore, after replicating M _o ^s and M _o ^d for each of the overlapping pixel positions x, the refined mixing masks Ms and M d can be estimated as shown in the equation below.

[Equation 2]

Here GTd(x) represents the correct depth value for Id at location x.

As an example, mixed learning data can be generated using Ms and Md. The areas of Is and Id can be extracted respectively according to Ms and Md, and after mixing Is and Id, the remaining area can be extracted from It to fill the empty area. Accordingly, the mixed learning data (TripleMix) image I+ can be generated as shown in the equation below.

[Equation 3]

here, represents pixel-wise multiplication.

As an example, the pseudo label of I+ can be obtained using a ground truth map and predictions of an Exponential Moving Average (EMA) model. For example, a ground-truth segmentation map and a ground-truth depth map can be defined by GT and GTd, respectively, and semantic segmentation (S+) and depth estimation (D+ ) The pseudo label of I+ can be obtained as in the equation below.

[Equation 4]

For post-processing, additional prior knowledge can be used to modify the pseudo-labels of I+. For example, since the depth of a pixel designated as 'sky' must be infinite, the D+ value of a pixel designated as 'sky' pixel in S+ may be set to an infinite value. Therefore, if the depth value of D+ is infinite at that location, the pixel class of S+ can be changed to the 'sky' class.

According to one embodiment, the analysis unit inputs domain learning data, semantic segmentation domain learning data, and input image domain learning data into an Exponential Moving Average (EMA) model composed of the same model as the analysis unit, and generates a doctor ( pseudo) Depth label and pseudo-semantic segmentation Label and pseudo-semantic segmentation can be unsupervised based on the difference between the depth map and the semantic segmentation map generated by inputting mixed learning data into the analysis unit.

As an example, before estimating the training loss, the pixel-level confidence for similar labels in the mixed learning data (TripleMix) image can be predicted. The reliability of a mixed learning data (TripleMix) image can be estimated by multiplying the uncertainty score of the pseudo label with the disparity map of the reconstructed image.

According to one example, the outputs of the three reconstruction modules should be identical to the mixed learning data (TripleMix) image if the embedding features are well integrated in the backbone network, even if trained separately in different domains. Based on this, the inconsistencies in the reconstructed images can be utilized in each of the three reconstruction modules. For example, based on the Euclidean distance between reconstructed images, a segmentation task disparity map for unclassified regions and a depth task disparity map for unlabeled depth regions can be obtained. This can be expressed as follows.

[Equation 5]

here, and represents a clipping operation to limit the input value to 0 to 1.

According to one example, the maximum probability can be utilized to determine the uncertainty score of the split pseudo label in Id and It. For example, the number of pixels for which the maximum probability for semantic segmentation exceeds a predefined threshold can be calculated using the equation below.

[Equation 6]

Here, ζs represents a user-defined hyperparameter.

As an example, the uncertainty score of depth estimation can be estimated by augmented variance. Based on the intuition that the uncertain prediction will be unstable for the augmented input image, the uncertainty score for the depth pseudo label can be estimated as shown in the equation below.

[Equation 7]

Here, the output of the σ function is the variance of the input vector, Ij is the augmented image of I with j ∈ {1, ..., J}, and ζd represents the user-defined hyperparameter.

According to one example, from the perspective of the labeled domain, the other two domains are unlabeled domains. Accordingly, the reliability of similar labels in two unlabeled domains can be calculated. For example, Bs and Bd can be obtained as confidence maps for semantic segmentation and depth tasks, respectively. For example, Bs and Bd can be calculated by multiplying the estimated discrepancy map and uncertainty score as shown in the equation below.

[Equation 8]

According to one example, the analysis unit can perform three tasks: semantic segmentation, depth estimation, and region-specific reconstruction. First, for semantic segmentation, the cross entropy loss can be calculated as shown in the equation below.

[Equation 9]

Here S and represents the target and predicted segmentation maps, respectively, B is the weighted map for the expectations, and X represents the set containing all positions of the input image.

As an example, the depth estimation loss can be calculated as the difference between the actual depth and the predicted depth map as shown in the equation below.

[Equation 10]

Here with D are the target and predicted depth maps, respectively, represents the inverted Huber norm.

As an example, the reconstruction loss can be expressed as the equation below using the Frobenius norm.

[Equation 11]

where I and refers to the target and reconstructed image, respectively, represents the Frobenius norm.

As an example, the total training loss can be expressed as supervision loss L _su and adaptation loss L _da as shown in the equation below.

[Equation 12]

As an example, the total loss can be used to update the entire network architecture, including the backbone network.

For example, in supervised learning, the loss of each domain can be calculated from the corresponding labeled task. To consider the learning of the initial task module together, the supervised loss can be defined as in the equation below.

[Equation 13]

Here, 1 represents a map that evaluates only to 1 of that size.

As an example, for domain adaptation, L _da can be set to learn reliable pseudo labels of mixed learning data (TripleMix) images based on the reliability of pseudo labels. For example, the learning loss for the mixed learning data (TripleMix) image can be calculated by considering the pseudo label as the target map as shown in the equation below.

[Equation 14]

Figure 5 is a flowchart showing an image analysis method according to an embodiment.

According to one embodiment, the image analysis device can receive input image data (510) and generate a depth map and a semantic segmentation map from the input image data (510). 520).

According to one embodiment, the image analysis device extracts feature information from input image data using an encoder, and generates depth feature information, semantic segmentation feature information, and reconstruction feature information based on the features of the encoder using a bottleneck module. , Using the attention module, the correlation between depth feature information and semantic segmentation feature information (task correlation), the correlation between depth feature information and reconstruction feature information, and the correlation between semantic segmentation feature information and reconstruction feature information are calculated, Using a decoder, it is possible to generate a depth map, generate a semantic segmentation map, and reconstruct each of the depth domain, semantic segmentation domain, and input image domain using a combination of at least two of the feature information and task correlations.

In the description of the embodiment of FIG. 5, descriptions that overlap with those described with reference to FIGS. 1 to 4 will be omitted.

FIG. 6 is a block diagram illustrating and illustrating a computing environment 10 including computing devices suitable for use in example embodiments. In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

The illustrated computing environment 10 includes a computing device 12 . In one embodiment, computing device 12 may be image analysis device 100.

Computing device 12 includes at least one processor 14, a computer-readable storage medium 16, and a communication bus 18. Processor 14 may cause computing device 12 to operate in accordance with the example embodiments noted above. For example, processor 14 may execute one or more programs stored on computer-readable storage medium 16. The one or more programs may include one or more computer-executable instructions, which, when executed by the processor 14, cause computing device 12 to perform operations according to example embodiments. It can be.

Computer-readable storage medium 16 is configured to store computer-executable instructions or program code, program data, and/or other suitable form of information. The program 20 stored in the computer-readable storage medium 16 includes a set of instructions executable by the processor 14. In one embodiment, computer-readable storage medium 16 includes memory (volatile memory, such as random access memory, non-volatile memory, or an appropriate combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash It may be memory devices, another form of storage medium that can be accessed by computing device 12 and store desired information, or a suitable combination thereof.

Communication bus 18 interconnects various other components of computing device 12, including processor 14 and computer-readable storage medium 16.

Computing device 12 may also include one or more input/output interfaces 22 and one or more network communication interfaces 26 that provide an interface for one or more input/output devices 24. The input/output interface 22 and the network communication interface 26 are connected to the communication bus 18. Input/output device 24 may be coupled to other components of computing device 12 through input/output interface 22. Exemplary input/output devices 24 include, but are not limited to, a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touch screen), a voice or sound input device, various types of sensor devices, and/or imaging devices. It may include input devices and/or output devices such as display devices, printers, speakers, and/or network cards. The exemplary input/output device 24 may be included within the computing device 12 as a component constituting the computing device 12, or may be connected to the computing device 12 as a separate device distinct from the computing device 12. It may be possible.

So far, the present invention has been examined focusing on its preferred embodiments. A person skilled in the art to which the present invention pertains will understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Accordingly, the scope of the present invention is not limited to the above-described embodiments, but should be construed to include various embodiments within the scope equivalent to the content described in the patent claims.

100: video analysis device
110: input unit
120: analysis department
121: Encoder unit
123: bottleneck
125: Attention unit
127: Reconstruction unit

Claims (20)

an input unit that receives input image data; and
An analysis unit that generates a depth map and a semantic segmentation map from the input image data,
The analysis unit,
an encoder unit including an encoder that extracts feature information from the input image data;
A first bottleneck module that generates depth feature information based on the feature information of the encoder, a second bottleneck module that generates semantic segmentation feature information based on the feature information of the encoder, and feature information of the encoder a bottleneck unit including a third bottleneck module that generates reconstruction feature information on a basis;
A first attention module for calculating the correlation (task correlation) between the depth feature information received from the bottleneck and the semantic segmentation feature information, and a first attention module for calculating the correlation between the depth feature information and the reconstruction feature information. an attention unit including two attention modules and a third attention module for calculating a correlation between semantic segmentation feature information and reconstruction feature information; and
A depth map decoder that generates a depth map using a combination of at least two of the feature information received from the bottleneck unit and the task correlations received from the attention unit, a semantic segmentation map decoder that generates a semantic segmentation map, and It includes a reconstruction unit including reconstruction decoders for reconstruction of each of the depth domain, semantic segmentation domain, and input image domain,
The analysis unit,
Supervised learning is performed using labeled depth domain training data, labeled semantic segmentation domain training data, and unlabeled input image domain training data, and after supervised learning, depth domain training data and semantic segmentation domain training are performed. Unsupervised learning is performed based on unlabeled mixed learning data generated by combining data and input image domain learning data.
The mixed learning data is,
Depth blended learning data and semantic segmentation created by applying a depth blending mask to determine the extraction region to the depth domain training data. Semantic segmentation created by applying a semantic segmentation blending mask to determine the extraction region to the domain training data. An image analysis device generated by combining mixed learning data and then combining input image domain learning data with remaining areas that do not correspond to the depth mixing mask and the semantic segmentation mixing mask.
delete According to claim 1,
The depth map decoder is
Image analysis that generates a depth map using depth feature information received from the bottleneck, correlation between depth feature information and semantic segmentation feature information received from the attention portion, and correlation between depth feature information and reconstruction feature information. Device.
According to claim 1,
The semantic segmentation map decoder is
A semantic segmentation map is created using the semantic segmentation feature information received from the bottleneck, the correlation between the semantic segmentation feature information and the depth feature information received from the attention unit, and the correlation between the semantic segmentation feature information and the reconstruction feature information. A video analysis device that generates.
delete According to claim 1,
The depth map decoder is supervised based on the generated depth map and the labels of the depth domain learning data,
A semantic segmentation map decoder is supervised based on the generated semantic segmentation map and the labels of the semantic segmentation domain training data,
Image analysis, where each reconstruction decoder for reconstruction of the depth domain, semantic segmentation domain, and input image domain is supervised based on the reconstructed data, depth domain training data, semantic segmentation domain training data, and input image domain training data. Device.
delete delete According to claim 1,
In the case of areas where depth mixing training data generated by applying a depth mixing mask to depth domain training data and semantic segmentation mixing learning data generated by applying a semantic segmentation mixing mask to semantic segmentation domain training data overlap, depth mixing An image analysis device that selects and combines data with a closer depth among the depth of learning data and semantic segmentation mixed learning data.
According to claim 1,
The analysis department
Pseudo depth label and pseudo meaning generated by inputting domain learning data, semantic segmentation domain learning data, and input image domain learning data into an Exponential Moving Average (EMA) model composed of the same model as the analysis unit. logical segmentation labels and
An image analysis device that performs unsupervised learning based on the difference between a depth map and a semantic segmentation map generated by inputting mixed learning data into the analysis unit.
An input step of receiving input image data; and
An analysis step of generating a depth map and a semantic segmentation map from the input image data,
The analysis step is
Feature information is extracted from the input image data using an encoder,
Using a bottleneck module, depth feature information, semantic segmentation feature information, and reconstruction feature information are generated based on the features of the encoder,
Using the attention module, the correlation between depth feature information and semantic segmentation feature information (task correlation), the correlation between depth feature information and reconstruction feature information, and the correlation between semantic segmentation feature information and reconstruction feature information are calculated;
Using a decoder, generate a depth map, generate a semantic segmentation map, and reconstruct each of the depth domain, semantic segmentation domain, and input image domain using a combination of at least two of the feature information and task correlations,
The analysis step is,
Supervised learning is performed using labeled depth domain training data, labeled semantic segmentation domain training data, and unlabeled input image domain training data, and after supervised learning, depth domain training data and semantic segmentation domain training are performed. Unsupervised learning is performed based on unlabeled mixed learning data generated by combining data and input image domain learning data.
The mixed learning data is,
Depth blended learning data and semantic segmentation created by applying a depth blending mask to determine the extraction region to the depth domain training data. Semantic segmentation created by applying a semantic segmentation blending mask to determine the extraction region to the domain training data. An image analysis method generated by combining mixed learning data and then combining input image domain learning data with the remaining regions that do not correspond to the depth mixing mask and the semantic segmentation mixing mask.
delete According to claim 11,
The decoder for generating the depth map is
An image analysis method that generates a depth map using depth feature information, the correlation between depth feature information and semantic segmentation feature information, and the correlation between depth feature information and reconstruction feature information.
According to claim 11,
The decoder for generating the semantic segmentation map is
An image analysis method that generates a semantic segmentation map using semantic segmentation feature information, the correlation between semantic segmentation feature information and depth feature information received from the attention unit, and the correlation between semantic segmentation feature information and reconstruction feature information. .
delete According to claim 11,
The decoder for generating the depth map is supervised based on the generated depth map and the labels of the depth domain learning data,
A decoder for generating a semantic segmentation map is supervised based on the generated semantic segmentation map and the labels of the semantic segmentation domain learning data,
Image analysis, where each reconstruction decoder for reconstruction of the depth domain, semantic segmentation domain, and input image domain is supervised based on the reconstructed data, depth domain training data, semantic segmentation domain training data, and input image domain training data. method.
delete delete According to claim 11,
In the case of areas where depth mixing training data generated by applying a depth mixing mask to depth domain training data and semantic segmentation mixing learning data generated by applying a semantic segmentation mixing mask to semantic segmentation domain training data overlap, depth mixing An image analysis method that selects and combines data with a closer depth among the depth of learning data and semantic segmentation mixed learning data.
According to claim 11,
The analysis step is
Pseudo depth labels and pseudo-semantic data are generated by inputting domain training data, semantic segmentation domain training data, and input image domain training data into an Exponential Moving Average (EMA) model composed of the same model as the analysis model. split label and
An image analysis method in which unsupervised learning is performed based on the difference between a depth map and a semantic segmentation map generated by inputting mixed learning data into the analysis model.