CN110363777A - A Sea Image Semantic Segmentation Method Based on Reducible Spatial Constrained Mixture Model - Google Patents

Abstract

The sea image semantic segmentation method based on reducible space constraint mixed model that the invention discloses a kind of, comprising the following steps: (1) input sea color image to be detected；(2) assume that there are sky, seashore/haze, three main semantic regions of seawater and potential barrier regions for sea image, and establish the mixed model of space constraint with this；(3) space constraint mixed model is optimized using expectation-maximization algorithm (EM)；(4) sky, seashore/haze classification Gaussian Profile KL distance (Kullback-Leibler divergence) are calculated, if KL distance is less than the threshold value of setting, abbreviation is carried out to space constraint mixed model；(5) sea image semantic segmentation result is exported.Method of the invention effectively can carry out semantic segmentation to sea image, and have the characteristics that speed is fast, robustness is good.

Description

A Semantic Segmentation of Sea Surface Image Based on Reducible Spatial Constrained Mixture Model method

technical field

The invention relates to the technical field of image processing, in particular to a sea surface image semantic segmentation method based on a simplified space-constrained mixed model.

Background technique

Image semantic segmentation is a basic technology for computer vision understanding. Its task is to segment different objects in the image from the perspective of pixels and semantically label each pixel (that is, classify). Applying semantic segmentation technology to sea surface images can help enhance the perception of the surrounding environment of unmanned surface vehicles, thereby ensuring their safe operations.

In recent years, with the rapid development of deep learning, the semantic segmentation method based on convolutional neural network has been widely researched and applied in the fields of unmanned driving and medical image analysis. However, there are few related studies on semantic segmentation of sea surface images using convolutional neural networks. The main reason is the lack of a large amount of sea image annotation data. In 2016, Kristan et al. proposed a semantic segmentation method for sea surfaces based on clustering ideas in "Fast Image-Based Obstacle Detection From Unmanned Surface Vehicles". This method defines a spatially constrained mixture model to model the pixel features of sea surface images. Specifically, the method uses three Gaussian distributions and one uniform distribution to describe the sky region, mid-coast/haze mixed region, seawater region and potential obstacle region (singular value region) of the sea surface image respectively, and through The expectation maximization algorithm (EM algorithm) is used to realize the semantic segmentation of sea surface images. This method has better detection performance and faster speed; however, by observing the sea surface images taken by the unmanned surface vehicle, it can be found that when the unmanned surface vehicle is driving away from the coast, the sea surface image usually only has the sky area, sea water area and potential obstacles There is no middle coast/haze mixed area assumed by Kristan et al. Therefore, the semantic segmentation result of the method proposed by Kristan et al. has a large error in this case.

Contents of the invention

Aiming at the deficiencies of the prior art, the present invention provides a method for semantic segmentation of sea surface images based on a simplified space-constrained hybrid model, which can effectively perform semantic segmentation of sea surface images, and has the characteristics of fast speed and good robustness .

To achieve the above object, the present invention adopts the following technical solutions:

A method for semantic segmentation of sea surface images based on a simplified spatially constrained mixture model, comprising the following steps:

(1) Input the color image of the sea surface to be detected;

(2) Assume that there are three main semantic areas of sky, coast/haze, and sea water in the sea surface image, as well as potential obstacle areas, and establish a hybrid model of spatial constraints;

(3) Optimizing the space-constrained hybrid model by using the Expectation-Maximization Algorithm (EM);

(4) Calculate the KL distance (Kullback-Leibler divergence) of the Gaussian distribution of the sky, coast/haze category. If the KL distance is less than the set threshold, the spatially constrained hybrid model is simplified;

(5) Output the semantic segmentation results of sea surface images.

Further, in the step (2), it is assumed that the mixed model consists of three Gaussian distributions and one uniform distribution, wherein the three Gaussian distributions are used to describe the sky, haze/coastal mixed area, and seawater area respectively, and the uniform distribution Used to describe potential obstacle regions (singular value regions). Then, the probability of the feature vector y _i of the i-th pixel in the image can be expressed as:

In the above formula, N(·|m,C) represents a Gaussian distribution function with mean m and covariance C, and U(·)=ε represents a uniform distribution function (where ε is a very small positive hyperparameter) ; y _i represents the feature vector of the i-th pixel in the image (also known as observation data), which is mainly composed of the color features (c1, c2, c3) and coordinates (r, c) of the pixel; θ represents all Gaussian distributions in the model The parameters of (ie θ={m _k ,C _k } _k=1,2,3 ); π represents the class prior distribution of all pixels in the image (ie π={π _i } _i=1:M , where, M is the number of pixels in the image), π _i represents the category prior distribution of the i-th pixel (that is, π _i =[π _i1 ,…,π _ik ,…,π _i4 ], where π _ik =p( _xi =k) represents the probability when the category x _i of the i-th pixel is k, assuming that k=1 represents the sky category, k=2 represents the middle coast/haze mixed category, k=3 represents the seawater category, and k=4 represents obstacles object category);

In order to overcome the adverse effects of local noise in the sea surface image on image segmentation, a Markov Random Field (MRF) is introduced to place space constraints on the mixture model, that is, it is assumed that the class prior distribution of all pixels in the image is π = {π _i } _i=1:M and the posterior distribution P={p _i } _i=1:M is an MRF about the neighborhood system. According to the Besag method, the joint probability distribution of the prior distribution π can be approximated as:

In the above formula, N _i is the neighborhood of pixel i, is the category prior distribution of the neighborhood N _i :

Wherein, λ _ij is a fixed positive weight, when the distance between pixel j and pixel i is smaller, λ _ij is larger, and Σ _j λ _ij =1.

Furthermore, the potential energy function in the MRF (i.e. can be defined as:

In the above formula, is the KL divergence item, and H(π _i ) is the entropy item.

And the joint probability distribution of the posterior distribution P={p _i } _i=1:M is:

Wherein, the calculation formula of the posterior distribution p _i ={p _ik } _k=1:4 of pixel i is as follows:

Combining formulas (1), (2), (4) and (5), the joint probability density function of the semantic segmentation model based on Gaussian and uniform mixed distribution can be obtained:

In the above formula, because and There is a coupling relationship in , so it is difficult to estimate the model parameters directly. In order to solve this problem, the auxiliary class prior distribution set s={s _i } _i=1:M and the auxiliary posterior distribution set q={q _i } _i=1:M can be introduced into the above formula, and the equation The natural logarithm operation is taken on both sides at the same time, so as to obtain the penalty log likelihood function of the space-constrained mixed model:

In the above formula, ° represents the Hadamard product operation; and when s _i ≡ π _i and q _i ≡ p _i , it can be simplified to formula (7). In addition, according to the maximum a posteriori criterion, the above formula can be maximized by the EM algorithm, thereby realizing the optimization of the mixture model.

Further, in the step (3), the specific steps of expectation maximization are:

①Initialize the normal distribution parameter set θ＝{m _k ,C _k } _k＝1,2,3 . Divide the sea surface image into three regions from top to bottom according to the proportion {0,0.3}, {0.3,0.5} and {0.5,1}, and then calculate the mean value m of the sky category according to the characteristics of the pixels in these three regions ₁ and covariance C ₁ , mean m ₂ and covariance C ₂ for the Mid Coast/Haze mixed category, mean m ₃ and covariance C ₃ for the seawater category;

② Initialize the class prior distribution of all pixels π={π _i } _i=1:M . For the class prior distribution π _i of each pixel, its initialization formula is as follows:

In the above formula, ε is a very small positive hyperparameter.

In E-step:

③ Substitute θ and π into formula (6) to calculate the posterior distribution P={p _i } _i=1:M of all pixels.

④ According to the following formula, calculate the auxiliary category prior distribution s={s _i } _i=1:M of all pixels;

In the above formula, ° means Hadamard product operation, * means convolution operation, is the normalization constant.

⑤ Calculate the auxiliary posterior distribution q={q _i } _i=1:M of all pixels, the calculation formula is as follows:

In the above formula, is the normalization constant.

In M-step:

⑥Update the category prior distribution set, the calculation formula is as follows:

⑦Update Gaussian distribution parameters, the calculation formula is as follows:

⑧Judge whether the EM algorithm reaches the iteration termination condition; if so, stop the iteration, otherwise, continue ③～⑧. Among them, the iteration termination condition is as follows:

Further, in the step (4), since the unmanned surface vehicle sails towards the coast, the sea surface image captured by the boat-mounted camera has three main semantic areas of the sky, the middle coast/haze and sea water from top to bottom; however, When the unmanned surface vehicle sails away from the coast, there are only two main semantic areas of the sky and sea water in the sea surface image captured by the boat-mounted camera from top to bottom. Therefore, when there are only two main semantic regions of the sky and sea water in the sea surface image, the spatially constrained mixture model assumed in step (2) needs to be simplified. The specific steps are:

① On the basis of step (3), calculate the KL distance (Kullback-Leibler divergence) of the Gaussian distribution of the sky category after EM and the mixed category of Mid Coast/Haze:

dst＝KL(N ₁ ||N ₂ ) (16)

In the above formula, N ₁ represents the Gaussian distribution N(·|m ₁ ,C ₁ ) of the sky category, and N ₂ represents the Gaussian distribution N(·|m ₂ ,C ₂ ) of the mid-coast/haze mixed category.

② If the KL distance dst is less than the preset fixed threshold T, then simplify the hybrid model; otherwise, directly perform step (5). Specifically, when dst<T, the Gaussian distribution of the sky category and the mid-coast/haze mixed category is fused to form a new sky category Gaussian distribution:

In the above formula, m ₁ and C ₁ are the Gaussian parameters of the sky category after EM optimization, and m ₂ and C ₂ are the Gaussian parameters of the coast/haze mixed category after EM optimization.

Therefore, the mixture model is simplified to consist of 2 Gaussian distributions and 1 uniform distribution, where 2 Gaussian distributions are used to describe the sky and sea water areas, and the uniform distribution is used to describe potential obstacle areas (singular value areas). Therefore, the posterior distribution p _i ={pi _ik } _k=1:4 of the i-th pixel in the sea surface image can be calculated by the following formula:

In the above formula, m′ ₁ and C′ ₁ are the Gaussian parameters of the new sky category, m ₃ and C ₃ are the Gaussian parameters of the seawater category after EM optimization, and _πi3 and _πi4 are the sea surface images after EM optimization The i-th pixel belongs to the category prior probability of seawater and obstacle category, and π′ _i1 is the category prior probability of the i-th pixel in the sea surface image belonging to the new sky category:

π′ _i1 = π _i1 + π _i2 (19)

In the above formula, π _i1 and π _i2 are the category prior probability of the i-th pixel in the EM-optimized sea surface image belonging to the mixed category of sky, middle coast/haze, respectively.

Further, in the step (5), when dst<T, use the posterior distribution p={p _i } _i=1:M calculated by the formula (18) to obtain the semantic segmentation result of the sea surface image When dst≥T, use the posterior distribution q={q _i } _i=1:M optimized by step (3) EM to obtain the semantic segmentation result of the sea surface image

Compared with prior art, the beneficial effect of the present invention is:

Firstly, the method proposed by the present invention can automatically choose whether to simplify the mixed model of space constraints according to the actual situation of the sea surface image, thereby improving the accuracy of the semantic segmentation of the sea surface image. Secondly, the sea surface image semantic segmentation method designed in the present invention has the characteristics of simple model structure and fast speed, and is easy to deploy in actual projects.

Description of drawings

Fig. 1 is the flowchart of the method of the present invention.

Fig. 2 is the schematic diagram of an embodiment of the method of the present invention, wherein (a) is the figure to be detected of the embodiment; (b) is the semantic segmentation result figure of the embodiment;

Fig. 3 is a schematic diagram of the semantic segmentation of the sea surface image in the embodiment without coast/haze background, wherein (a) is the image to be detected in the embodiment; (b) is the result image of the semantic segmentation in the embodiment.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. The methods or steps involved in the following embodiments, unless otherwise specified, are conventional methods or steps in the technical field, and those skilled in the art can make conventional selections or adaptive adjustments according to specific application scenarios. The following embodiments are implemented using the python programming language.

Example 1

As shown in Figure 1, a method for semantic segmentation of sea surface images based on a simplified spatially constrained hybrid model, the specific implementation steps are as follows:

(1) Input the color image of the sea surface to be detected;

The color image of the sea surface (with a resolution of 512×512) was obtained by the camera carried by the unmanned surface vehicle. In order to reduce the execution time of subsequent algorithms, the sea surface image is scaled to a resolution of 100×100. As shown in FIG. 2 a , the image of the sea surface to be detected in this embodiment mainly includes the sky, the coast, sea waves, and obstacle buoys.

(2) Assume that there are three main semantic areas of sky, coast/haze, and sea water in the sea surface image, as well as potential obstacle areas, and establish a hybrid model of spatial constraints;

It is assumed that the mixture model consists of 3 Gaussian distributions and 1 uniform distribution, of which 3 Gaussian distributions are used to describe the sky, haze/coast mixed area, sea water area, and the uniform distribution is used to describe potential obstacle areas (singular value field). In addition, the mixture model is spatially constrained by introducing a Markov Random Field (MRF), that is, it is assumed that the category prior distribution of all pixels in the image π={π _i } _i=1:M and the posterior distribution P={p _i } _i=1:M is an MRF about the neighborhood system. Through derivation, the penalty log-likelihood function of the spatially constrained mixed model is finally obtained:

In addition, the feature vector y _i of each pixel of the sea surface image is a 5-dimensional vector [c1, c2, c3, r, c] composed of color features (c1, c2, c3) and position features (r, c) ^T . Among them, the color feature component is determined by the YCbCr color space.

(3) Optimizing the space-constrained hybrid model by using the Expectation-Maximization Algorithm (EM);

The specific steps of expectation maximization (EM) are:

①Initialize the normal distribution parameter set θ＝{m _k ,C _k } _k＝1,2,3 . Divide the sea surface image into three regions from top to bottom according to the proportion {0,0.3}, {0.3,0.5} and {0.5,1}, and then calculate the mean value m of the sky category according to the characteristics of the pixels in these three regions ₁ and covariance C ₁ , mean m ₂ and covariance C ₂ for the Mid Coast/Haze mixed category, mean m ₃ and covariance C ₃ for the seawater category;

② Initialize the class prior distribution of all pixels π={π _i } _i=1:M . For the class prior distribution π _i of each pixel, its initialization formula is as follows:

In the above formula, ε is a very small positive hyperparameter. In this embodiment, ε=1×10 ⁻¹⁵ .

In E-step:

③Substitute θ and π into formula (6) to calculate the posterior distribution P={p _i } _i=1:M of all pixels, where the uniform distribution U(·)=ε=1×10 ^-15 .

④Use the following formula to calculate the auxiliary category prior distribution s={s _i } _i=1:M of all pixels;

In the above formula, Indicates Hadamard product operation, * indicates convolution operation, is the normalization constant.

⑤ Calculate the auxiliary posterior distribution q={q _i } _i=1:M of all pixels, the calculation formula is as follows:

In the above formula, ξ _qi is a normalization constant.

In M-step:

⑥Update the category prior distribution set, the calculation formula is as follows:

⑦Update Gaussian distribution parameters, the calculation formula is as follows:

⑧Judge whether the EM algorithm reaches the iteration termination condition; if so, stop the iteration, otherwise, continue ③～⑧. Among them, the iteration termination condition is as follows:

(4) Calculate the KL distance (Kullback-Leibler divergence) of the Gaussian distribution of the sky, coast/haze category. If the KL distance is less than the set threshold, the spatially constrained hybrid model is simplified;

When the KL distance dst of the Gaussian distribution of the sky, coast/haze category is less than the preset fixed threshold T, the mixture model is simplified; otherwise, step (5) is directly performed. Wherein, the preset fixed threshold T=8 in this embodiment.

(5) Output the semantic segmentation results of sea surface images. Fig. 2b is a diagram of the result of semantic segmentation in this embodiment.

Example 2

Fig. 3 is a preferred embodiment of the method of the present invention under the background of no coast/haze. Its specific implementation steps are the same as those in Embodiment 1, so they are not repeated here. As can be seen from the semantic segmentation results of Embodiment 1 and Embodiment 2, the method of the present invention can automatically select whether to simplify the mixed model of space constraints according to the actual situation of the sea surface image, thereby improving the accuracy of the semantic segmentation of the sea surface image .

Claims (3)

1. A sea surface image semantic segmentation method based on a simplified space-constrained hybrid model, is characterized in that, comprising the following steps: (1) Input the color image of the sea surface to be detected; (2) Assume that there are three main semantic areas of sky, coast/haze, and sea water in the sea surface image, as well as potential obstacle areas, and establish a hybrid model of spatial constraints; (3) Using the expectation maximization algorithm EM to optimize the space-constrained hybrid model; (4) Calculate the KL distance of the Gaussian distribution of the sky, coast/haze category, if the KL distance is less than the set threshold, the spatially constrained hybrid model is simplified; (5) Output the semantic segmentation results of sea surface images. 2. the method for semantic segmentation of sea surface images based on a simplified space-constrained hybrid model according to claim 1, characterized in that, in the step (4), when the unmanned surface vehicle sails towards the coast, the vehicle-mounted camera shoots From top to bottom, there are three main semantic areas of the sea surface image, sky, middle coast/haze, and sea water; There are two main semantic areas of sky and sea water; therefore, when there are only two main semantic areas of sky and sea water in the sea surface image, it is necessary to simplify the space-constrained hybrid model assumed in step (2), and the specific steps are as follows: ① On the basis of step (3), calculate the KL distance between the sky category after EM and the Gaussian distribution of the middle coast/haze mixed category: dst＝KL(N ₁ ||N ₂ ) (1) In the above formula, N ₁ represents the Gaussian distribution N(·|m ₁ ,C ₁ ) of the sky category, and N ₂ represents the Gaussian distribution N(·|m ₂ ,C ₂ ) of the middle coast/haze mixed category; ② If the KL distance dst is less than the preset fixed threshold T, simplify the hybrid model; otherwise, directly perform step (5); specifically, when dst<T, combine the sky category and the middle coast/haze mixed category Gaussian distributions are fused to form a new sky category Gaussian distribution: In the above formula, m ₁ and C ₁ are the Gaussian parameters of the sky category after EM optimization, and m ₂ and C ₂ are the Gaussian parameters of the coast/haze mixed category after EM optimization; Therefore, the mixture model is simplified to be composed of 2 Gaussian distributions and 1 uniform distribution, where 2 Gaussian distributions are used to describe the sky and sea water areas, and the uniform distribution is used to describe the potential obstacle area, that is, the singular value area; therefore , the posterior distribution p _i ={p _ik } _k=1:4 of the i-th pixel in the sea surface image is calculated by the following formula: In the above formula, N(·|m,C) represents a Gaussian distribution function with mean m and covariance C, U( )=ε represents a uniform distribution function, where ε is a very small positive hyperparameter; y _i represents the feature vector of the i-th pixel in the image, also known as observation data, which is mainly composed of pixel color features (c1, c2, c3) and coordinates (r, c); m′ ₁ and C′ ₁ are the new The Gaussian parameters of the sky category, m ₃ and C ₃ are the Gaussian parameters of the seawater category after EM optimization, and _πi3 and _πi4 are the i-th pixel in the sea surface image after EM optimization, respectively. Prior probability, π′ _i1 is the category prior probability of the i-th pixel in the sea surface image belonging to the new sky category: π′ _i1 ＝π _i1 +π _i2 (4) In the above formula, π _i1 and π _i2 are the category prior probability of the i-th pixel in the EM-optimized sea surface image belonging to the mixed category of sky, middle coast/haze, respectively. 3. the sea surface image semantic segmentation method based on the simplifiable space constraint mixture model according to claim 1, is characterized in that, in described step (5), when the sky category after EM and middle coast/haze mixed category When the KL distance of the Gaussian distribution dst<T, use the posterior distribution p={p _i } _i=1:M calculated by the formula (3) to obtain the semantic segmentation result of the sea surface image When dst≥T, use the posterior distribution q={q _i } _i=1:M optimized by step (3) EM to obtain the semantic segmentation result of the sea surface image