CN102509104A - Confidence map-based method for distinguishing and detecting virtual object of augmented reality scene - Google Patents

Abstract

A method for discriminating and detecting virtual objects in an augmented reality scene based on a confidence map, including: selecting virtual and real classification features; using virtual and real classification features; constructing a pixel-level virtual and real classifier; using virtual and real classification features to extract the regional comparison between the augmented reality scene and the real scene features, build a region-level virtual-real classifier; given a test augmented reality scene, use a pixel-level virtual-real classifier and a small-sized detection window for detection, and obtain a virtual score map reflecting the virtual-real classification result of each pixel; define a virtual confidence map, use The virtual confidence map of the test augmented reality scene is obtained by thresholding; according to the distribution of high virtual response points in the virtual confidence map, the rough shape and position of the bounding box of the virtual object are obtained; in the test augmented reality scene, the region-level virtual-real classifier and The large-size detection window is used for detection, and the final detection result of the virtual object is obtained. The invention can be applied to fields such as film and television production, digital entertainment, education and training.

Description

Discrimination and detection method of virtual objects in augmented reality scenes based on confidence map

technical field

The invention relates to the fields of image processing, computer vision and augmented reality, in particular to a method for discriminating and detecting a virtual object in an augmented reality scene based on a confidence map.

Background technique

Augmented reality is a further extension of virtual reality. With the help of necessary equipment, computer-generated virtual objects and objectively existing real environments coexist in the same augmented reality system, presenting virtual objects and real environments to users from the perspective of sensory and experience effects. An integrated augmented reality environment. With the development of augmented reality technology and the emergence of augmented reality scenes with higher image realism, there is an urgent need for standards and basis for measuring and evaluating the credibility of augmented reality scenes. How to determine whether a scene is an augmented reality scene, and further detect the virtual objects in the augmented reality scene, as a way to evaluate the credibility of augmented reality scene images, has important research significance and application requirements.

In 2011, researchers at the University of Trento in Italy proposed an image forgery identification method that can detect computer-generated components integrated into real scenes. This work is the only known existing work dealing with augmented reality scenes. However, the detection in this work is not based on the object, but only detects the virtual components in the augmented reality scene, that is, the detection result may be a region or scattered points.

In 2005, researchers at Dartmouth University in the United States proposed a statistical model of natural images based on wavelet decomposition and used support vector machines and linear discriminant analysis to classify virtual images and real images. First, extract the fourth-order statistical features (mean, variance, skewness, kurtosis) of the decomposition coefficients in each sub-band and direction of the color image after wavelet decomposition; at the same time, consider the fourth-order linear prediction error between adjacent decomposition coefficients after wavelet decomposition features, and then use the support vector machine and linear discriminant analysis to train the classifier, and then input the test set into the trained classifier to get the classification result. The virtual and real classification of this method is carried out for the entire image, and the classification accuracy fluctuates greatly with the size of the extraction area of the virtual and real classification features.

In 2007, researchers at the New York University of Science and Technology proposed a method to distinguish virtual images from real images by using color filter array interpolation detection characteristics and the consistency of color difference in images. This method first extracts the features based on the color filter array interpolation detection characteristics and the consistency of color difference in the image from the positive and negative samples of the training set, and then inputs the extracted features into the support vector machine to train the classifier, and then inputs the test set into the trained The classifier gets the classification result.

In 2009, researchers at the University of Alberta in Canada proposed a method to classify virtual images and real images by using the consistency of image block resampling parameters. The principle of this method is based on the fact that the process of mapping the surface texture of the model in the virtual image generation may use operations such as rotation and scaling of the texture image, resulting in inconsistencies in the resampling parameters of each image block in the virtual image. In this way, the virtual image and the real image can be distinguished by detecting whether the parameters of image block resampling are consistent. The parameter estimation of image block resampling in this method is carried out for the whole image.

In 2004, researchers at the Cambridge Research Laboratory of Compaq Computer Corporation of the United States proposed a method for face detection based on the Haar filter and the AdaBoost classification algorithm. This method first extracts the classification features from the training set, and then trains a classifier based on the statistical features of faces and non-faces, and then inputs the extracted classification features of the image to be detected into the classifier and reduces the calculation time by cascading classifiers. The number of detection windows is used to improve efficiency, and finally the detection results are obtained. The feature extraction of this method is based on the Haar filter, which describes the regional contrast brought about by the inherent structure of the face.

In 2005, researchers at the French National Institute of Computer and Automation proposed a method for person detection using a histogram of oriented gradients and a linear support vector machine. This method firstly normalizes the color of the input image, then calculates the gradient in the image, counts the pixels falling in different directions and orientation intervals, compares and normalizes the overlapping spatial blocks, and then generates each detection window The histogram of the direction gradient, and finally use the linear support vector machine classifier to classify the person/non-person area, and get the detection result. Compared with other detection methods, this method has a higher detection effect, but requires the characters in the picture to roughly maintain a vertical standing state. The feature extraction of this method uses the image gradient histogram, which describes the inherent characteristics of the human body contour.

The above-mentioned methods for distinguishing virtual images from real images have in common that none of the features extracted by them for the virtual-real classification are suitable for any given region in the image. In addition, existing object detection works generally deal with objects that have strong and easy-to-describe appearance characteristics as prior information. Relatively speaking, in the detection of virtual objects in augmented reality scenes, the detection target (that is, the virtual object) does not have explicit prior information that is easy to describe in appearance, such as color, shape, size, etc., so it is difficult to distinguish and detect .

Contents of the invention

The technical solution of the present invention: overcome the deficiencies of the prior art, and provide a method for discriminating and detecting virtual objects in augmented reality scenes based on confidence maps, which does not need to know any appearance information of virtual objects in advance, such as color, shape, It does not need to know the position of the virtual object in the augmented reality scene, but uses the physical imaging difference between the virtual object and the real image to extract the virtual and real classification features, and calculates the region's own characteristics of the positive and negative samples of the training set and Based on the regional comparison features, a pixel-level virtual-real classifier and a region-level virtual-real classifier are constructed; on this basis, the virtual object is initially shaped and accurately detected through the virtual object discrimination and detection based on the virtual confidence map.

The technical scheme adopted by the present invention: a method for discriminating and detecting virtual objects in augmented reality scenes based on a confidence map, the steps are as follows: construct an augmented reality scene training data set, and use the physical imaging difference between virtual objects and real images to select virtual and real classification features; On the training data set, use the virtual and real classification features to extract the regional characteristics of the augmented reality scene and the real scene respectively, and construct a pixel-level virtual and real classifier; on the training data set, use the virtual and real classification features to extract the augmented reality scene and the real scene respectively Construct a region-level false-real classifier based on the regional comparison features of the region; given a test augmented reality scene, use a pixel-level virtual-real classifier and a small-sized detection window for detection, and obtain a virtual score map reflecting the virtual-real classification result of each pixel; define virtual confidence , and on the basis of the virtual score map, thresholding is used to obtain a virtual confidence map of the test augmented reality scene; according to the distribution of high virtual response points in the virtual confidence map, the rough shape and position of the virtual object's bounding box are obtained; Based on the rough positioning of the virtual object, the region-level virtual-real classifier and large-size detection window are used to detect in the test augmented reality scene image, and the final detection result of the virtual object is obtained.

Build an augmented reality scene training dataset. In the training dataset, images of augmented reality scenes containing virtual objects are used as positive samples, and images of real scenes are used as negative samples. Using the physical imaging difference between the virtual object and the real image, the virtual and real classification features are selected. The selected virtual classification features include: local statistics, surface gradient, second fundamental form, Beltrami flow. At each pixel point of the image, the above-mentioned virtual-real classification feature corresponding to the point can be extracted.

On the training data set, the features of virtual and real classification are used to extract the region's own characteristics of the augmented reality scene, and a pixel-level virtual and real classifier is constructed. When building a pixel-level classifier, for augmented reality scene images, only virtual object regions are selected as positive sample regions; for real scene images, only regions similar to virtual objects in positive samples are selected as negative sample regions. For a given image area, the virtual and real classification features of each point in the area are calculated (including: local statistics, surface gradient, second basic form, Beltrami flow); The classification features are compressed to obtain the region's own characteristics corresponding to the region. Input the region's own feature set of positive and negative samples into the support vector machine classifier for training, and obtain a pixel-level virtual-real classifier.

On the training data set, the virtual-real classification feature is used to extract the regional contrast features of the augmented reality scene, and a region-level virtual-real classifier is constructed. For the positive and negative sample areas, it is regarded as the object area to be determined; and the equal-area rectangular area outside the bounding box of the area is regarded as the background area where the object is located; the virtual and real points of each point in the object area and the background area are respectively extracted Classification features; count the virtual and real classification features corresponding to all points in the object area and the background area, respectively constitute the joint distribution histogram of the object area features and the joint distribution histogram of the background area features; calculate the chi-square distance between the two histograms, It is regarded as a feature to measure the difference between the object and its background, which is called the regional contrast feature; the extracted regional contrast feature set of positive and negative samples is input into the support vector machine classifier for training, and a region-level virtual-real classifier is obtained.

The virtual score map is constructed. The step is to use a small-sized detection window (the detection window size is [10, 30] × [10, 30] pixels) for the input augmented reality scene image with a small moving step (such as {1, 2, 3, 4, 5} pixels) to scan the entire image; calculate the regional self-features of the small image blocks in each small-size detection window; input the regional self-features of all small image blocks into the pixel-level virtual-real classifier, Obtain the region's own feature score of each small image block, and a high score indicates that the pixel-level classifier has a high degree of certainty in classifying the image block as a virtual area; since the size of the detection window is relatively small and densely distributed compared to the entire image, it is possible to The regional characteristic score of each small image block is mapped to the central pixel of the image block, and it is used as the virtual score of the central pixel; thus, a virtual score map of the entire augmented reality scene image is formed. This procedure improves computational efficiency through 2D integral plots.

The virtual confidence map is constructed, and the step is to threshold the virtual score map of the augmented reality scene image, and record all points with positive virtual scores; set a fixed percentage N%, and record the top N of points with positive virtual scores % and the positions of these points on the original image, these points are called high virtual response points; set a fixed and relatively small constant M (such as let M∈[10, 100]), record all virtual scores as positive The first M points of the point and the positions of these points on the original image are called the highest virtual response points; through parameter setting, it can be guaranteed that the highest virtual response point is also included in the set of high virtual response points, that is, the highest The virtual response point is the part with the highest virtual score value among the high virtual response points. The high virtual response point, the highest virtual response point and its location information on the original image are combined to form a virtual confidence map.

The rough shape and position inference steps of the virtual object bounding box are as follows: Divide the obtained virtual confidence map into five equal-area, overlapping sub-regions, and obtain the distribution of high virtual response points in each sub-region respectively Center; regard the center of the sub-area as the candidate center point of the virtual object, and search outwards from each center point to obtain the area with dense distribution of high virtual response points. For the area with dense distribution of high virtual response points, approximately calculate the The candidate object shape (reflected as the candidate virtual object bounding box), combined with the location information of the region, constitutes a virtual object preliminary candidate region; in multiple virtual object preliminary candidate regions, according to their respective high virtual response points and the highest The number of virtual response points, select the one with the largest weighted number, and use it as a virtual object candidate area, which contains the rough shape and position information of the virtual object's bounding box.

The obtained rough positioning of the virtual object is further optimized to obtain the final detection result of the virtual object. The specific steps are: take an area around the virtual object candidate area twice as large as the virtual object candidate area, and construct a plurality of overlapping large-size detection windows (large-size detection windows) with the same shape and size as the virtual object candidate area in this area. The size range of is usually [200, 500] × [200, 500], and the specific values of its length and width are equal to the length and width of the virtual object bounding box in the virtual object candidate area); take each large-size detection window image block and calculate its regional contrast features; the regional contrast features of all image blocks in the large-scale detection window are input into the region-level virtual-real classifier for classification, and the detection window with the highest corresponding score is selected as the final detection result of the virtual object.

Compared with the prior art, the present invention has the beneficial effects of:

(1) The present invention takes the virtual object in the augmented reality scene as the detection object, and can distinguish and detect the virtual object in the augmented reality scene as a whole.

(2) The present invention builds a two-level virtual-real classifier, including a pixel-level virtual-real classifier and an area-level virtual-real classifier, to meet the requirements of confidence map construction and virtual object final detection.

(3) The present invention builds a confidence map, based on the virtual confidence map, the approximate position and shape of the virtual object in the augmented reality scene can be obtained without prior information such as the appearance, shape, and position of the virtual object.

(4) The present invention does not need to know any appearance information of the virtual object in advance, such as prior information such as color, shape, size, and does not need to know the position of the virtual object in the augmented reality scene, so it has wider applicability. It can be widely applied to the fields of film and television production, digital entertainment, education and training, etc.

Description of drawings

Fig. 1 is the overall design structure of the present invention;

Fig. 2 is a flow chart of constructing a virtual confidence map of the present invention;

Fig. 3 is the flow chart of the virtual object bounding box shape and position inference of the present invention;

Fig. 4 is the flow chart of obtaining candidate central point of the present invention;

Fig. 5 is a flow chart of the present invention to expand the search and obtain densely distributed areas of high virtual response points.

Detailed ways

As shown in Figure 1, the main steps of the present invention are as follows: build the augmented reality scene training data set, and use the physical imaging difference between the virtual object and the real image to select the virtual and real classification features; on the training data set, use the virtual and real classification features to extract Augment the region's own characteristics of the augmented reality scene, and build a pixel-level virtual-real classifier; on the training data set, use the virtual-real classification feature to extract the regional contrast features of the augmented reality scene, and build a region-level virtual-real classifier; given the test augmented reality scene, use The pixel-level virtual and real classifier performs small-scale detection to obtain a virtual score map reflecting the virtual and real classification results of each pixel; define a virtual confidence map, and use thresholding to obtain the virtual confidence of the test augmented reality scene on the basis of the virtual score map degree map; according to the distribution of high virtual response points in the virtual confidence map, the rough shape and position of the bounding box of the virtual object are obtained; on the basis of the rough positioning of the virtual object, a region-level virtual-real classifier is used in the test augmented reality scene image and the large-size detection window are detected to obtain the final detection result of the virtual object.

Construct a training data set for training a virtual-real classifier. The training dataset consists of augmented reality scene images containing virtual objects as positive samples and real scene images as negative samples. When training a pixel-level classifier, for augmented reality scene images, only virtual object regions are selected as positive samples; for real scene images, only regions similar to virtual objects in positive samples are selected as negative samples. When training the region-level classifier, for the augmented reality scene image, select the image area of the same area as the virtual object and its surrounding area as the positive sample; for the real scene image, select the area similar to the virtual object in the positive sample and its surrounding area, etc. The image region of the area is used as a negative sample.

The extraction of the region's own features. For a given image area, calculate the virtual and real classification features of each point in the area, including: local statistics, surface gradient, second basic form, Beltrami flow; use the moment of inertia compression method to classify the virtual and real in a given area The features are compressed to obtain the region's own characteristics corresponding to the region.

The physical meanings and calculation methods of local statistics, surface gradient, second basic form, and Beltrami flow are as follows:

Local statistics reflect the local tiny edge structure. The calculation method of local statistics is as follows: take any point P on the grayscale image of the original image, a small image block of 3×3 pixels centered on point P, and arrange the pixel values of each point in order into a 9-dimensional vector x=[x ₁ , x ₂ , . . . x ₉ ]. The local statistic y of point P is a 9-dimensional vector, which is defined as:

$\mathrm{the\; y}=\frac{x-\stackrel{\‾}{x}}{{\left|\right|x-\stackrel{\‾}{x}\left|\right|}_{\mathrm{D.}}}.$ in, $\stackrel{\‾}{x}=\frac{1}{9}{\mathrm{\Σ}}_{i=1}^9{x}_i;$ And ||·|| _D is the D norm operation.

其中i～j表示图像块中所有四邻域关系的点对。The definition of the D-norm operation is:

Among them, i~j represent all four-neighborhood relationship point pairs in the image block.

The virtual-real classification feature of local statistics at any point p is the 9-dimensional vector y at that point.

The surface gradient is used to measure the nonlinear change characteristics in the real scene imaging process. The surface gradient S at any point in the image is defined as:

其中，

为该点处的图像梯度模值，I_x、I_x分别表示图像x方向(水平方向)和y方向(竖直方向)的偏导。α为常数，α＝0.25。

in,

is the image gradient modulus at this point, I _x , I _x represent the partial derivatives in the x direction (horizontal direction) and y direction (vertical direction) of the image, respectively. α is a constant, α=0.25.

The surface gradient virtual-real classification feature at any point p is composed of the image pixel value I at this point and the surface gradient S at this point.

The second basic form is used to describe the local unevenness of the image surface. The two components λ ₁ and λ ₂ of the second basic form correspond to the two eigenvalues of the matrix A, respectively.

$A=\frac{1}{\sqrt{1+I_x^2+I_{\mathrm{the\; y}}^2}}\left(\begin{array}{cc}{I}_{\mathrm{xx}}& I_{\mathrm{xy}}\\ I_{\mathrm{xy}}& I_{\mathrm{yy}}\end{array}\right),$ Among them, I _x , I _x represent the partial derivatives in the x direction and y direction of the image respectively; I _xx , I _xy , I _yy represent the second order partial derivatives in the xx direction, the xy direction, and the yy direction of the image respectively; from this formula, we can calculate The value of matrix A. Let us write A as: $A=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{\mathrm{twenty\; one}}& a_{\mathrm{twenty\; two}}\end{array}\right),$ Wherein, a ₁₁ , a ₁₂ , α ₂₁ , and α ₂₂ represent the corresponding four element values in the matrix A, respectively. Therefore, the two eigenvalues λ ₁ and λ ₂ of matrix A are calculated as follows:

$<span\; class="notranslate">\; \{</span>{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}<span\; class="notranslate">\; \&PlusMinus;</span>\sqrt{{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 4</span>}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}}}{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&lambda;</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}$

与该点处第二基本形式的两个分量λ₁、λ₂联合构成。The second basic form of virtual and real classification features at any point p is determined by the image gradient modulus at this point

Formed jointly with the two components λ ₁ , λ ₂ of the second fundamental form at this point.

Beltrami flow can be used to describe the correlation between different color channels. The Beltrami flow Δ _g I _c corresponding to the color channel c (c={R, G, B}) is defined as:

${\mathrm{<span\; class="notranslate">\; \&Delta;</span>}}_{\mathrm{<span\; class="notranslate">\; g</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; \&PartialD;</span>}_{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; (</span>\sqrt{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; g</span>}}^{\mathrm{<span\; class="notranslate">\; xx</span>}}{<span\; class="notranslate">\; \&PartialD;</span>}_{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; g</span>}}^{\mathrm{<span\; class="notranslate">\; xy</span>}}{<span\; class="notranslate">\; \&PartialD;</span>}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; (</span>{<span\; class="notranslate">\; \&PartialD;</span>}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; (</span>\sqrt{<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; g</span>}}^{\mathrm{<span\; class="notranslate">\; yx</span>}}{<span\; class="notranslate">\; \&PartialD;</span>}_{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; g</span>}}^{\mathrm{<span\; class="notranslate">\; yy</span>}}{<span\; class="notranslate">\; \&PartialD;</span>}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

分别表示对于作用量取x方向和y方向的偏导；Among them, I _c represents the image corresponding to the color channel c (c={R, G, B}) of the original image; the operator

Respectively represent the partial derivatives in the x-direction and y-direction for the action;

分别表示图像R通道(红色通道)x方向和y方向的偏导；

分别表示图像G通道(绿色通道)x方向和y方向的偏导；

分别表示图像B通道(蓝色通道)x方向和y方向的偏导；|g|为矩阵g的行列式；而g^xx、g^xy、g^yy、g^yx则由 $g^{-1}=\left(\begin{array}{cc}{g}^{\mathrm{xx}}& g^{\mathrm{xy}}\\ g^{\mathrm{yx}}& g^{\mathrm{yy}}\end{array}\right)$ 分别给出，即g^xx、g^xy、g^yy、g^yx分别为矩阵g的逆矩阵对应的四个元素值。matrix $g=\left(\begin{array}{cc}1+{\left(I_x^R\right)}^2+{\left(I_x^G\right)}^2+{\left(I_x^B\right)}^2& I_x^R{I}_{\mathrm{the\; y}}^R+I_x^G{I}_{\mathrm{the\; y}}^G+I_x^B{I}_{\mathrm{the\; y}}^B\\ I_x^R{I}_{\mathrm{the\; y}}^R+I_x^G{I}_{\mathrm{the\; y}}^G+I_x^B{I}_{\mathrm{the\; <figure-callout\; id="1"\; label="y\; B"\; filenames="HDA0000095251700000022.png"\; state="\{\{state\}\}">y</figure-callout>}}^B& 1+{\left(I_{\mathrm{the\; y}}^R\right)}^2+{\left(I_{\mathrm{the\; y}}^G\right)}^2+{\left(I_{\mathrm{the\; y}}^B\right)}^2\end{array}\right),$

Respectively represent the partial derivatives in the x direction and y direction of the image R channel (red channel);

Respectively represent the partial derivatives in the x direction and y direction of the image G channel (green channel);

respectively represent the partial derivatives in the x direction and y direction of the image B channel (blue channel); |g| is the determinant of the matrix g; and g ^xx , g ^xy , g ^yy , g ^yx are represented by $g^{-1}=\left(\begin{array}{cc}{g}^{\mathrm{xx}}& g^{\mathrm{xy}}\\ g^{\mathrm{yx}}& g^{\mathrm{yy}}\end{array}\right)$ They are respectively given, that is, g ^xx , g ^xy , g ^yy , and g ^yx are the four element values corresponding to the inverse matrix of the matrix g.

联合构成。The Beltrami flow virtual-real classification feature at any point p is composed of the Beltrami flow component Δ _g I _c of each color channel c (c={R, G, B}) at this point and the image of each color channel gradient modulus

joint composition.

矩阵J的任意一个元素记为J_jk(j，k＝1，...，M)。J_jk的计算方法为：

(j，k＝1，...，M)。其中m_i表示质点P_i的质量，v_i＝(v_i1，...，v_im)表示点P_i在特征空间中的位置坐标；|v_i|表示质点P_i到坐标原点的欧氏距离，即

δ_jk为克罗内克函数，其计算方法为 ${\mathrm{\&delta;}}_{\mathrm{jk}}=\left\{\begin{array}{cccc}1& ,& \mathrm{if}& i=j\\ 0& ,& \mathrm{if}& i\&NotEqual;j\end{array}\right..$ 由此可以确定转动惯量矩阵J的所有元素J_jk(j，k＝1，...，M)。由转动惯量矩阵J的对称性可知，J_jk＝J_kj，因此只取矩阵J的主对角线以及主对角线以上的所有元素J_jk(j，k＝1，...，M且j≤k)，这些元素即可表示原始矩阵J的所有信息。After calculating the four sets of virtual and real classification features (including: local statistics, surface gradient, second basic form, Beltrami flow) for each point in the region, it is necessary to use the moment of inertia compression method to compress the virtual and real classification features. The steps of the moment of inertia compression method are as follows: First, consider any one of the four groups of virtual and real classification features, such as local statistics, surface gradient, second basic form, and Beltrami flow (the processing method of each group of virtual and real classification features is the same. ). If there are N points in the given area, the total M dimension of a group of virtual and real classification features of any point P _i (i=1, ..., N) (the value of M is determined according to one of the four groups of virtual and real classification features. can be determined), and a group of virtual and real classification features of point P _i (i=1, ..., N) is recorded as v _i = (v _i1 , ..., vi _im ). At this time, the virtual-real classification feature v _i =(v _i1 ,...,v _im ) of point P _i is regarded as a particle in the M-dimensional feature space, and the quality of the particle is stipulated as , the position coordinates of the particle in the M-dimensional feature space are v _i =(v _i1 ,...,v _im ), then the moment of inertia matrix of the mass point system composed of all N mass points can be calculated by the rigid body moment of inertia matrix formula J. The moment of inertia matrix J is an M×M dimensional matrix, and the matrix J can be written as follows:

Any element of the matrix J is denoted as J _jk (j, k=1, . . . , M). The calculation method of J _jk is:

(j, k=1, . . . , M). Where m _i represents the mass of particle P _i , v _i = (v _i1 ,..., v _im ) represents the position coordinates of point P _i in the feature space; |v _i | represents the Euclidean distance from the particle P _i to the coordinate origin, namely

δ _jk is the Kronecker function, and its calculation method is ${\mathrm{\&delta;}}_{\mathrm{jk}}=\left\{\begin{array}{cccc}1& ,& \mathrm{if}& i=j\\ 0& ,& \mathrm{if}& i\&NotEqual;j\end{array}\right..$ All elements J _jk (j, k=1, . . . , M) of the moment of inertia matrix J can thus be determined. It can be seen from the symmetry of the moment of inertia matrix J that J _jk =J _kj , so only the main diagonal of matrix J and all elements above the main diagonal J _jk (j, k=1,..., M and j≤k), these elements can represent all the information of the original matrix J.

联合所有质点与坐标原点距离|v_i|的均值、方差、偏度、峰度；组合构成一个特征向量。该特征向量即为该区域内所有点的该组虚实分类特征经转动惯量压缩方法得到的压缩表示结果。将分别得到的四组压缩表示结果联合，即可得到该区域对应的区域自身特征。由于转动惯量矩阵可以较好地描述多个质点在特征空间中的分布，因此转动惯量矩阵压缩方法可以在对区域多个高维数据点进行压缩的同时尽可能低保证较大程度地保留原有数据分布的信息。Take all elements J _jk (j, k=1,..., M and j≤k) in the moment of inertia matrix; combine the centroid vectors of all mass points

Combine the mean, variance, skewness, and kurtosis of the distance |v _i | of all particles and the coordinate origin; the combination forms a feature vector. The eigenvector is the compressed expression result obtained by the moment of inertia compression method for the group of virtual and real classification features of all points in the area. Combining the four sets of compressed representation results obtained separately, the region's own characteristics corresponding to the region can be obtained. Since the moment of inertia matrix can better describe the distribution of multiple mass points in the feature space, the moment of inertia matrix compression method can compress multiple high-dimensional data points in the region while keeping the original Information about the data distribution.

Extraction of regional contrast features. For a given image area, the area itself is regarded as the object area to be determined; and the equal-area rectangular area outside the bounding box of the area is regarded as the background area where the object is located; the object area and the background area are calculated separately The virtual and real classification features of each point in the object area; the virtual and real classification features corresponding to all points in the object area and the background area are counted separately to form a joint distribution histogram of the object area and background area features; the joint distribution histogram of the object area features and the background area are calculated. The chi-square distance between the joint distribution histograms of regional features is regarded as a feature that can measure the contrast or difference between the object and its background, called the regional contrast feature.

A pixel-level virtual-real classifier and a region-level virtual-real classifier are constructed to distinguish whether a given region belongs to the region where the virtual object is located from the perspective of the region's own characteristics and the region's comparative characteristics.

The pixel-level virtual-real classifier is constructed by inputting the positive and negative samples of the training set; the region's own characteristics of the positive and negative samples are extracted respectively; the extracted region's own feature set of the positive and negative samples is input into the support vector machine classifier for training, and the pixel-level virtual-real classification is obtained device. The feature of the pixel-level virtual-real classifier is that the feature compression method it adopts makes its classification results have certain size adaptability. That is, when the features of the region to be classified are extracted from regions whose size is significantly different from that of the training set, the pixel-level virtual-real classifier has better accuracy in classifying whether a given region belongs to the region where the virtual object is located. Specifically: Although the pixel-level classifier is trained by the region’s own feature set of the virtual object (with a size of [10, 30]×[10, 30] pixels) in the training set, the experimental results show that the classifier is relatively The classification results of regional features of much smaller regions ([10, 30]×[10, 30] pixels in size) still have better accuracy. Since the objects classified by this classifier are for small-sized areas, and these small areas are used to approximately describe the pixels corresponding to the center point of the area, the classifier is called a pixel-level virtual-real classifier.

The region-level virtual-real classifier is constructed by inputting the positive and negative samples of the training set; the regional comparison features of the positive and negative samples are extracted respectively; the extracted regional comparison feature set of the positive and negative samples is input into the support vector machine classifier for training, and the regional-level virtual and real classification is obtained device. Since the classification feature used by the area-level virtual-real classifier is the overall distribution difference between the reaction area and its background, it can better distinguish and detect the object to be detected as a whole.

Build a virtual score map. For the input augmented reality scene image, use a small size detection window (the detection window size is [10, 30]×[10, 30] pixels) with a small moving step (such as {1, 2, 3, 4, 5 } pixel) to scan the entire image; calculate the regional self-features of the small image blocks in each small-size detection window; input the regional self-features of all small image blocks into the pixel-level virtual-real classifier, and obtain the region-specific features of each small image block The feature score of the area itself, a high score indicates that the pixel-level classifier has a high degree of certainty in classifying the image block as a virtual area; since the size of the detection window is relatively small and densely distributed compared to the entire image, the area of each small image block can be The self-feature score is mapped to the central pixel of the image block, and it is used as the virtual score of the central pixel; thus, a virtual score map of the entire augmented reality scene image is formed. Since the calculation of virtual and real classification features and feature compression operations in the area's own physical signs calculation are time-consuming, and for the generated large number of overlapping image blocks, the area's own features need to be calculated one by one, so the integral map method is used in this step to speed up the calculation process. The resulting virtual score map construction results can better reflect the relationship between the point on the image and whether the point belongs to the virtual object. That is: the experimental results show that: in the virtual score map, the points with high virtual scores are generally concentrated in the area where the virtual object is located; on the contrary, the points in the area where the virtual object is located have higher virtual scores.

The process of constructing a virtual confidence map is shown in Figure 2. First, threshold the obtained virtual score map, first select and record all points with positive virtual scores; set a fixed percentage N%, select and record the top N% of all points with positive virtual scores and these points position on the original image. These points are called high virtual response points. Set a fixed and relatively small constant M (for example, let M∈[10, 100]), select and record the first M points of all points with positive virtual scores and the positions of these points on the original image. These points are called the highest virtual response points. The number of highest virtual response points is much smaller than the number of high virtual response points. Combining high virtual response points, highest virtual response points and their location information on the original image constitutes a virtual confidence map. The construction result of the virtual confidence map can better reflect the relationship between a point on the image and whether the point belongs to a virtual object. That is: the experimental results show that: in the virtual confidence graph, the high virtual response points are generally concentrated in the area where the virtual object is located; on the contrary, the points in the area where the virtual object is located correspond to a relatively dense distribution of high virtual response points. Similarly, the highest virtual response point generally appears only in the area where the virtual object is located; on the contrary, in the area where the virtual object is located, there are generally more highest virtual response points. Among the high virtual response points and the highest virtual response points that appear outside the virtual object area, they are called noise points.

The rough shape and position inference process of the virtual object’s bounding box is shown in Figure 3, including the following steps: divide sub-regions and obtain candidate center points; expand the search to obtain densely distributed regions with high virtual response points; determine the preliminary candidate regions of virtual objects; virtual Object candidate area determination. Specifically, dividing the sub-regions is to divide the obtained virtual confidence map into five equal-area, overlapping sub-regions. The flow chart of obtaining candidate center points is shown in Figure 4. According to the distribution of high virtual response points in each sub-region, the mean shift algorithm is used to obtain the center point of the distribution of high virtual response points in each sub-region. The center point called candidate center points. The number of candidate center points is k (k ≤ 5, the case of k less than 5 corresponds to the absence of high virtual response points in some sub-regions), here it may be assumed that the center points of the area corresponding to the virtual object must be the above k candidate centers one of the points. Extend the search to obtain a densely distributed area of high virtual response points. The process is shown in Figure 5: For each candidate center point, the candidate center point is the center of the circle, and the length increased by a fixed step size is the radius, and the dynamic structure increases in turn. Large circular search area, until the number of high virtual response points in the current search area no longer increases, it can be considered that the boundary of the virtual object area has been searched; ideally, the condition for the extended search to stop is when the search radius increases, The increment of the number of high virtual response points in the search area is zero, but in order to eliminate the influence of noise points in the virtual confidence map, a noise suppression parameter is set to strengthen the condition of extended search stop as follows: when the search radius increases, The increment of the number of high virtual response points in the search area must be greater than the noise suppression parameter.

The determination of the preliminary candidate area of the virtual object is to approximate the shape of the candidate object in the area with dense distribution of high virtual response points, and combine the position information of the area to form the preliminary candidate area of the virtual object. When the extended search stops, the densely distributed area of high virtual response points is obtained, and the set P of all high virtual response points in this area can be known. According to P, the shape of the candidate object in the area can be obtained, which is reflected as the candidate object bounding box shape:

x _min = min({x|<x, y>∈P}); x _max = max({x|<x, y>∈P});

y _min = min({y|<x, y>∈P}); y _max = max({y|<x, y>∈P});

Among them, x _min and x _max represent the minimum and maximum values in the x direction in the image coordinates corresponding to the area where the bounding box of the candidate object is located; y _min and y _max represent the minimum and maximum values in the y direction in the image coordinates where the bounding box of the candidate object is located. maximum value. From this, the location and shape of the candidate object bounding box relative to the image can be determined.

The shape of the candidate object in the area is combined with the position information of the area (the position of the candidate center point) to form a preliminary candidate area of the virtual object.

Among the obtained k virtual object preliminary candidate areas, according to the number of high virtual response points and the highest virtual response points contained in each of them, select the one with the largest weighted number, and use it as the virtual object candidate area, which contains Information about the approximate shape and position of the bounding box of the virtual object.

For the obtained rough positioning of the virtual object, it is further optimized to reduce errors that may occur during the calculation of the candidate region of the virtual object, so as to obtain the final detection result of the virtual object. The specific steps are: take an area around the virtual object candidate area that is twice the area of the virtual object candidate area; construct a plurality of overlapping large-size detection windows (large-size detection windows) that have the same shape and size as the virtual object candidate area in this area The size range of is usually [200, 500] × [200, 500], and the specific values of its length and width are equal to the length and width of the virtual object bounding box in the virtual object candidate area); take each large-size detection window image block and calculate its regional contrast features; the regional contrast features of all image blocks in the large-scale detection window are input into the region-level virtual-real classifier for classification, and the detection window with the highest corresponding score is selected as the final detection result of the virtual object.

The above descriptions are only some basic explanations of the present invention, and any equivalent transformation made according to the technical solution of the present invention shall fall within the scope of protection of the present invention.

Parts not described in detail in the present invention belong to the well-known technology in the art.

Claims (8)

1. The augmented reality scene virtual object discrimination and detection method based on confidence map, it is characterized in that the realization steps are as follows: (1) Construct an augmented reality scene training data set with augmented reality images containing virtual objects as positive samples and real scene images as negative samples; and use the physical imaging difference between virtual objects and real images to select virtual and real classification features; (2) On the training data set, using the virtual-real classification features, extract the regional features of the augmented reality scene and the real scene respectively, and construct a pixel-level virtual-real classifier; (3) On the training data set, use the virtual-real classification feature to extract the regional comparison features of the augmented reality scene and the real scene respectively, and construct a region-level virtual-real classifier; (4) Given a test augmented reality scene, use a pixel-level virtual-real classifier and a small-sized detection window to detect, and obtain a virtual score map reflecting the virtual-real classification result of each pixel; (5) Define a virtual confidence map, and on the basis of the virtual score map, use thresholding to obtain a virtual confidence map of the test augmented reality scene; (6) Based on the virtual confidence map, roughly locate the virtual object, and obtain the rough shape and position of the bounding box of the virtual object; (7) On the basis of the rough positioning of the virtual object, the region-level virtual-real classifier and the large-size detection window are used for detection in the test augmented reality scene image, and the final detection result of the virtual object is obtained. 2. the augmented reality scene virtual object discrimination and detection method based on confidence map according to claim 1, is characterized in that: the virtual and real classification feature that chooses in the described step (1), comprises: local statistic, surface gradient, Second elementary form and Beltrami flow. At each pixel point of the image, the above-mentioned virtual-real classification feature corresponding to the point can be extracted. 3. the augmented reality scene virtual object discrimination and detection method based on confidence map according to claim 1, is characterized in that: when constructing pixel-level classifier in described step (2), on training data set, to enhanced In the real scene image, only the virtual object area is selected as the positive sample area; for the real scene image, only the area similar to the virtual object in the positive sample is selected as the negative sample area; for a given image area, each point in the area is calculated The virtual and real classification features of the given positive and negative sample areas are compressed using the moment of inertia compression method to obtain the corresponding regional features of the area; the positive and negative sample regional feature sets are input into the support vector machine classifier for training , to obtain a pixel-level virtual-real classifier. 4. the augmented reality scene virtual object discrimination and detection method based on confidence map according to claim 1, is characterized in that: when described step (3) constructs region-level virtual and real classifier, on training data set, for positive Negative sample area, regard itself as the object area to be determined; and regard the equal-area rectangular area outside the bounding box of the area as the background area where the object is located; extract the virtual and real classification features of each point in the object area and the background area respectively ; The virtual and real classification features corresponding to all points in the object area and the background area are counted to form the joint distribution histogram of the object area features and the joint distribution histogram of the background area features; calculate the chi-square distance between the two histograms, and combine them As a feature to measure the difference between the object and its background, it is called the regional contrast feature; the extracted regional contrast feature set of the positive and negative samples is input into the support vector machine classifier for training, and a region-level virtual-real classifier is obtained. 5. the augmented reality scene virtual object discrimination and detection method based on confidence map according to claim 1, is characterized in that: described step (4) virtual score map construction step is: for the augmented reality scene image of input, utilize The small size detection window scans the entire image with a small moving step; calculates the region's own characteristics of the small image blocks in each small size detection window; inputs the region's own characteristics of all small image blocks into the pixel-level virtual-real classifier , to get the region's own feature score of each small image block, a high score indicates that the pixel-level classifier has a high degree of certainty in classifying the image block as a virtual area; since the size of the detection window is relatively small compared to the entire image and the distribution is dense, it can be The regional feature score of each small image block is mapped to the central pixel of the image block, and it is used as the virtual score of the central pixel; thus, a virtual score map of the entire augmented reality scene image is formed. 6. the augmented reality scene virtual object discrimination and detection method based on confidence map according to claim 1, is characterized in that: described step (5) virtual confidence map construction method is: for the virtual score of augmented reality scene image Thresholding the graph, recording all points with positive virtual scores; setting a fixed percentage N%, recording the top N% of all points with positive virtual scores and the positions of these points on the original image, these points are called High virtual response points; set a fixed and relatively small constant M, record the first M points of all points with positive virtual scores and the positions of these points on the original image, these points are called the highest virtual response points; through The parameter setting can ensure that the highest virtual response point is also included in the set of high virtual response points, that is, the highest virtual response point is the part with the highest virtual score value among the high virtual response points; the comprehensive high virtual response point, the highest virtual response point and its location information on the original image constitute a virtual confidence map. 7. the augmented reality scene virtual object discrimination and detection method based on confidence map according to claim 1, is characterized in that: the method that described step (6) obtains the rough shape and the position of virtual object bounding box is: to obtain Divide it into five equal-area, overlapping sub-regions, and obtain the distribution center of high virtual response points in each sub-region respectively; regard the center of the sub-region as a candidate virtual object center point, from Each central point expands the search outwards to obtain the area with dense distribution of high virtual response points; for the area with dense distribution of high virtual response points, the shape of the candidate object in the area is approximately calculated, and combined with the location information of the area, a virtual Object preliminary candidate area; in the virtual object preliminary candidate area, according to the number of high virtual response points and the highest virtual response point contained in each, select the one with the largest weighted number, and use it as the virtual object candidate area, this area contains The rough shape and position information of the bounding box of the virtual object is obtained. 8. The augmented reality scene virtual object discrimination and detection method based on the confidence map according to claim 1, characterized in that: the step (7) virtual object detection method is specifically: testing the virtual object in the augmented reality scene Densely sample around the candidate area, construct multiple overlapping detection windows, and use a region-level virtual-real classifier for classification, and select the detection window with the best score as the final detection result of the virtual object.

Images