CN104822058B - A kind of stereo-picture notable figure extracting method - Google Patents

Abstract

The invention discloses a method for extracting a saliency map of a stereoscopic image. In the training stage, the contrast feature vector, the general feature vector and the background prior feature vector of each area in the right viewpoint image of multiple training stereo images are extracted and fused. Obtain the feature vector used to reflect the visual salience of each region in the right view point image of each stereo image, and establish a random forest regression training model between the feature vector and the average eye movement value; in the test phase, calculate the test stereo image The feature vector used to reflect the visual saliency of each region in the right viewpoint image, and according to the trained random forest regression training model, predict the 3D saliency map of the test stereo image; the advantage is that the extracted features can be compared with It can well reflect the significant changes of various factors, thus effectively improving the prediction accuracy of visually significant values.

Description

A method for extracting saliency map from stereo images

technical field

The invention relates to a method for processing image signals, in particular to a method for extracting saliency maps of stereoscopic images.

Background technique

In human visual reception and information processing, due to limited brain resources and differences in the importance of external environmental information, the human brain does not treat external environmental information equally in the processing process, but shows selective characteristics. When people watch images or video clips, their attention is not evenly distributed to every area of the image, but they pay more attention to certain salient areas. How to detect and extract salient regions with high visual attention in videos is an important research content in the field of computer vision and content-based video retrieval.

However, the process of human perception of stereoscopic images to produce stereoscopic vision is not a simple process of superimposing left-viewpoint images and right-viewpoint images. Therefore, stereoscopic vision features (such as: three-dimensional visual attention) are not simply an extension of planar vision characteristics. How to effectively extract stereoscopic features from stereoscopic images and how to make the extracted stereoscopic features conform to the three-dimensional viewing behavior of human eyes are all problems that need to be studied and solved in the process of extracting visual saliency maps from stereoscopic images.

Contents of the invention

The technical problem to be solved by the present invention is to provide a method for extracting saliency maps of stereoscopic images, which conforms to saliency semantic features, and has strong extraction stability and high extraction accuracy.

The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a method for extracting a saliency map of a stereoscopic image, which is characterized in that it includes two processes of a training phase and a testing phase, and the specific steps of the training phase are as follows:

①-1. The selected N sets of different stereoscopic images and the right disparity images of each stereoscopic image form a set, which is recorded as {L _i , R _i , d _i |1≤i≤N}, where N ≥1, L _i represents the left viewpoint image of the i-th stereo image in {L _i , R _i , d _i |1≤i≤N}, and R _i represents {L _i , R _i , d _i |1≤i ≤N} in the right viewpoint image of the i-th stereo image, d _i represents the right parallax image of the i-th stereo image in {L _i , R _i , d _i |1≤i≤N};

①-2. Use superpixel segmentation technology to divide the right viewpoint image of each stereoscopic image in {L _i , R _i , d _i |1≤i≤N} into M non-overlapping regions, and divide R _i The hth region of is denoted as SP _i,h , where M≥1, 1≤h≤M;

①-3. Calculate the contrast feature vector of each area in the right view point image of each stereoscopic image in {L _i , R _i , d _i |1≤i≤N}, and divide the hth area in R _i The contrast feature vector of SP _{i, h} is denoted as $x_{i,h}^c=[d_{i,h}{,d}_{i,h}^1,d_{i,h}^2,d_{i,h}^3,d_{i,h}^4,d_{i,h}^5,d_{i,h}^6],$ in, The dimension of is 36, where the symbol “[]” is a vector representation symbol, and d _{i, h} represent the distance between the first feature vector u _{i, h} of SP _{i, h} and the first feature vector of the adjacent area, The dimension of f _{i, h} is 20, f _{i, h} represents the mean value of the frequency response feature vector of all pixels in SP _{i, h} , the dimension of c _{i, h} is 9, c _{i, h} represents SP _i, The mean value of the color feature vectors of all pixels in _h , d _{i, h} represents the mean value of the disparity magnitude of SP _{i, h} , Represents the color histogram of all pixels in SP _{i, h} in the R component, G component and B component of the RGB color space and the R component, G component of all pixels in the adjacent area of SP _{i, h} in the RGB color space The distance of the color histogram of component and B component, Represents the color histogram of all pixels in SP _{i, h} in the L component, a component and b component of CIELAB color space and the L component, a of all pixels in the adjacent area of SP _{i, h} in CIELAB color space The distance of the color histogram of component and b component, Represents the distance between the color histogram of all pixels in SP _{i, h} in the H component of the HVS color space and the color histogram of all pixels in the adjacent area of SP _{i, h} in the H component of the HVS color space, Indicates the distance between the color histogram of all pixels in SP _{i, h} in the S component of the HVS color space and the color histogram of all pixels in the adjacent area of SP _{i, h} in the S component of the HVS color space, Indicates the distance between the LBP feature statistical histogram of all pixels in SP i, _h and the LBP feature statistical histogram of all pixels in the adjacent area of SP _{i, h} , Indicates the distance between the disparity statistical histogram of all pixels in SP i, _h and the disparity statistical histogram of all pixels in the adjacent area of SP _{i, h} , where the adjacent area refers to the distance between R _i and SP _{i, h} adjacent area;

①-4. Calculate the general feature vector of each region in the right view point image of each stereoscopic image in {L _i , R _i , d _i |1≤i≤N}, and divide the hth region in R _i The general eigenvector of SP _i,h is denoted as in, The dimension of is 33, where the symbol “[]” is a vector representation symbol, The dimension of is 20, Represents the variance of the frequency response feature vector of all pixels in SP _i,h , The dimension of is 9, Represents the variance of the color feature vectors of all pixels in SP _i,h , Represents the variance of the parallax magnitude of SP _{i, h} , the dimension of _{xi, h} is 2, _{xi, h} represents the coordinate position of the center pixel of SP _{i, h} , s _{i, h} represents the area of SP _{i, h} ;

①-5. Calculate the background prior feature vector of each region in the right view point image of each stereoscopic image in {L _i , R _i , d _i |1≤i≤N}, and set the hth in R _i The background prior feature vector of a region SP _{i, h} is denoted as $x_{i,h}^b=[e_{i,h},e_{i,h}^1,e_{i,h}^2,e_{i,h}^3,e_{i,h}^4,e_{i,h}^5,e_{i,h}^6],$ in, The dimension of is 36, where the symbol “[]” is a vector representation symbol, e _i,h represents the distance between the first feature vector u _i,h of SP _i,h and the first feature vector of the background area, The dimension of f _{i, h} is 20, f _{i, h} represents the mean value of the frequency response feature vector of all pixels in SP _{i, h} , the dimension of c _{i, h} is 9, c _{i, h} represents SP _i, The mean value of the color feature vectors of all pixels in _h , Indicates the mean value of the disparity magnitude of SP _i,h , Represents the color histogram of the R component, G component and B component of all pixels in SP _{i, h} in the RGB color space and the R component, G component and B component of all pixels in the background area in R _i in the RGB color space The distance of the color histogram of the B component, Represents the color histogram of all pixels in SP _{i, h} in the L component, a component and b component of CIELAB color space and the L component, a component and b component of all pixels in the background area in R _i in CIELAB color space The distance of the color histogram of the b component, Indicates the distance between the color histogram of the H component of all pixels in SP _i,h in the H component of the HVS color space and the color histogram of the H component of all pixels in the background area in R _i in the HVS color space, Represents the distance between the color histogram of the S component of all pixels in SP _i,h in the S component of the HVS color space and the color histogram of the S component of all pixels in the background area in R _i in the HVS color space, Indicates the distance between the LBP feature statistical histogram of all pixels in SP _{i, h} and the LBP feature statistical histogram of all pixels in the background area in R _i , Indicates the distance between the disparity statistical histogram of all pixels in SP _i,h and the disparity statistical histogram of all pixels in the background area in R _{i ,} where the background area refers to the leftmost, most Right, top, and bottom areas;

①-6. The contrast feature vector, general feature vector and background prior feature vector of each region in the right view image of each stereo image in {L _i , R _i , d _i |1≤i≤N} Arranged in order to form the feature vector used to reflect the visual salience of each region in the right view image of each stereoscopic image in {L _i , R _i , d _i |1≤i≤N}, R The feature vector used to reflect the visual salience of the hth region SP _{i ,h} in i is denoted as Xi _,h , Among them, the dimensions of X _{i, h} are 105, and the symbol “[]” here is a vector representation symbol;

①-7. Use random forest regression to perform feature vectors for reflecting visual salience of all areas in the right viewpoint images of all stereoscopic images in {L _i , R _i , d _i |1≤i≤N} training, and make the error between the regression function value obtained through training and the average eye movement value the smallest, and obtain the optimal random forest regression training model, which is denoted as f(D _inp ), where f() is the function representation, D _inp represents the input vector of the random forest regression training model;

The specific steps of the testing phase are as follows:

②-1. For any pair of test stereo images S _test , record the left viewpoint image, right viewpoint image, and right disparity image of S _test as L _test , R _test , and d _{test respectively} ; then use superpixel segmentation technology to divide R _test Divide into M non-overlapping areas, and record the hth area in R _test as SP _h '; where, M≥1, 1≤h≤M;

②-2. According to the process from step ①-3 to step ①-6, obtain the feature vector used to reflect the visual salience of each region in the R _test in the same operation mode, and convert the hth region in the R _test to The feature vector used to reflect the visual salience of SP _h ' is recorded as F _test,h ; then according to the optimal random forest regression training model f(D _inp ) obtained in the training stage, F _test,h is used as the optimal random The input vector of the forest regression training model, obtain the 3D visual saliency value of each region in the R _test , and record the 3D visual saliency value of the hth region SP _h ' in the R _test as S _3D,h , S _3D,h = f(F _{test, h} ); then take the 3D visual saliency value of each region in the R _test as the saliency value of all pixels in the corresponding region, thereby obtaining the 3D saliency map of the R _test , denoted as {S _3D ( x, y)}, where (x, y) represents the coordinate position of the pixel in the S _test , 1≤x≤W, 1≤y≤H, W and H correspond to the width and height of the S _test , S _3D (x, y) represents the pixel value of the pixel at the coordinate position (x, y) in {S _3D (x, y)}.

The contrast feature vector of the hth region SP _i,h in R _i in the step ①-3 The acquisition process is:

a1. Calculate the mean value of the frequency response feature vectors of all the pixel points in the hth region SP _i,h in R _i , denoted as f _i,h , the first in f _i,h The value of the element is equal to the h-th area SP _i,h in the frequency response feature vector of all pixels in R _i The mean value of the frequency response amplitude of elements, where the dimensions of f _i,h are 20,

a2. Calculate the mean value of the color feature vectors of all pixels in the hth area SP _i,h in R _i , denoted as c _i,h , $c_{i,h}=[{\stackrel{\‾}{R}}_{i,h},{\stackrel{\‾}{G}}_{i,h},{\stackrel{\‾}{B}}_{i,h},{\stackrel{\‾}{L}}_{i,h},{\stackrel{\‾}{a}}_{i.h},{\stackrel{\‾}{b}}_{i,h},{\stackrel{\‾}{h}}_{i,h},{\stackrel{\‾}{V}}_{i,h},{\stackrel{\‾}{S}}_{i,h}],$ Among them, the dimensions of c _{i and h} are 9, and the symbol “[]” here is a vector representation symbol, Represents the mean value of the color values of the R component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Represents the mean value of the color values of the G component of the RGB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the B component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Represents the mean value of the color values of the L component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the a component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the b component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the H component of the HVS color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of all pixels in the hth region SP _i,h in R _i in the V component of the HVS color space, Represents the mean value of the color values of the S component of the HVS color space for all pixels in the hth region SP _i,h in R _i ;

a3. Calculate the mean value of the parallax magnitude of the hth region SP _{i, h} in R _i , denoted as Equal to the mean value of the pixel values of all pixels in the area corresponding to SP _{i, h} in d _i ;

a4. Put f _{i, h} , c _{i, h} and Arranged in order to form the first feature vector of the hth region SP _i,h in R _i , denoted as u _i,h , Among them, the dimensions of u _{i and h} are 30, and the symbol “[]” here is a vector representation symbol;

a5. Calculate the distance between the first feature vector u _i,h of the hth area SP _i,h in R _i and the first feature vector of the adjacent area, denoted as d _i,h , Among them, the dimensions of d _{i, h} are 30, 1≤p≤M, Represents the set of serial numbers of the adjacent regions of SP _{i, h} in the hth region in R _i , u _{i, p} represents the first feature vector of the pth region SP _{i, p} in R _i , symbol "||" In order to take the absolute value sign, P represents the total number of adjacent areas of the hth area SP _{i, h} in R _i , where the adjacent area refers to the area adjacent to SP _{i, h} in R _i ;

a6. Calculate the color histogram of the R component, G component and B component of all pixels in the hth region SP _i,h in R _i in the RGB color space, denoted as Calculate the color histogram of the L component, a component and b component of all pixels in the hth area SP _{i, h} in R _i in the CIELAB color space, denoted as Calculate the color histogram of the H component of all pixels in the hth region SP _i,h in R _i in the HVS color space, denoted as Calculate the color histogram of the S component of the HVS color space for all pixels in the hth region SP _i,h in R _i , denoted as Calculate the LBP feature statistical histogram of all pixels in the hth region SP _i,h in R _i , denoted as Calculate the disparity statistical histogram of all pixels in the hth area SP _i,h in R _i , denoted as in, The dimension of is 16 ³ , The dimension of is 16 ³ , The dimension of is 16, The dimension of is 16, The dimension of is 256, The dimension of is 16;

a7. Calculate The distance from the color histogram of the R component, G component and B component of all pixels in the adjacent area of the hth area SP _{i, h} in R _i in the RGB color space is denoted as

calculate The distance from the color histogram of the L component, a component and b component of the CIELAB color space to all pixels in the adjacent area of the hth area SP _{i, h} in R _i is denoted as

calculate The distance from the color histogram of the H component of the HVS color space to all pixels in the adjacent area of the h-th area SP _{i, h} in R _i is denoted as

calculate The distance from the color histogram of the S component of the HVS color space to all pixels in the adjacent area of the hth area SP _i,h in R _i is denoted as

calculate The distance from the LBP feature statistical histogram of all pixels in the adjacent area of the hth area SP _i,h in R _i is denoted as

calculate The distance from the disparity statistical histogram of all pixels in the adjacent area of SP i, h in the hth area SP _{i, h} in R _i is denoted as

Among them, 1≤p≤M, Indicates the set of serial numbers of the adjacent areas of the hth area SP _{i, h} in R _i , P indicates the total number of adjacent areas of the hth area SP _{i, h} in R _i , χ() is chi-square distance function, Indicates the color histogram of the R component, G component and B component of all pixels in the pth area SP _i,p in R _i in the RGB color space, Represents the color histogram of the L component, a component and b component of all pixels in the pth area SP _i,p in R _i in the CIELAB color space, Represents the color histogram of the H component of the HVS color space for all pixels in the p-th region SP _i,p in R _i , Represents the color histogram of the S component of the HVS color space for all pixels in the p-th region SP _i,p in R _i , Represents the LBP feature statistical histogram of all pixels in the p-th region SP _i,p in R _i , Represents the disparity statistical histogram of all pixels in the p-th region SP _{i, p} in R _i ;

a8, combine d _{i, h} , and Arranged in order to form the contrast feature vector of the hth region SP _i,h in R _i , denoted as $x_{i,h}^b=[e_{i,h},e_{i,h}^1,e_{i,h}^2,e_{i,h}^3,e_{i,h}^4,e_{i,h}^5,e_{i,h}^6],$ in, The dimension of is 36, where the symbol “[]” is a vector symbol.

The general feature vector of the h-th area SP _i,h in R _i in the step ①-4 The acquisition process is:

b1. Calculate the variance of the frequency response feature vectors of all pixels in the hth region SP _i,h in R _i , denoted as in the first The value of the element is equal to the h-th area SP _i,h in the frequency response feature vector of all pixels in R _i The variance of the frequency response amplitude of elements, where, The dimension of is 20,

b2. Calculate the variance of the color feature vectors of all pixels in the hth region SP _{i, h} in R _i , denoted as ${\hat{c}}_{i,h}=[{\hat{R}}_{i,h},{\hat{G}}_{i,h},{\hat{B}}_{i,h},{\hat{L}}_{i,h},{\hat{a}}_{i,h},{\hat{b}}_{i,h},{\hat{h}}_{i,h},{\hat{V}}_{i,h},{\hat{S}}_{i,h}],$ in, The dimension of is 9, where the symbol “[]” is a vector representation symbol, Represents the variance of the color value of the R component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Indicates the variance of the color value of the G component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Indicates the variance of the color value of the B component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Represents the variance of the color value of the L component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Represents the variance of the color value of the a component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Represents the variance of the color value of the b component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Represents the variance of the color value of the H component of the HVS color space for all pixels in the hth region SP _i,h in R _i , Represents the variance of the color value of the V component of the HVS color space for all pixels in the hth region SP _i,h in R _i , Represents the variance of the color value of the S component of the HVS color space for all pixels in the hth region SP _i,h in R _i ;

b3. Calculate the variance of the parallax magnitude of the hth region SP i in R _{i , h} , denoted as Equal to the variance of the pixel values of all pixels in the area corresponding to SP _{i, h} in d _i ;

b4. Obtain the coordinate position of the central pixel point of the hth region SP _i,h in R _i , denoted as x _i,h , where the dimension of x _i,h is 2;

b5. Calculate the area of the hth region SP _i,h in R _i , and denote it as s _i,h ;

b6, will x _{i, h} and s _{i, h} are arranged in order to form the general feature vector of the hth region SP _{i, h} in R _i , denoted as $x_{i,h}^a=[{\hat{f}}_{i,h},{\hat{c}}_{i,h},{\hat{d}}_{i,h},x_{i,h},{\mathrm{the\; s}}_{i,h}],$ in, The dimension of is 33, where the symbol “[]” is a vector symbol.

The background priori feature vector of the hth region SP _i,h in R _i in the step ①-5 The acquisition process is:

c1. Calculate the mean value of the frequency response feature vectors of all pixels in the hth area SP _i,h in R _i , denoted as f _i,h , the first in f _i,h The value of the element is equal to the h-th area SP _i,h in the frequency response feature vector of all pixels in R _i The mean value of the frequency response amplitude of elements, where the dimensions of f _i,h are 20,

c2. Calculate the mean value of the color feature vectors of all pixels in the hth area SP _i,h in R _i , denoted as c _i,h , $c_{i,h}=[{\stackrel{\‾}{R}}_{i,h},{\stackrel{\‾}{G}}_{i,h},{\stackrel{\‾}{B}}_{i,h},{\stackrel{\‾}{L}}_{i,h},{\stackrel{\‾}{a}}_{i,h},{\stackrel{\‾}{b}}_{i,h},{\stackrel{\‾}{h}}_{i,h},{\stackrel{\‾}{V}}_{i,h},{\stackrel{\‾}{S}}_{i,h}],$ Among them, the dimensions of c _{i and h} are 9, and the symbol “[]” here is a vector representation symbol, Represents the mean value of the color values of the R component of the RGB color space for all pixels in the hth region SP _i,h in R _i , Represents the mean value of the color values of the G component of the RGB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the B component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Represents the mean value of the color values of the L component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the a component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the b component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the H component of the HVS color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of all pixels in the hth region SP _i,h in R _i in the V component of the HVS color space, Represents the mean value of the color values of the S component of the HVS color space for all pixels in the hth region SP _i,h in R _i ;

c3. Calculate the mean value of the parallax magnitude of the hth region SP _{i, h} in R _i , denoted as Equal to the mean value of the pixel values of all pixels in the area corresponding to SP _{i, h} in d _i ;

c4, put f _{i, h} , c _{i, h} and Arranged in order to form the first feature vector of the hth region SP _i,h in R _i , denoted as u _i,h , Among them, the dimensions of u _{i and h} are 30, and the symbol “[]” here is a vector representation symbol;

c5. Calculate the distance between the first feature vector u _i,h of the hth area SP _i,h in R _i and the first feature vector of the background area, denoted as e _i,h , Among them, the dimensions of e _{i, h} are 30, 1≤q≤M, Represents the set of serial numbers of all background regions in R _i , u _{i, q} represents the first feature vector of the qth region SP _{i, q} in R _i , the symbol "||" is an absolute value symbol, Q represents R The total number of background regions in _i , where the background region refers to the regions located in the leftmost, rightmost, uppermost, and lowermost in R _i ;

c6. Calculate the color histogram of the R component, G component and B component of all pixels in the hth region SP _i,h in R _i in the RGB color space, denoted as Calculate the color histogram of the L component, a component and b component of all pixels in the hth area SP _{i, h} in R _i in the CIELAB color space, denoted as Calculate the color histogram of the H component of all pixels in the hth region SP _i,h in R _i in the HVS color space, denoted as Calculate the color histogram of the S component of the HVS color space for all pixels in the hth region SP _i,h in R _i , denoted as Calculate the LBP feature statistical histogram of all pixels in the hth area SP _i,h in R _i , denoted as Calculate the disparity statistical histogram of all pixels in the hth area SP _i,h in R _i , denoted as in, The dimension of is 16 ³ , The dimension of is 16 ³ , The dimension of is 16, The dimension of is 16, The dimension of is 256, The dimension of is 16;

c7, calculate The distance from the color histogram of the R component, G component and B component of the RGB color space to all pixels in the background area in R _i is denoted as

calculate The distance from the color histogram of the L component, a component and b component of the CIELAB color space to all pixels in the background area in R _i is denoted as

calculate The distance from the color histogram of the H component of the HVS color space to all pixels in the background area in R _i is denoted as

calculate The distance from the color histogram of the S component of the HVS color space to all pixels in the background area in R _i is denoted as

calculate The distance from the LBP feature statistical histogram of all pixels in the background area in R _i is denoted as

calculate The distance from the disparity statistical histogram of all pixels in the background area in R _i is denoted as

Among them, 1≤q≤M, Represent the collection of the sequence numbers of all background regions in R _i , Q represents the total number of background regions in R _i , χ () is the chi-square distance function, Represents the color histogram of the R component, G component and B component of all pixels in the qth area SP _i,q in R _i in the RGB color space, Represents the color histogram of the L component, a component and b component of all pixels in the qth region SP _{i, q} in R _i in the CIELAB color space, Represents the color histogram of the H component of all pixels in the qth region SP _i,q in R _i in the HVS color space, Represents the color histogram of the S component of the HVS color space for all pixels in the qth region SP _i,q in R _i , Represents the LBP feature statistical histogram of all pixels in the qth region SP _i,q in R _i , Represents the disparity statistical histogram of all pixels in the qth region SP _{i, q} in R _i ;

c8, put e _i,h , Arranged in order to form the background prior feature vector of the hth region SP _i,h in R _i , denoted as $x_{i,h}^b=[e_{i,h},e_{i,h}^1,e_{i,h}^2,e_{i,h}^3,e_{i,h}^4,e_{i,h}^5,e_{i,h}^6],$ in, The dimension of is 36, where the symbol “[]” is a vector symbol.

The acquisition process of the frequency response feature vector of each pixel in the R _i is:

1)-1. Use the Gabor filter to filter R _i to obtain the frequency response amplitude of each pixel in R _i at different center frequencies and different direction factors, and set the coordinate position in R _i as (x, y ) pixel at center frequency ω and direction factor θ is denoted as G(x,y; ω,θ), where (x,y) means {L _i ,R _i ,d The coordinate position of the pixel in the stereoscopic image in _i |1≤i≤N}, 1≤x≤W, 1≤y≤H, W and H correspond to {L _i , R _i , d _i |1≤i ≤ N} in the width and height of the stereo image, ω represents the center frequency of the Gabor filter, ω∈Φ _ω , θ represents the direction factor of the Gabor filter, θ∈Φ _θ , Φ _ω represents all the center frequencies of the Gabor filter The set of , Φ _θ represents the set of all direction factors of the Gabor filter;

1)-2. Arrange the frequency response amplitudes of each pixel point in R _i in different center frequencies and different direction factors in order to form the frequency response feature vector of each pixel point in R _i , and set R _i The frequency response feature vector of the pixel at the middle coordinate position (x, y) is recorded as f _i (x, y), where the dimension of f _i (x, y) is 20.

The acquisition process of the color feature vector of each pixel in the R _i is:

2)-1. Calculate the color value of each pixel point in R _i in different color spaces, and use the R component, G component and B component of the pixel point whose coordinate position is (x, y) in R _i in the RGB color space The color values of are recorded as R( _x , y), G(x, y) and B(x, y) respectively, and the L component, The color values of a component and b component are recorded as L(x, y), a(x, y) and b(x, y) respectively, and the pixel point whose coordinate position in R _i is (x, y) is in HVS color The color values of the H component, V component and S component of the space are respectively recorded as H(x, y), V(x, y) and S(x, y), where (x, y) here represents {L _i , R _i , d _i |1≤i≤N} in the coordinate position of the pixel in the stereo image, 1≤x≤W, 1≤y≤H, W and H correspond to {L _i ,R _i ,d Width and height of stereo images in _i |1≤i≤N};

2)-2. Arrange the color values of each pixel in R _i in different color spaces in order to form the color feature vector of each pixel in R _i , and set the coordinate position in R _i as (x, y) is recorded as c _i (x, y), c _i (x, y) = [R(x, y), G(x, y), B(x, y), L (x,y),a(x,y),b(x,y),H(x,y),V(x,y),S(x,y)], where c _i (x,y ) has a dimension of 9, where the symbol “[]” is a vector representation symbol.

Compared with the prior art, the present invention has the advantages of:

1) The method of the present invention simultaneously considers the contrast feature vector, general feature vector and background prior feature vector of each region in the right viewpoint image of the stereoscopic image, and fuses to obtain the use of each region in the right viewpoint image of the stereoscopic image. The method of the present invention has high extraction accuracy and strong stability, and can better reflect the significant changes of various factors, and conforms to the significant semantic features.

2) The method of the present invention constructs the random forest regression training model between the feature vector and the average eye movement value for reflecting visual salience by training, then utilizes this random forest regression training model to predict the The 3D visual saliency value of each region is obtained to obtain the 3D saliency map of the test stereo image, which effectively improves the prediction accuracy of the visual saliency value.

Description of drawings

Fig. 1 is the overall realization block diagram of the inventive method;

Figure 2a is the right view image of "Image1";

Figure 2b is the real eye movement diagram of the right viewpoint image of "Image1";

Figure 2c is the 3D saliency map of "Image1";

Figure 3a is the right view image of "Image2";

Figure 3b is the real eye movement diagram of the right viewpoint image of "Image2";

Figure 3c is the 3D saliency map of "Image2";

Figure 4a is the right view image of "Image3";

Figure 4b is the real eye movement diagram of the right viewpoint image of "Image3";

Figure 4c is the 3D saliency map of "Image3";

Figure 5a is the right view image of "Image4";

Figure 5b is the real eye movement diagram of the right viewpoint image of "Image4";

Figure 5c is the 3D saliency map of "Image4";

Figure 6a is the right view image of "Image5";

Figure 6b is the real eye movement diagram of the right viewpoint image of "Image5";

Figure 6c is the 3D saliency map of "Image5";

Figure 7a is the right view image of "Image6";

Figure 7b is the real eye movement diagram of the right viewpoint image of "Image6";

Figure 7c is the 3D saliency map of "Image6";

Figure 8a is the right view image of "Image7";

Figure 8b is the real eye movement diagram of the right viewpoint image of "Image7";

Figure 8c is the 3D saliency map of "Image7".

detailed description

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

A kind of three-dimensional image saliency map extracting method that the present invention proposes, its overall realization block diagram as shown in Figure 1, it comprises two processes of training phase and testing phase, and the concrete steps of training phase are as follows:

①-1. The selected N sets of different stereoscopic images and the right disparity images of each stereoscopic image form a set, which is recorded as {L _i , R _i , d _i |1≤i≤N}, where N ≥1, N=600 in this embodiment, L _i represents the left viewpoint image of the i-th stereoscopic image in {L _i , R _i , d _i |1≤i≤N}, and R _i represents {L _i ,R _i ,d _i |1≤i≤N} in the right view image of the i-th stereo image, d _i represents the i-th stereo in {L _i ,R _i ,d _i |1≤i≤N} Right disparity image of the image.

In this embodiment, the three-dimensional human eye tracking database (NUS 3D-Saliency database) provided by the National University of Singapore (NUS 3D-Saliency database) is used to construct a training stereoscopic image set. The stereoscopic image database contains 600 pairs of stereoscopic images and corresponding right parallax images. Realistic eye-movement maps for each stereoscopic image.

①-2. Use the existing superpixel segmentation technology to segment the right viewpoint image of each stereoscopic image in {L _i , R _i , d _i |1≤i≤N} into M non-overlapping regions, and The h-th region in R _i is denoted as SP _i,h , and the right viewpoint image of each stereo image in {L _i ,R _i ,d _i |1≤i≤N} can be re-expressed as M regions A set, the set of M regions re-expressed by R _i is denoted as {SP _i,h }; where M≥1, M=400 in this embodiment, 1≤h≤M.

①-3. Calculate the contrast feature vector of each area in the right view point image of each stereoscopic image in {L _i , R _i , d _i |1≤i≤N}, and divide the hth area in R _i The contrast feature vector of SP _{i, h} is denoted as $x_{i,h}^c=[d_{i,h}{,d}_{i,h}^1,d_{i,h}^2,d_{i,h}^3,d_{i,h}^4,d_{i,h}^5,d_{i,h}^6],$ in, The dimension of is 36, where the symbol “[]” is a vector representation symbol, and d _{i, h} represent the distance between the first feature vector u _{i, h} of SP _{i, h} and the first feature vector of the adjacent area, The dimension of f _{i, h} is 20, f _{i, h} represents the mean value of the frequency response feature vector of all pixels in SP _{i, h} , the dimension of c _{i, h} is 9, c _{i, h} represents SP _i, The mean value of the color feature vectors of all pixels in _h , Indicates the mean value of the disparity magnitude of SP _i,h , Represents the color histogram of all pixels in SP _{i, h} in the R component, G component and B component of the RGB color space and the R component, G component of all pixels in the adjacent area of SP _{i, h} in the RGB color space The distance of the color histogram of component and B component, Represents the color histogram of all pixels in SP _{i, h} in the L component, a component and b component of CIELAB color space and the L component, a of all pixels in the adjacent area of SP _{i, h} in CIELAB color space The distance of the color histogram of component and b component, Indicates the distance between the color histogram of all pixels in SP _{i, h} in the H component of the HVS color space and the color histogram of all pixels in the adjacent area of SP _{i, h} in the H component of the HVS color space, Indicates the distance between the color histogram of all pixels in SP _{i, h} in the S component of the HVS color space and the color histogram of all pixels in the adjacent area of SP _{i, h} in the S component of the HVS color space, Indicates the distance between the LBP feature statistical histogram of all pixels in SP i, _h and the LBP feature statistical histogram of all pixels in the adjacent area of SP _{i, h} , Indicates the distance between the disparity statistical histogram of all pixels in SP i, _h and the disparity statistical histogram of all pixels in the adjacent area of SP _{i, h} , where the adjacent area refers to the distance between R _i and SP The area adjacent to _{i, h} .

In this specific embodiment, the contrast feature vector of the h-th region SP _i,h in R _i in step ①-3 The acquisition process is:

a1. Calculate the mean value of the frequency response feature vectors of all pixels in the hth area SP _i,h in R _i , denoted as f _i,h , the first in f _i,h The value of the element is equal to the h-th area SP _i,h in the frequency response feature vector of all pixels in R _i The mean value of the frequency response amplitude of elements, where the dimension of f _i,h is 20,

a2. Calculate the mean value of the color feature vectors of all pixels in the hth area SP _i,h in R _i , denoted as c _i,h , $c_{i,h}=[{\stackrel{\‾}{R}}_{i,h},{\stackrel{\‾}{G}}_{i,h},{\stackrel{\‾}{B}}_{i,h},{\stackrel{\‾}{L}}_{i,h},{\stackrel{\‾}{a}}_{i,h},{\stackrel{\‾}{b}}_{i,h},{\stackrel{\‾}{h}}_{i,h},{\stackrel{\‾}{V}}_{i,h},{\stackrel{\‾}{S}}_{i,h}],$ Among them, the dimensions of c _{i and h} are 9, and the symbol “[]” here is a vector representation symbol, Represents the mean value of the color values of the R component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Represents the mean value of the color values of the G component of the RGB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the B component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Represents the mean value of the color values of the L component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the a component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Represents the mean value of the color values of the b component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Represents the mean value of the color values of the H component of the HVS color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of all pixels in the hth region SP _i,h in R _i in the V component of the HVS color space, Indicates the mean value of the color values of the S component of the HVS color space for all pixels in the hth region SP _i,h in R _i .

a3. Calculate the mean value of the parallax magnitude of the hth region SP _{i, h} in R _i , denoted as It is equal to the mean value of the pixel values of all the pixel points in the area corresponding to SP _{i, h} in d _i .

a4. Put f _{i, h} , c _{i, h} and Arranged in order to form the first feature vector of the hth area SP _i,h in R _i , denoted as u _i,h , Wherein, the dimensions of u _{i, h} are 30, and the symbol “[]” here is a vector representation symbol.

a5. Calculate the distance between the first feature vector u _i,h of the hth area SP _i,h in R _i and the first feature vector of the adjacent area, denoted as d _i,h , Among them, the dimensions of d _{i, h} are 30, 1≤p≤M, Indicates the set of serial numbers of the adjacent areas of the hth area SP _i,h in R _i , u _i,p indicates the pth area SP _i,p in R _i (SP _i,p is the value of SP _i,h Adjacent region), the symbol "||" is an absolute value symbol, and P represents the total number of adjacent regions of the hth region SP _{i, h} in R _i , which is taken in this embodiment P=20, the adjacent region here refers to the region adjacent to SP _i,h in R _i .

a6. Calculate the color histogram of the R component, G component and B component of all pixels in the hth region SP _i,h in R _i in the RGB color space, denoted as Calculate the color histogram of the L component, a component and b component of all pixels in the hth area SP _{i, h} in R _i in the CIELAB color space, denoted as Calculate the color histogram of the H component of all pixels in the hth region SP _i,h in R _i in the HVS color space, denoted as Calculate the color histogram of the S component of the HVS color space for all pixels in the hth region SP _i,h in R _i , denoted as Calculate the LBP feature statistical histogram of all pixels in the hth region SP _i,h in R _i , denoted as Calculate the disparity statistical histogram of all pixels in the hth area SP _i,h in R _i , denoted as in, The dimension of is 16 ³ , The dimension of is 16 ³ , The dimension of is 16, The dimension of is 16, The dimension of is 256, The dimension of is 16.

a7. Calculate The distance from the color histogram of the R component, G component and B component of all pixels in the adjacent area of the hth area SP _{i, h} in R _i in the RGB color space is denoted as

calculate The distance from the color histogram of the L component, a component and b component of the CIELAB color space to all pixels in the adjacent area of the hth area SP _{i, h} in R _i is denoted as

calculate The distance from the color histogram of the H component of the HVS color space to all pixels in the adjacent area of the hth area SP _i,h in R _i is denoted as

calculate The distance from the color histogram of the S component of the HVS color space to all pixels in the adjacent area of the hth area SP _i,h in R _i is denoted as

calculate The distance from the LBP feature statistical histogram of all pixels in the adjacent area of the hth area SP _i,h in R _i is denoted as

calculate The distance from the disparity statistical histogram of all pixels in the adjacent area of SP i, h in the hth area SP _{i, h} in R _i is denoted as

Among them, 1≤p≤M, Represents the set of the sequence numbers of the adjacent regions of the hth region SP _{i, h} in R _i , and P represents the total number of adjacent regions of the hth region SP _{i, h} in R _i , in this embodiment Take P=20, χ() is the function of seeking chi-square distance (Chi-distance measure), Represents the color histogram of the R component, G component and B component of all pixels in the pth area SP _i,p in R _i in the RGB color space, Represents the color histogram of the L component, a component and b component of all pixels in the pth area SP _i,p in R _i in the CIELAB color space, Represents the color histogram of the H component of the HVS color space for all pixels in the p-th region SP _i,p in R _i , Represents the color histogram of the S component of the HVS color space for all pixels in the p-th region SP _i,p in R _i , Represents the LBP feature statistical histogram of all pixels in the p-th region SP _i,p in R _i , Represents the disparity statistical histogram of all pixels in the p-th region SP _i,p in R _i .

a8, combine d _{i, h} , and Arranged in order to form the contrast feature vector of the hth region SP _i,h in R _i , denoted as $x_{i,h}^b=[e_{i,h},e_{i,h}^1,e_{i,h}^2,e_{i,h}^3,e_{i,h}^4,e_{i,h}^5,e_{i,h}^6],$ in, The dimension of is 36, where the symbol “[]” is a vector symbol.

①-4. Calculate the general feature vector of each region in the right view point image of each stereoscopic image in {L _i , R _i , d _i |1≤i≤N}, and divide the hth region in R _i The general eigenvector of SP _i,h is denoted as in, The dimension of is 33, where the symbol “[]” is a vector representation symbol, The dimension of is 20, Represents the variance of the frequency response feature vector of all pixels in SP _i,h , The dimension of is 9, Represents the variance of the color feature vectors of all pixels in SP _i,h , Represents the variance of the parallax magnitude of SP _{i, h} , the dimension of _{xi, h} is 2, _{xi, h} represents the coordinate position of the center pixel of SP _{i, h} , s _{i, h} represents the area of SP _{i, h} .

In this specific embodiment, the universal feature vector of the hth region SP _i,h in R _i in step ①-4 The acquisition process is:

b1. Calculate the variance of the frequency response feature vectors of all pixels in the hth region SP _{i, h} in R _i , denoted as in the first The value of the element is equal to the h-th area SP _i,h in the frequency response feature vector of all pixels in R _i The variance of the frequency response amplitude of elements, where, The dimension of is 20,

b2. Calculate the variance of the color feature vectors of all pixels in the hth region SP _{i, h} in R _i , denoted as ${\hat{c}}_{i,h}=[{\hat{R}}_{i,h},{\hat{G}}_{i,h},{\hat{B}}_{i,h},{\hat{L}}_{i,h},{\hat{a}}_{i,h},{\hat{b}}_{i,h},{\hat{h}}_{i,h},{\hat{V}}_{i,h},{\hat{S}}_{i,h}],$ in, The dimension of is 9, where the symbol “[]” is a vector representation symbol, Represents the variance of the color value of the R component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Indicates the variance of the color value of the G component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Indicates the variance of the color value of the B component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Represents the variance of the color value of the L component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Represents the variance of the color value of the a component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Represents the variance of the color value of the b component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Represents the variance of the color value of the H component of the HVS color space for all pixels in the hth region SP _i,h in R _i , Represents the variance of the color value of the V component of the HVS color space for all pixels in the hth region SP _i,h in R _i , Indicates the variance of the color value of the S component of the HVS color space for all pixels in the h-th region SP _i,h in R _i .

b3. Calculate the variance of the parallax magnitude of the hth region SP _{i, h} in R _i , denoted as It is equal to the variance of the pixel values of all pixels in the area corresponding to SP _{i, h} in d _i .

b4. Obtain the coordinate position of the center pixel of the hth area SP _i,h in R _i , denoted as x _i,h , where the dimension of x _i,h is 2.

b5. Calculate the area of the hth region SP _i,h in R _i , which is denoted as s _i,h .

b6, will x _{i, h} and s _{i, h} are arranged in order to form the general feature vector of the hth region SP _{i, h} in R _i , denoted as $x_{i,h}^a=[{\hat{f}}_{i,h},{\hat{c}}_{i,h},{\hat{d}}_{i,h},x_{i,h},{\mathrm{the\; s}}_{i,h}],$ in, The dimension of is 33, where the symbol “[]” is a vector symbol.

①-5. Calculate the background prior feature vector of each region in the right view point image of each stereoscopic image in {L _i , R _i , d _i |1≤i≤N}, and set the hth in R _i The background prior feature vector of a region SP _{i, h} is denoted as $x_{i,h}^b=[e_{i,h},e_{i,h}^1,e_{i,h}^2,e_{i,h}^3,e_{i,h}^4,e_{i,h}^5,e_{i,h}^6],$ in, The dimension of is 36, where the symbol “[]” is a vector representation symbol, e _i,h represents the distance between the first feature vector u _i,h of SP _i,h and the first feature vector of the background area, The dimension of f _{i, h} is 20, f _{i, h} represents the mean value of the frequency response feature vector of all pixels in SP _{i, h} , the dimension of c _{i, h} is 9, c _{i, h} represents SP _i, The mean value of the color feature vectors of all pixels in _h , Indicates the mean value of the disparity magnitude of SP _i,h , Represents the color histogram of the R component, G component and B component of all pixels in SP _{i, h} in the RGB color space and the R component, G component and B component of all pixels in the background area in R _i in the RGB color space The distance of the color histogram of the B component, Represents the color histogram of all pixels in SP _{i, h} in the L component, a component and b component of CIELAB color space and the L component, a component and b component of all pixels in the background area in R _i in CIELAB color space The distance of the color histogram of the b component, Indicates the distance between the color histogram of the H component of all pixels in SP _i,h in the H component of the HVS color space and the color histogram of the H component of all pixels in the background area in R _i in the HVS color space, Represents the distance between the color histogram of the S component of all pixels in SP _i,h in the S component of the HVS color space and the color histogram of the S component of all pixels in the background area in R _i in the HVS color space, Indicates the distance between the LBP feature statistical histogram of all pixels in SP _{i, h} and the LBP feature statistical histogram of all pixels in the background area in R _i , Indicates the distance between the disparity statistical histogram of all pixels in SP _i,h and the disparity statistical histogram of all pixels in the background area in R _{i ,} where the background area refers to the leftmost, most Right, top, and bottom areas.

In this specific embodiment, the background prior feature vector of the hth region SP _i,h in R _i in step ①-5 The acquisition process is:

c1. Calculate the mean value of the frequency response feature vectors of all the pixel points in the hth area SP _i,h in R _i , denoted as f _i,h , the first in f _i,h The value of the element is equal to the h-th area SP _i,h in the frequency response feature vector of all pixels in R _i The mean value of the frequency response amplitude of elements, where the dimension of f _i,h is 20,

c2. Calculate the mean value of the color feature vectors of all pixels in the hth area SP _i,h in R _i , denoted as c _i,h , $c_{i,h}=[{\stackrel{\‾}{R}}_{i,h},{\stackrel{\‾}{G}}_{i,h},{\stackrel{\‾}{B}}_{i,h},{\stackrel{\‾}{L}}_{i,h},{\stackrel{\‾}{a}}_{i,h},{\stackrel{\‾}{b}}_{i,h},{\stackrel{\‾}{h}}_{i,h},{\stackrel{\‾}{V}}_{i,h},{\stackrel{\‾}{S}}_{i,h}],$ Among them, the dimensions of c _{i and h} are 9, and the symbol “[]” here is a vector representation symbol, Represents the mean value of the color values of the R component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Represents the mean value of the color values of the G component of the RGB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the B component of the RGB color space for all pixels in the hth area SP _i,h in R _i , Represents the mean value of the color values of the L component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of the a component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Represents the mean value of the color values of the b component of the CIELAB color space for all pixels in the hth region SP _i,h in R _i , Represents the mean value of the color values of the H component of the HVS color space for all pixels in the hth region SP _i,h in R _i , Indicates the mean value of the color values of all pixels in the hth region SP _i,h in R _i in the V component of the HVS color space, Indicates the mean value of the color values of the S component of the HVS color space for all pixels in the hth region SP _i,h in R _i .

c3. Calculate the mean value of the parallax magnitude of the hth region SP _{i, h} in R _i , denoted as It is equal to the mean value of the pixel values of all the pixel points in the area corresponding to SP _{i, h} in d _i .

c4, put f _{i, h} , c _{i, h} and Arranged in order to form the first feature vector of the hth region SP _i,h in R _i , denoted as u _i,h , Among them, the dimensions of u _{i and h} are 30, and the symbol “[]” here is a vector representation symbol;

c5. Calculate the distance between the first feature vector u _i,h of the hth area SP _i,h in R _i and the first feature vector of the background area, denoted as e _i,h , Among them, the dimension of e _{i, h} is 30, 1≤q≤M, Represents the set of serial numbers of all background regions in R _i , u _i,q represents the first feature vector of the qth region SP _i, q in R _i (SP _i,q is the background region in R _i ), symbol "||" is the absolute value symbol, Q represents the total number of background areas in R _{i ,} where the background area refers to the leftmost, rightmost, uppermost, and lowermost areas in The leftmost, rightmost, uppermost and lowermost regions in R _i are used as background regions.

c6. Calculate the color histogram of the R component, G component and B component of all pixels in the hth region SP _i,h in R _i in the RGB color space, denoted as Calculate the color histogram of the L component, a component and b component of all pixels in the hth area SP _{i, h} in R _i in the CIELAB color space, denoted as Calculate the color histogram of the H component of all pixels in the hth region SP _i,h in R _i in the HVS color space, denoted as Calculate the color histogram of the S component of the HVS color space for all pixels in the hth region SP _i,h in R _i , denoted as Calculate the LBP feature statistical histogram of all pixels in the hth region SP _i,h in R _i , denoted as Calculate the disparity statistical histogram of all pixels in the hth area SP _i,h in R _i , denoted as in, The dimension of is 16 ³ , The dimension of is 16 ³ , The dimension of is 16, The dimension of is 16, The dimension of is 256, The dimension of is 16.

c7, calculate The distance from the color histogram of the R component, G component and B component of the RGB color space to all pixels in the background area in R _i is denoted as

calculate The distance from the color histogram of the L component, a component and b component of the CIELAB color space to all pixels in the background area in R _i is denoted as

calculate The distance from the color histogram of the H component of the HVS color space to all pixels in the background area in R _i is denoted as

calculate The distance from the color histogram of the S component of the HVS color space to all pixels in the background area in R _i is denoted as

calculate The distance from the LBP feature statistical histogram of all pixels in the background area in R _i is denoted as

calculate The distance from the disparity statistical histogram of all pixels in the background area in R _i is denoted as

Among them, 1≤q≤M, Indicates the collection of the serial numbers of all background regions in R _i , Q represents the total number of background regions in R _i , χ() is the function of seeking chi-square distance (Chi-distance measure), Represents the color histogram of the R component, G component and B component of all pixels in the qth region SP _i,q in R _i in the RGB color space, Represents the color histogram of the L component, a component and b component of all pixels in the qth region SP _{i, q} in R _i in the CIELAB color space, Represents the color histogram of the H component of all pixels in the qth region SP _i,q in R _i in the HVS color space, Represents the color histogram of the S component of the HVS color space for all pixels in the qth region SP _i,q in R _i , Represents the LBP feature statistical histogram of all pixels in the qth region SP _i,q in R _i , Represents the disparity statistical histogram of all pixels in the qth region SP _i,q in R _i .

c8, put e _i,h , and Arranged in order to form the background prior feature vector of the hth region SP _i,h in R _i , denoted as $x_{i,h}^b=[e_{i,h},e_{i,h}^1,e_{i,h}^2,e_{i,h}^3,e_{i,h}^4,e_{i,h}^5,e_{i,h}^6],$ in, The dimension of is 36, where the symbol “[]” is a vector symbol.

①-6. The contrast feature vector, general feature vector and background prior feature vector of each region in the right view image of each stereo image in {L _i , R _i , d _i |1≤i≤N} Arranged in order to form the feature vector used to reflect the visual salience of each region in the right view image of each stereoscopic image in {L _i , R _i , d _i |1≤i≤N}, R The feature vector used to reflect the visual salience of the hth region SP _{i ,h} in i is denoted as Xi _,h , Wherein, the dimensions of X _{i, h} are 105, and the symbol “[]” here is a vector representation symbol.

①-7. Use the existing random forest regression to reflect the visual salience of all regions in the right view point images of all stereoscopic images in {L _i , R _i , d _i |1≤i≤N} The feature vector is trained, and the error between the regression function value obtained through training and the average eye movement value is minimized, and the optimal random forest regression training model is obtained, which is denoted as f(D _inp ), where f() is the function Representation, D _inp represents the input vector of the random forest regression training model.

The specific steps in the testing phase are as follows:

②-1. For any pair of test stereo images S _test , record the left viewpoint image, right viewpoint image, and right disparity image of S _test as L _test , R _test , and d _test correspondingly; then use the existing superpixel segmentation technology Divide the R _test into M non-overlapping areas, record the hth area in the R _test as SP _h ', and re-express the R _test as a collection of M areas, denoted as {SP _h '}; where , M≥1, M=400 in this embodiment, 1≤h≤M.

②-2. According to the process from step ①-3 to step ①-6, obtain the feature vector used to reflect the visual salience of each region in the R _test in the same operation mode, and convert the hth region in the R _test to The feature vector of SP _h ' used to reflect visual salience is recorded as F _test,h , and the acquisition process of F _test,h is: the contrast feature vector of the hth region SP _h ' in R _test is calculated as ${x_h^c}^{\′}=[{d_h}^{\′},{d_h^1}^{\′},{d_h^2}^{\′},{d_h^3}^{\′},{d_h^4}^{\′},{d_h^5}^{\′},{d_h^6}^{\′}],$ in, The dimension of is 36, d _h ' represents the distance between the first feature vector u _h ' of SP _h ' and the first feature vector of the adjacent area, The dimension of f _h ' is 20, f _h ' represents the mean value of the frequency response feature vector of SP _h ', the dimension of c _h ' is 9, and c _h ' represents the mean value of the color feature vector of SP _h ', denote the mean of the disparity magnitudes of SP _h ', Represents the color histogram of all pixels in SP _h ' in the R component, G component and B component of the RGB color space and the R component, G component and B component of all pixels in the adjacent area of SP _h ' in the RGB color space The distance of the color histogram of the B component, Indicates the color histogram of all pixels in SP _h ' in the L component, a component and b component of CIELAB color space and the L component, a component and b component of all pixels in the adjacent area of SP _h ' in CIELAB color space The distance of the color histogram of the b component, Represents the distance between the color histogram of all pixels in SP _h ' in the H component of the HVS color space and the color histogram of all pixels in the adjacent area of SP _h ' in the H component of the HVS color space, Represents the distance between the color histogram of the S component of all pixels in the HVS color space in SP _h ' and the color histogram of the S component of all pixels in the adjacent area of SP _h ' in the HVS color space, Represents the distance between the LBP feature statistical histogram of all pixels in SP _h ' and the LBP feature statistical histogram of all pixels in the adjacent area of SP _h ', Indicates the distance between the disparity statistical histogram of all pixels in SP _h ' and the disparity statistical histogram of all pixels in the adjacent area of SP _h ', where the adjacent area refers to the area adjacent to SP _h 'region; calculate the general feature vector of the hth region SP _h ' in the R _test and denote it as ${x_h^a}^{\′}=[{{\hat{f}}_h}^{\′},{{\hat{c}}_h}^{\′},{{\hat{d}}_h}^{\′},{x_h}^{\′},{{\mathrm{the\; s}}_h}^{\′}],$ in, The dimension of is 33, The dimension of is 20, represents the variance of the frequency response eigenvector of SP _h ', The dimension of is 9, represents the variance of the color feature vector of SP _h ', Indicates the variance of the parallax magnitude of SP _h ', the dimension of x _h 'is 2, x _h 'indicates the coordinate position of the central pixel of SP _h ', s _h 'indicates the area of SP _h '; calculate the R _test in The background prior feature vector of the hth region SP _h ' is denoted as ${x_h^b}^{\′}=[{e_h}^{\′},{e_h^1}^{\′},{e_h^2}^{\′},{e_h^3}^{\′},{e_h^4}^{\′},{e_h^5}^{\′},{e_h^6}^{\′}],$ in, The dimension of is 36, e _h ' represents the distance between the first feature vector u _h ' of SP _h ' and the first feature vector of the background area, Represents the color histogram of all pixels in SP _h ' in the R component, G component and B component of the RGB color space and the color histogram of all pixels in the background area in the R component, G component and B component of the RGB color space the distance of the graph, Represents the color histogram of all pixels in SP _h ' in the L component, a component and b component of CIELAB color space and the color histogram of all pixels in the background area in CIELAB color space L component, a component and b component the distance of the graph, Represents the distance between the color histogram of the H component of all pixels in SP _h ' in the HVS color space and the color histogram of the H component of all pixels in the background area in the HVS color space, Represents the distance between the color histogram of all pixels in SP _h ' in the S component of the HVS color space and the color histogram of all pixels in the background area in the S component of the HVS color space, Indicates the distance between the LBP feature statistical histogram of all pixels in SP _h ' and the LBP feature statistical histogram of all pixels in the background area, Indicates the distance between the disparity statistical histogram of all pixels in SP _h ' and the disparity statistical histogram of all pixels in the background area, where the background area refers to the leftmost, rightmost, uppermost, and lowermost region; the contrast feature vector, general feature vector and background prior feature vector of the hth region SP _h ' in the R _test are arranged in order to form the hth region SP _h ' in the R _test to reflect the visual Significant feature vector, denoted as F _test,h , Among them, the dimension of F _{test, h} is 105; then according to the optimal random forest regression training model f(D _inp ) obtained in the training stage, F _{test, h} is used as the input vector of the optimal random forest regression training model, Obtain the 3D visual saliency value of each region in the R _test , and record the 3D visual saliency value of the hth region SP _h ' in the R _test as S _3D,h , S _3D,h = f(F _test,h ) ; Then use the 3D visual saliency value of each region in the R _test as the saliency value of all pixels in the corresponding region, such as for the hth region SP _h ' in the R _test , use S _3D,h as SP _h ' The saliency values of all the pixels in the R _test , so as to obtain the three-dimensional saliency map of the R test, which is recorded as {S _3D (x,y)}, where (x, y) represents the coordinate position of the pixel in the S _test , 1≤x≤W, 1≤y≤H, W and H correspond to the width and height of S _test , the width of S _test is consistent with the width of the stereo image selected in the training stage, and the height of S _test is the same as the stereo image selected in the training stage The heights are consistent, and S _3D (x, y) represents the pixel value of the pixel at the coordinate position (x, y) in {S _3D (x, y)}.

In this specific embodiment, the acquisition process of the frequency response feature vector of each pixel in R _i is:

1)-1. Use the Gabor filter to filter R _i to obtain the frequency response amplitude of each pixel in R _i at different center frequencies and different direction factors, and set the coordinate position in R _i as (x, y ) pixel at the center frequency ω and the direction factor θ is denoted as G(x,y; ω,θ), where (x,y) means {L _i ,R _i ,d The coordinate position of the pixel in the stereoscopic image in _i |1≤i≤N}, 1≤x≤W, 1≤y≤H, W and H correspond to {L _i , R _i , d _i |1≤i ≤ N} in the width and height of the stereo image, ω represents the center frequency of the Gabor filter, ω∈Φ _ω , θ represents the direction factor of the Gabor filter, θ∈Φ _θ , Φ _ω represents all the center frequencies of the Gabor filter The set of, in this embodiment Φ _ω ={1.74,2.47,3.49,4.93,6.98}, Φ _θ represents the set of all direction factors of the Gabor filter, in this embodiment Φ _θ ={0°,90° ,180°,270°}.

1)-2. Arrange the frequency response amplitudes of each pixel point in R _i in different center frequencies and different direction factors in order to form the frequency response feature vector of each pixel point in R _i , and set R _i The frequency response feature vector of the pixel point whose coordinate position is (x, y) is recorded as f _i (x, y), and f _i (x, y) is defined by G(x, y; 1.74,0°), G( x,y; 2.47,0°), G(x,y; 3.49,0°), G(x,y; 4.93,0°), G(x,y; 6.98,0°), G(x,y; y; 1.74,90°), G(x,y; 2.47,90°), G(x,y; 3.49,90°), G(x,y; 4.93,90°), G(x,y; 6.98,90°), G(x,y; 1.74,180°), G(x,y; 2.47,180°), G(x,y; 3.49,180°), G(x,y; 4.93, 180°), G(x,y; 6.98,180°), G(x,y; 1.74,270°), G(x,y; 2.47,270°), G(x,y; 3.49,270°) ), G(x,y; 4.93,270°), G(x,y; 6.98,270°) are arranged in order to form a vector, where the dimension of f _i (x,y) is 20.

In this specific embodiment, the acquisition process of the color feature vector of each pixel in R _i is:

2)-1. Calculate the color value of each pixel point in R _i in different color spaces, and use the R component, G component and B component of the pixel point whose coordinate position is (x, y) in R _i in the RGB color space The color values of are recorded as R( _x , y), G(x, y) and B(x, y) respectively, and the L component, The color values of a component and b component are recorded as L(x, y), a(x, y) and b(x, y) respectively, and the pixel point whose coordinate position in R _i is (x, y) is in HVS color The color values of the H component, V component and S component of the space are respectively recorded as H(x, y), V(x, y) and S(x, y), where (x, y) here represents {L _i , R _i , d _i |1≤i≤N} in the coordinate position of the pixel in the stereo image, 1≤x≤W, 1≤y≤H, W and H correspond to {L _i ,R _i ,d Width and height of stereo images in _i |1≤i≤N}.

2)-2. Arrange the color values of each pixel in R _i in different color spaces in order to form the color feature vector of each pixel in R _i , and set the coordinate position in R _i as (x, y) is recorded as c _i (x, y), c _i (x, y) = [R(x, y), G(x, y), B(x, y), L (x,y),a(x,y),b(x,y),H(x,y),V(x,y),S(x,y)], where c _i (x,y ) has a dimension of 9, where the symbol “[]” is a vector representation symbol.

Below, the three-dimensional saliency maps of Image1, Image2, Image3, Image4 and Image5 five stereoscopic images in the three-dimensional human eye tracking database (3Deye-tracking database) provided by the University of Nantes in France are extracted by using the method of the present invention. Figure 2a shows the right view image of "Image1", Figure 2b shows the real eye movement image of the right view image of "Image1", Figure 2c shows the 3D saliency map of "Image1"; Figure 3a shows the " The right view image of Image2", Figure 3b shows the real eye movement image of the right view image of "Image2", Figure 3c shows the 3D saliency map of "Image2"; Figure 4a shows the right view image of "Image3" , Figure 4b shows the real eye movement image of the right-viewpoint image of "Image3", and Figure 4c shows the 3D saliency map of "Image3"; Figure 5a shows the right-viewpoint image of "Image4", and Figure 5b shows the The real eye movement image of the right view point image of "Image4", Fig. 5c shows the 3D saliency map of "Image4"; Fig. 6a shows the right view point image of "Image5", and Fig. 6b shows the right view point of "Image5" The real eye movement diagram of the image, Figure 6c shows the 3D saliency map of "Image5"; Figure 7a shows the right viewpoint image of "Image6", and Fig. 7b shows the real eye movement diagram of the right viewpoint image of "Image6" , Figure 7c shows the 3D saliency map of "Image6"; Figure 8a shows the right view image of "Image7", Figure 8b shows the real eye movement image of the right view image of "Image7", and Figure 8c shows The 3D saliency map of "Image7". It can be seen from Figures 2a to 8c that the 3D saliency map obtained by the method of the present invention can well conform to the features of saliency semantics because the contrast feature, general feature and background prior feature are considered.

Claims (6)

1. A method for extracting a three-dimensional image saliency map is characterized by comprising a training stage and a testing stage, wherein the training stage comprises the following specific steps:

① -1, the selected N sets of different stereo images and the right parallax image of each stereo image form a set, which is marked as { L_i,R_i,d_iI is more than or equal to 1 and less than or equal to N, wherein N is more than or equal to 1, and L_iRepresents { L_i,R_i,d_iL 1. ltoreq. i. ltoreq.N } left viewpoint image of ith stereoscopic image, R_iRepresents { L_i,R_i,d_iI is not less than 1 and not more than N, d_iRepresents { L_i,R_i,d_iI is not less than 1 and not more than N, and right parallax images of the ith stereo image;

① -2, using superpixel splitting technique to divide { L_i,R_i,d_iI is more than or equal to 1 and less than or equal to N, the right viewpoint image of each stereo image is divided into M non-overlapping areas, R is_iThe h-th area in (1) is denoted as SP_i,hWherein M is more than or equal to 1, h is more than or equal to 1 and less than or equal to M;

① -3, calculation of { L_i,R_i,d_iI is not less than 1 and not more than N, R is the contrast characteristic vector of each region in the right viewpoint image of each stereoscopic image_iH-th area SP of (1)_i,hIs recorded as a contrast feature vector Wherein,has a dimension of 36, here the symbol "[ 2 ]]"is a vector representing a symbol, d_i,hRepresents SP_i,hFirst feature vector u of_i,hThe distance from the first feature vector of the neighboring area,f_i,hhas a dimension of 20, f_i,hRepresents SP_i,hC mean value of the frequency response feature vectors of all the pixels in (1)_i,hHas a dimension of 9, c_i,hRepresents SP_i,hThe mean of the color feature vectors of all the pixel points in (a),represents SP_i,hIs measured in the mean value of the parallax amplitude of (c),represents SP_i,hColor histograms of R, G and B components of all pixel points in RGB color space and SP_i,hThe distances of all pixel points in the adjacent regions of the color histogram of the R component, the G component and the B component of the RGB color space,represents SP_i,hThe color histograms of L component, a component and b component of all pixel points in CIELAB color space and SP_i,hThe distances of the color histograms of the L component, the a component and the b component of the CIELAB color space of all the pixel points in the neighboring region of (a),represents SP_i,hColor histogram and SP of H component of all pixel points in HVS color space_i,hAll pixel points in the neighboring region of (a) are at the distance of the color histogram of the H component of the HVS color space,represents SP_i,hColor histogram of S component of all pixel points in HVS color space and SP_i,hThe distance of all pixel points in the neighboring region of (a) in the color histogram of the S component of the HVS color space,represents SP_i,hLBP feature statistical histogram and SP of all pixel points in_i,hThe distance of the statistical histogram of the LBP features of all pixel points in the neighboring region,represents SP_i,hThe parallax statistical histogram and SP of all the pixel points in_i,hThe distance of the parallax statistical histogram of all the pixel points in the adjacent region, where the adjacent region is R_iNeutral SP_i,hAn adjacent region;

① -4, calculation of { L_i,R_i,d_iI is not less than 1 and not more than N, R is the general feature vector of each region in the right viewpoint image of each stereoscopic image_iH-th area SP of (1)_i,hIs given as Wherein,has a dimension of 33, here the symbol "[ 2 ]]"is a vector representing a symbol and,has a dimension of 20 a and has a high degree of,represents SP_i,hThe variance of the frequency response feature vectors of all the pixel points in (a),has a dimension of 9 a and has a high degree of,represents SP_i,hThe variance of the color feature vectors of all the pixel points in (a),represents SP_i,hOf the parallax amplitude, x_i,hHas a dimension of 2, x_i,hRepresents SP_i,hS of the center pixel point_i,hRepresents SP_i,hThe area of (d);

① -5, calculating { L_i,R_i,d_iI 1 ≦ i ≦ N } for each region in the right view image for each stereoscopic imageA background prior feature vector of (2), R_iH-th area SP of (1)_i,hIs recorded as Wherein,has a dimension of 36, here the symbol "[ 2 ]]"is a vector representing a symbol, e_i,hRepresents SP_i,hFirst feature vector u of_i,hThe distance from the first feature vector of the background region,f_i,hhas a dimension of 20, f_i,hRepresents SP_i,hC mean value of the frequency response feature vectors of all the pixels in (1)_i,hHas a dimension of 9, c_i,hRepresents SP_i,hThe mean of the color feature vectors of all the pixel points in (a),represents SP_i,hIs measured in the mean value of the parallax amplitude of (c),represents SP_i,hThe color histogram of R component, G component and B component of all pixel points in RGB color space and R_iThe distances of all the pixel points in the background region in the color histogram of the R component, G component and B component of the RGB color space,represents SP_i,hThe color histograms of L component, a component and b component and R of all pixel points in CIELAB color space_iOf all pixels in the background region in CIELAB color spaceThe distances of the color histograms of the L component, the a component and the b component,represents SP_i,hColor histogram and R of H component of all pixel points in HVS color space_iThe distance of all the pixel points in the background region in the color histogram of the H component of the HVS color space,represents SP_i,hColor histogram of S component of all pixel points in HVS color space and R_iThe distance of all pixel points in the background region in the color histogram of the S component of the HVS color space,represents SP_i,hLBP feature statistical histogram and R of all pixel points in_iThe distance of the statistical histogram of the LBP features of all the pixels in the background region in (1),represents SP_i,hThe parallax statistical histogram and R of all the pixel points in_iThe distance of the parallax statistical histogram of all the pixel points in the background region is R_iThe areas positioned at the leftmost side, the rightmost side, the uppermost side and the lowermost side;

① -6, will { L_i,R_i,d_iI is more than or equal to 1 and less than or equal to N), the contrast characteristic vector, the general characteristic vector and the background prior characteristic vector of each region in the right viewpoint image of each stereo image are arranged in sequence to form { L ≦_i,R_i,d_iI 1 ≦ i ≦ N } for each region in the right view image of each stereoscopic image, R is a feature vector for reflecting visual saliency_iH-th area SP of (1)_i,hThe feature vector for reflecting the visual saliency is marked as X_i,h，Wherein, X_i,hHas a dimension of 105, here the symbol "[ 2 ]]"is a vector representation symbol;

① -7, using random forest regression, on { L }_i,R_i,d_iI is more than or equal to 1 and less than or equal to N), training the feature vectors for reflecting the visual saliency in all the regions of the right viewpoint images of all the stereo images, minimizing the error between the regression function value obtained through training and the average eye movement value, obtaining the optimal random forest regression training model, and marking the optimal random forest regression training model as f (D)_inp) Wherein f () is a functional representation form, D_inpRepresenting an input vector of a random forest regression training model;

the specific steps of the test stage are as follows:

② -1, for any one test stereo image S_testWill S_testThe left viewpoint image, the right viewpoint image, and the right parallax image are expressed as L_test、R_test、d_test(ii) a Then adopting super pixel segmentation technique to divide R_testDividing into M non-overlapping regions, and dividing R_testThe h-th area in (1) is denoted as SP_h'; wherein M is more than or equal to 1, h is more than or equal to 1 and less than or equal to M;

② -2, following the procedure of step ① -3 through step ① -6, R is obtained in the same manner of operation_testFor each region of (a) to reflect visual saliency, R_testH-th area SP of (1)_h' the feature vector for reflecting the visual saliency is denoted as F_test,h(ii) a Then, training a model f (D) according to the optimal random forest regression obtained in the training stage_inp) Will F_test,hObtaining R as an input vector of an optimal random forest regression training model_testOf each region of (a), R_testH-th area SP of (1)_h' the three-dimensional visual saliency value is denoted as S_3D,h，S_3D,h＝f(F_test,h) (ii) a Then R is put_testThe three-dimensional visual saliency value of each region in the R-image is taken as the saliency value of all pixel points in the corresponding region, thereby obtaining R_testIII of (2)Dimension saliency map, denoted as { S_3D(x, y) }, wherein (x, y) here denotes S_testX is more than or equal to 1 and less than or equal to W, y is more than or equal to 1 and less than or equal to H, and W and H correspondingly represent S_testWidth and height of (S)_3D(x, y) represents S_3DAnd the coordinate position in the (x, y) is the pixel value of the pixel point of (x, y).

2. The method for extracting saliency map of stereoscopic images as claimed in claim 1, wherein said step ① -3R_iH-th area SP of (1)_i,hContrast feature vector ofThe acquisition process comprises the following steps:

a1, calculating R_iH-th area SP of (1)_i,hThe mean value of the frequency response characteristic vectors of all the pixel points is recorded as f_i,h，f_i,hTo (1)The value of each element is equal to R_iH-th area SP of (1)_i,hIn the frequency response feature vector of all the pixel pointsMean of the frequency response amplitudes of the elements, where f_i,hHas a dimension of 20 a and has a high degree of,

a2, calculating R_iH-th area SP of (1)_i,hThe mean value of the color feature vectors of all the pixel points in (1) is marked as c_i,h，Wherein, c_i,hHas a dimension of 9, here the symbol "[ 2 ]]"is a vector representing a symbol and,represents R_iH-th area SP of (1)_i,hThe mean value of the color values of the R components of the RGB color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i,hThe average of the color values of the G components of the RGB color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i,hThe average of the color values of the B components of the RGB color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i,hThe average value of the color values of the L components of all the pixel points in the CIELAB color space,represents R_iH-th area SP of (1)_i,hThe average value of the color values of the a component of all the pixel points in the CIELAB color space,represents R_iH-th area SP of (1)_i,hThe average value of the color values of the b components of all the pixel points in the CIELAB color space,represents R_iThe h region SP of_i,hThe average of the color values of the H component of the HVS color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i,hAll the pixel points inThe mean of the color values of the V component in the HVS color space,represents R_iH-th area SP of (1)_i,hThe mean value of the color values of the S components of all the pixel points in the HVS color space;

a3, calculating R_iH-th area SP of (1)_i,hIs the mean of the parallax amplitude of Is equal to d_iNeutral SP_i,hThe mean value of the pixel values of all the pixel points in the corresponding region;

a4, mixing f_i,h、c_i,hAndarranged in sequence to form R_iH-th area SP of (1)_i,hIs recorded as a first feature vector ofWherein u is_i,hHas a dimension of 30, here the symbol "[ 2 ]]"is a vector representation symbol;

a5, calculating R_iH-th area SP of (1)_i,hFirst feature vector u of_i,hThe distance from the first feature vector of the neighboring region, denoted d_i,h，Wherein d is_i,hHas a dimension of 30, p is more than or equal to 1 and less than or equal to M,represents R_iH-th area SP of (1)_i,hU of adjacent areas of the sequence number_i,pRepresents R_iP-th area SP in (1)_i,pThe symbol "|" is an absolute value symbol, and P represents R_iH-th area SP of (1)_i,hIs referred to as R_iNeutral SP_i,hAn adjacent region;

a6, calculating R_iH-th area SP of (1)_i,hThe color histograms of R, G and B components of all the pixel points in RGB color space are recorded asCalculation of R_iH-th area SP of (1)_i,hThe color histograms of the L component, the a component and the b component of all the pixel points in the CIELAB color space are recorded asCalculation of R_iH-th area SP of (1)_i,hThe color histogram of the H component of all the pixel points in the HVS color space is recorded asCalculation of R_iH-th area SP of (1)_i,hThe color histogram of S component of all pixel points in HVS color space is recorded asCalculation of R_iH-th area SP of (1)_i,hThe statistical histogram of the LBP characteristics of all the pixel points in (1) is recorded asCalculation of R_iH-th area SP of (1)_i,hThe parallax statistical histogram of all the pixel points in (1) is recorded asWherein,has dimension of 16³，Has dimension of 16³，Has a dimension of 16 a and has a high degree of,has a dimension of 16 a and has a high degree of,has a dimension of 256, and has a high thermal conductivity,has a dimension of 16;

a7, calculatingAnd R_iH-th area SP of (1)_i,hThe distances of the color histograms of the R component, the G component and the B component of the RGB color space of all the pixel points in the adjacent areas are recorded as

ComputingAnd R_iH-th area SP of (1)_i,hThe distances of the color histograms of the L component, the a component and the b component of all the pixel points in the adjacent areas in the CIELAB color space are recorded as

ComputingAnd R_iH-th area SP of (1)_i,hThe distance of the color histogram of the H component of the HVS color space of all the pixel points in the adjacent region is recorded as

ComputingAnd R_iH-th area SP of (1)_i,hThe distance of the color histogram of the S component of the HVS color space of all the pixel points in the adjacent region is recorded as

ComputingAnd R_iH-th area SP of (1)_i,hThe distance of the LBP feature statistical histogram of all the pixel points in the adjacent region is recorded as

ComputingAnd R_iH-th area SP of (1)_i,hThe distance of the parallax statistical histogram of all the pixel points in the adjacent region is recorded as

Wherein p is more than or equal to 1 and less than or equal to M,represents R_iH-th area SP of (1)_i,hP represents R_iH-th area SP of (1)_i,hIs the chi () function of the chi-squared distance,represents R_iP-th area SP in (1)_i,pThe color histograms of the R, G and B components of all the pixel points in the RGB color space,represents R_iP-th area SP in (1)_i,pThe color histograms of the L component, the a component and the b component of all the pixel points in the CIELAB color space,represents R_iP-th area SP in (1)_i,pThe color histogram of the H component of the HVS color space for all the pixel points in (a),represents R_iP-th area SP in (1)_i,pColor of S component of all pixel points in HVS color spaceThe color histogram is a histogram of the color,represents R_iP-th area SP in (1)_i,pThe statistical histogram of LBP features of all pixel points in (1),represents R_iP-th area SP in (1)_i,pThe parallax statistics histogram of all the pixel points in the image;

a8, d_i,h、Andarranged in sequence to form R_iH-th area SP of (1)_i,hIs recorded as the contrast feature vector of Wherein,has a dimension of 36, here the symbol "[ 2 ]]"is a vector representing a symbol.

3. The method for extracting saliency map of stereoscopic images as claimed in claim 2, wherein said step ① -4R_iH-th area SP of (1)_i,hIs a universal feature vectorThe acquisition process comprises the following steps:

b1, calculating R_iH-th area SP of (1)_i,hFrequency response characteristics of all pixel points inVariance of vector, noted To (1)The value of each element is equal to R_iH-th area SP of (1)_i,hIn the frequency response feature vector of all the pixel pointsThe variance of the frequency response amplitude of the individual elements, wherein,has a dimension of 20 a and has a high degree of,

b2, calculating R_iH-th area SP of (1)_i,hThe variance of the color feature vectors of all the pixel points is recorded as Wherein,has a dimension of 9, here the symbol "[ 2 ]]"is a vector representing a symbol and,represents R_iH-th area SP of (1)_i，hThe variance of the color values of the R component of the RGB color space for all pixel points in (a),represents R_iH-th area SP of (1)_i,hThe variance of the color values of the G component of the RGB color space for all pixel points in (a),represents R_iH-th area SP of (1)_i,hThe variance of the color values of the B component of the RGB color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i,hThe variance of the color values of the L component of the CIELAB color space for all pixel points in (a),represents R_iH-th area SP of (1)_i,hThe variance of the color values of the a component of the CIELAB color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i,hThe variance of the color values of the b-component in CIELAB color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i,hThe variance of the color values of the H component of the HVS color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i,hThe variance of the color values of the V component of the HVS color space for all pixel points in (a),represents R_iH of (1)Region SP_i,hThe variance of the color values of the S component of all the pixel points in the HVS color space;

b3, calculating R_iH-th area SP of (1)_i,hThe variance of the parallax amplitude of (1), is recorded as Is equal to d_iNeutral SP_i,hVariance of pixel values of all pixel points in the corresponding region;

b4, obtaining R_iH-th area SP of (1)_i,hThe coordinate position of the central pixel point is marked as x_i,hWherein x is_i,hHas a dimension of 2;

b5, calculating R_iH-th area SP of (1)_i,hArea of (d), denoted as s_i,h；

b6, willx_i,hAnd s_i,hArranged in sequence to form R_iH-th area SP of (1)_i,hIs a universal feature vector of Wherein,has a dimension of 33, here the symbol "[ 2 ]]"is a vector representing a symbol.

4. The method for extracting saliency map of stereoscopic images as claimed in claim 3, wherein said step ① -5R_iH-th area SP of (1)_i,hBackground prior feature vector ofThe acquisition process comprises the following steps:

c1, calculating R_iH-th area SP of (1)_i,hThe mean value of the frequency response characteristic vectors of all the pixel points is recorded as f_i,h，f_i,hTo (1)The value of each element is equal to R_iH-th area SP of (1)_i,hIn the frequency response feature vector of all the pixel pointsMean of the frequency response amplitudes of the elements, where f_i,hHas a dimension of 20 a and has a high degree of,

c2, calculating R_iH-th area SP of (1)_i,hThe mean value of the color feature vectors of all the pixel points in (1) is marked as c_i,h，Wherein, c_i,hHas a dimension of 9, here the symbol "[ 2 ]]"is a vector representing a symbol and,represents R_iH-th area SP of (1)_i,hThe mean value of the color values of the R components of the RGB color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i,hThe average of the color values of the G components of the RGB color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i,hThe average of the color values of the B components of the RGB color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i，hThe average value of the color values of the L components of all the pixel points in the CIELAB color space,represents R_iH-th area SP of (1)_i,hThe average value of the color values of the a component of all the pixel points in the CIELAB color space,represents R_iH-th area SP of (1)_i,hThe average value of the color values of the b components of all the pixel points in the CIELAB color space,represents R_iThe h region SP of_i,hThe average of the color values of the H component of the HVS color space for all the pixel points in (a),represents R_iH-th area SP of (1)_i,hThe average of the color values of the V components of all the pixel points in the HVS color space,represents R_iH-th area SP of (1)_i,hThe mean value of the color values of the S components of all the pixel points in the HVS color space;

c3, calculating R_iH-th area SP of (1)_i,hIs the mean of the parallax amplitude of Is equal to d_iNeutral SP_i,hThe mean value of the pixel values of all the pixel points in the corresponding region;

c4, mixing f_i,h、c_i,hAndarranged in sequence to form R_iH-th area SP of (1)_i,hFirst feature vector of (1), denoted as u_i,h，Wherein u is_i,hHas a dimension of 30, here the symbol "[ 2 ]]"is a vector representation symbol;

c5, calculating R_iH-th area SP of (1)_i,hFirst feature vector u of_i,hThe distance from the first feature vector of the background region, denoted as e_i,h，Wherein e is_i,hHas a dimension of 30, q is more than or equal to 1 and less than or equal to M,represents R_iSet of sequence numbers of all background regions in u_i,qRepresents R_iThe q-th region SP in (1)_i,qThe symbol "|" is an absolute value symbol, and Q represents R_iThe total number of background regions in (1), where the background region is R_iThe areas positioned at the leftmost side, the rightmost side, the uppermost side and the lowermost side;

c6, calculating R_iH-th area SP of (1)_i,hThe color histograms of R, G and B components of all the pixel points in RGB color space are recorded asCalculation of R_iH-th area SP of (1)_i,hThe color histograms of the L component, the a component and the b component of all the pixel points in the CIELAB color space are recorded asCalculation of R_iH-th area SP of (1)_i,hThe color histogram of the H component of all the pixel points in the HVS color space is recorded asCalculation of R_iH-th area SP of (1)_i,hThe color histogram of S component of all pixel points in HVS color space is recorded asCalculation of R_iH-th area SP of (1)_i,hThe statistical histogram of the LBP characteristics of all the pixel points in (1) is recorded asCalculation of R_iH-th area SP of (1)_i,hThe parallax statistical histogram of all the pixel points in (1) is recorded asWherein,has dimension of 16³，Has dimension of 16³，Has a dimension of 16 a and has a high degree of,has a dimension of 16 a and has a high degree of,has a dimension of 256, and has a high thermal conductivity,has a dimension of 16;

c7, calculationAnd R_iThe distances of the color histograms of the R component, the G component and the B component of the RGB color space of all the pixel points in the background region are recorded as

ComputingAnd R_iThe distances of the color histograms of the L component, the a component and the b component of all the pixel points in the background area in the CIELAB color space are recorded as the distances

ComputingAnd R_iThe distance of the color histogram of the H component of the HVS color space of all the pixel points in the background area is recorded as

ComputingAnd R_iThe distance of the color histogram of the S component of the HVS color space of all the pixel points in the background area is recorded as

ComputingAnd R_iThe distance of the LBP feature statistical histogram of all the pixel points in the background area is recorded as

ComputingAnd R_iThe distance of the parallax statistical histogram of all the pixel points in the background region is recorded as

Wherein q is more than or equal to 1 and less than or equal to M,represents R_iAll ofSet of sequence numbers of background regions, Q denotes R_iThe total number of background regions, χ () is a chi-squared distance function,represents R_iThe q-th region SP in (1)_i,qThe color histograms of the R, G and B components of all the pixel points in the RGB color space,represents R_iThe q-th region SP in (1)_i,qThe color histograms of the L component, the a component and the b component of all the pixel points in the CIELAB color space,represents R_iThe q-th region SP in (1)_i,qThe color histogram of the H component of the HVS color space for all the pixel points in (a),represents R_iThe q-th region SP in (1)_i,qThe color histogram of the S component of the HVS color space for all the pixel points in (a),represents R_iThe q-th region SP in (1)_i,qThe statistical histogram of LBP features of all pixel points in (1),represents R_iThe q-th region SP in (1)_i,qThe parallax statistics histogram of all the pixel points in the image;

c8, mixing_i,h、Andarranged in sequence to form R_iH-th area SP of (1)_i,hIs recorded as the background prior feature vector of Wherein,has a dimension of 36, here the symbol "[ 2 ]]"is a vector representing a symbol.

5. The method according to any one of claims 2 to 4, wherein R is the number of pixels in the R-map_iThe obtaining process of the frequency response characteristic vector of each pixel point comprises the following steps:

1) -1, using a Gabor filter pair R_iFiltering to obtain R_iUnder different central frequencies and different direction factors, each pixel point in the signal is subjected to frequency response amplitude R_iThe frequency response amplitude of the pixel point with the middle coordinate position (x, y) under the condition that the center frequency is omega and the direction factor is theta is marked as G (x, y; omega, theta), wherein (x, y) represents { L }_i,R_i,d_iI is more than or equal to 1 and less than or equal to N, x is more than or equal to 1 and less than or equal to W, y is more than or equal to 1 and less than or equal to H, and W and H correspondingly represent { L ≦ N }_i,R_i,d_iI 1 ≦ i ≦ N }, ω represents the center frequency of the Gabor filter, ω ∈ Φ_ωθ represents the directional factor of the Gabor filter, θ ∈ Φ_θ，Φ_ωRepresenting the set of all centre frequencies of the Gabor filter, phi_θRepresents the set of all directional factors of the Gabor filter;

1) -2, reacting R_iThe frequency response amplitude of each pixel point in the R-array is arranged in sequence under different central frequencies and different direction factors to form R_iThe frequency response feature vector of each pixel point in (a),r is to be_iThe frequency response characteristic vector of the pixel point with the middle coordinate position (x, y) is marked as f_i(x, y) wherein f_iThe dimension of (x, y) is 20.

6. The method as claimed in claim 5, wherein R is a number of terms_iThe obtaining process of the color feature vector of each pixel point comprises the following steps:

2) -1, calculating R_iThe color value of each pixel point in different color spaces is R_iThe color values of the R component, the G component and the B component of the pixel point with the (x, y) middle coordinate position in the RGB color space are respectively recorded as R (x, y), G (x, y) and B (x, y), and R (x, y) is recorded_iThe color values of the L component, the a component and the b component of the pixel point with the middle coordinate position (x, y) in the CIELAB color space are respectively recorded as L (x, y), a (x, y) and b (x, y), and R is used for indicating the color values of the L component, the a component and the b component in the CIELAB color space as L (x, y), a (x, y) and b (_iThe color values of the H component, the V component and the S component of the pixel point with the (x, y) middle coordinate position in the HVS color space are respectively recorded as H (x, y), V (x, y) and S (x, y), wherein (x, y) here represents { L }_i,R_i,d_iI is more than or equal to 1 and less than or equal to N, x is more than or equal to 1 and less than or equal to W, y is more than or equal to 1 and less than or equal to H, and W and H correspondingly represent { L ≦ N }_i,R_i,d_iThe width and height of the stereoscopic image in the condition that i is more than or equal to 1 and less than or equal to N;

2) -2, reacting R_iThe color values of each pixel point in the different color spaces are arranged in sequence to form R_iThe color feature vector of each pixel point in (1), R_iThe color feature vector of the pixel point with the middle coordinate position (x, y) is marked as c_i(x,y)，c_i(x,y)＝[R(x,y),G(x,y),B(x,y),L(x,y),a(x,y),b(x,y),H(x,y),V(x,y),S(x,y)]Wherein c is_iThe dimension of (x, y) is 9, where the symbol "[ 2 ],"]"is a vector representing a symbol.