CN104822058B - A kind of stereo-picture notable figure extracting method - Google Patents

A kind of stereo-picture notable figure extracting method Download PDF

Info

Publication number
CN104822058B
CN104822058B CN201510173098.5A CN201510173098A CN104822058B CN 104822058 B CN104822058 B CN 104822058B CN 201510173098 A CN201510173098 A CN 201510173098A CN 104822058 B CN104822058 B CN 104822058B
Authority
CN
China
Prior art keywords
component
area
color
pixel points
color space
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201510173098.5A
Other languages
Chinese (zh)
Other versions
CN104822058A (en
Inventor
邵枫
姜求平
李福翠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhonghe International Intellectual Property Co Ltd
Original Assignee
Ningbo University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ningbo University filed Critical Ningbo University
Priority to CN201510173098.5A priority Critical patent/CN104822058B/en
Publication of CN104822058A publication Critical patent/CN104822058A/en
Application granted granted Critical
Publication of CN104822058B publication Critical patent/CN104822058B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Landscapes

  • Image Analysis (AREA)

Abstract

本发明公开了一种立体图像显著图提取方法,其在训练阶段,提取多幅训练立体图像的右视点图像中的每个区域的对比度特征矢量、通用特征矢量和背景先验特征矢量,并融合得到每幅立体图像的右视点图像中的每个区域的用于反映视觉显著性的特征矢量,建立特征矢量与平均眼动值之间的随机森林回归训练模型;在测试阶段,计算测试立体图像的右视点图像中的每个区域的用于反映视觉显著性的特征矢量,并根据已训练得到的随机森林回归训练模型,预测得到测试立体图像的三维显著图;优点是所提取的特征能够较好地反映各种因素的显著变化情况,从而有效地提高了视觉显著值的预测准确性。

The invention discloses a method for extracting a saliency map of a stereoscopic image. In the training stage, the contrast feature vector, the general feature vector and the background prior feature vector of each area in the right viewpoint image of multiple training stereo images are extracted and fused. Obtain the feature vector used to reflect the visual salience of each region in the right view point image of each stereo image, and establish a random forest regression training model between the feature vector and the average eye movement value; in the test phase, calculate the test stereo image The feature vector used to reflect the visual saliency of each region in the right viewpoint image, and according to the trained random forest regression training model, predict the 3D saliency map of the test stereo image; the advantage is that the extracted features can be compared with It can well reflect the significant changes of various factors, thus effectively improving the prediction accuracy of visually significant values.

Description

一种立体图像显著图提取方法A method for extracting saliency map from stereo images

技术领域technical field

本发明涉及一种图像信号的处理方法,尤其是涉及一种立体图像显著图提取方法。The invention relates to a method for processing image signals, in particular to a method for extracting saliency maps of stereoscopic images.

背景技术Background technique

在人类视觉接收与信息处理中,由于大脑资源有限以及外界环境信息重要性区别,因此在处理过程中人脑对外界环境信息并不是一视同仁的,而是表现出选择特征。人们在观看图像或者视频片段时注意力并非均匀分布到图像的每个区域,而是对某些显著区域关注度更高。如何将视频中视觉注意度高的显著区域检测并提取出来是计算机视觉以及基于内容的视频检索领域的一个重要的研究内容。In human visual reception and information processing, due to limited brain resources and differences in the importance of external environmental information, the human brain does not treat external environmental information equally in the processing process, but shows selective characteristics. When people watch images or video clips, their attention is not evenly distributed to every area of the image, but they pay more attention to certain salient areas. How to detect and extract salient regions with high visual attention in videos is an important research content in the field of computer vision and content-based video retrieval.

然而,人眼感知立体图像产生立体视觉的过程并不是简单的左视点图像和右视点图像叠加的过程,因此,立体视觉特征(例如:三维视觉注意力)并不是平面视觉特性的简单拓展,如何从立体图像中有效地提取出立体视觉特征、如何使得提取出的立体视觉特征符合人眼三维观看行为,都是在对立体图像进行视觉显著图提取过程中需要研究解决的问题。However, the process of human perception of stereoscopic images to produce stereoscopic vision is not a simple process of superimposing left-viewpoint images and right-viewpoint images. Therefore, stereoscopic vision features (such as: three-dimensional visual attention) are not simply an extension of planar vision characteristics. How to effectively extract stereoscopic features from stereoscopic images and how to make the extracted stereoscopic features conform to the three-dimensional viewing behavior of human eyes are all problems that need to be studied and solved in the process of extracting visual saliency maps from stereoscopic images.

发明内容Contents of the invention

本发明所要解决的技术问题是提供一种立体图像显著图提取方法,其符合显著语义特征,且具有较强的提取稳定性和较高的提取准确性。The technical problem to be solved by the present invention is to provide a method for extracting saliency maps of stereoscopic images, which conforms to saliency semantic features, and has strong extraction stability and high extraction accuracy.

本发明解决上述技术问题所采用的技术方案为:一种立体图像显著图提取方法,其特征在于包括训练阶段和测试阶段两个过程,所述的训练阶段的具体步骤如下:The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a method for extracting a saliency map of a stereoscopic image, which is characterized in that it includes two processes of a training phase and a testing phase, and the specific steps of the training phase are as follows:

①-1、将选取的N副各不相同的立体图像以及每幅立体图像的右视差图像构成一个集合,记为{Li,Ri,di|1≤i≤N},其中,N≥1,Li表示{Li,Ri,di|1≤i≤N}中的第i幅立体图像的左视点图像,Ri表示{Li,Ri,di|1≤i≤N}中的第i幅立体图像的右视点图像,di表示{Li,Ri,di|1≤i≤N}中的第i幅立体图像的右视差图像;①-1. The selected N sets of different stereoscopic images and the right disparity images of each stereoscopic image form a set, which is recorded as {L i , R i , d i |1≤i≤N}, where N ≥1, L i represents the left viewpoint image of the i-th stereo image in {L i , R i , d i |1≤i≤N}, and R i represents {L i , R i , d i |1≤i ≤N} in the right viewpoint image of the i-th stereo image, d i represents the right parallax image of the i-th stereo image in {L i , R i , d i |1≤i≤N};

①-2、采用超像素分割技术将{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像分割成M个互不重叠的区域,将Ri中的第h个区域记为SPi,h,其中,M≥1,1≤h≤M;①-2. Use superpixel segmentation technology to divide the right viewpoint image of each stereoscopic image in {L i , R i , d i |1≤i≤N} into M non-overlapping regions, and divide R i The hth region of is denoted as SP i,h , where M≥1, 1≤h≤M;

①-3、计算{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像中的每个区域的对比度特征矢量,将Ri中的第h个区域SPi,h的对比度特征矢量记为 X i , h c = [ d i , h , d i , h 1 , d i , h 2 , d i , h 3 , d i , h 4 , d i , h 5 , d i , h 6 ] , 其中,的维数为36,此处符号“[]”为矢量表示符号,di,h表示SPi,h的第一特征矢量ui,h与相邻区域的第一特征矢量的距离,fi,h的维数为20,fi,h表示SPi,h中的所有像素点的频率响应特征矢量的均值,ci,h的维数为9,ci,h表示SPi,h中的所有像素点的颜色特征矢量的均值,di,h表示SPi,h的视差幅值的均值,表示SPi,h中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图与SPi,h的相邻区域中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图的距离,表示SPi,h中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图与SPi,h的相邻区域中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图的距离,表示SPi,h中的所有像素点在HVS颜色空间的H分量的颜色直方图与SPi,h的相邻区域中的所有像素点在HVS颜色空间的H分量的颜色直方图的距离,表示SPi,h中的所有像素点在HVS颜色空间的S分量的颜色直方图与SPi,h的相邻区域中的所有像素点在HVS颜色空间的S分量的颜色直方图的距离,表示SPi,h中的所有像素点的LBP特征统计直方图与SPi,h的相邻区域中的所有像素点的LBP特征统计直方图的距离,表示SPi,h中的所有像素点的视差统计直方图与SPi,h的相邻区域中的所有像素点的视差统计直方图的距离,此处的相邻区域是指Ri中与SPi,h相邻的区域;①-3. Calculate the contrast feature vector of each area in the right view point image of each stereoscopic image in {L i , R i , d i |1≤i≤N}, and divide the hth area in R i The contrast feature vector of SP i, h is denoted as x i , h c = [ d i , h , d i , h 1 , d i , h 2 , d i , h 3 , d i , h 4 , d i , h 5 , d i , h 6 ] , in, The dimension of is 36, where the symbol “[]” is a vector representation symbol, and d i, h represent the distance between the first feature vector u i, h of SP i, h and the first feature vector of the adjacent area, The dimension of f i, h is 20, f i, h represents the mean value of the frequency response feature vector of all pixels in SP i, h , the dimension of c i, h is 9, c i, h represents SP i, The mean value of the color feature vectors of all pixels in h , d i, h represents the mean value of the disparity magnitude of SP i, h , Represents the color histogram of all pixels in SP i, h in the R component, G component and B component of the RGB color space and the R component, G component of all pixels in the adjacent area of SP i, h in the RGB color space The distance of the color histogram of component and B component, Represents the color histogram of all pixels in SP i, h in the L component, a component and b component of CIELAB color space and the L component, a of all pixels in the adjacent area of SP i, h in CIELAB color space The distance of the color histogram of component and b component, Represents the distance between the color histogram of all pixels in SP i, h in the H component of the HVS color space and the color histogram of all pixels in the adjacent area of SP i, h in the H component of the HVS color space, Indicates the distance between the color histogram of all pixels in SP i, h in the S component of the HVS color space and the color histogram of all pixels in the adjacent area of SP i, h in the S component of the HVS color space, Indicates the distance between the LBP feature statistical histogram of all pixels in SP i, h and the LBP feature statistical histogram of all pixels in the adjacent area of SP i, h , Indicates the distance between the disparity statistical histogram of all pixels in SP i, h and the disparity statistical histogram of all pixels in the adjacent area of SP i, h , where the adjacent area refers to the distance between R i and SP i, h adjacent area;

①-4、计算{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像中的每个区域的通用特征矢量,将Ri中的第h个区域SPi,h的通用特征矢量记为 其中,的维数为33,此处符号“[]”为矢量表示符号,的维数为20,表示SPi,h中的所有像素点的频率响应特征矢量的方差,的维数为9,表示SPi,h中的所有像素点的颜色特征矢量的方差,表示SPi,h的视差幅值的方差,xi,h的维数为2,xi,h表示SPi,h的中心像素点的坐标位置,si,h表示SPi,h的面积;①-4. Calculate the general feature vector of each region in the right view point image of each stereoscopic image in {L i , R i , d i |1≤i≤N}, and divide the hth region in R i The general eigenvector of SP i,h is denoted as in, The dimension of is 33, where the symbol “[]” is a vector representation symbol, The dimension of is 20, Represents the variance of the frequency response feature vector of all pixels in SP i,h , The dimension of is 9, Represents the variance of the color feature vectors of all pixels in SP i,h , Represents the variance of the parallax magnitude of SP i, h , the dimension of xi, h is 2, xi, h represents the coordinate position of the center pixel of SP i, h , s i, h represents the area of SP i, h ;

①-5、计算{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像中的每个区域的背景先验特征矢量,将Ri中的第h个区域SPi,h的背景先验特征矢量记为 X i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , 其中,的维数为36,此处符号“[]”为矢量表示符号,ei,h表示SPi,h的第一特征矢量ui,h与背景区域的第一特征矢量的距离,fi,h的维数为20,fi,h表示SPi,h中的所有像素点的频率响应特征矢量的均值,ci,h的维数为9,ci,h表示SPi,h中的所有像素点的颜色特征矢量的均值,表示SPi,h的视差幅值的均值,表示SPi,h中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图与Ri中的背景区域中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图的距离,表示SPi,h中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图与Ri中的背景区域中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图的距离,表示SPi,h中的所有像素点在HVS颜色空间的H分量的颜色直方图与Ri中的背景区域中的所有像素点在HVS颜色空间的H分量的颜色直方图的距离,表示SPi,h中的所有像素点在HVS颜色空间的S分量的颜色直方图与Ri中的背景区域中的所有像素点在HVS颜色空间的S分量的颜色直方图的距离,表示SPi,h中的所有像素点的LBP特征统计直方图与Ri中的背景区域中的所有像素点的LBP特征统计直方图的距离,表示SPi,h中的所有像素点的视差统计直方图与Ri中的背景区域中的所有像素点的视差统计直方图的距离,此处的背景区域是指Ri中位于最左边、最右边、最上边、最下边的区域;①-5. Calculate the background prior feature vector of each region in the right view point image of each stereoscopic image in {L i , R i , d i |1≤i≤N}, and set the hth in R i The background prior feature vector of a region SP i, h is denoted as x i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , in, The dimension of is 36, where the symbol “[]” is a vector representation symbol, e i,h represents the distance between the first feature vector u i,h of SP i,h and the first feature vector of the background area, The dimension of f i, h is 20, f i, h represents the mean value of the frequency response feature vector of all pixels in SP i, h , the dimension of c i, h is 9, c i, h represents SP i, The mean value of the color feature vectors of all pixels in h , Indicates the mean value of the disparity magnitude of SP i,h , Represents the color histogram of the R component, G component and B component of all pixels in SP i, h in the RGB color space and the R component, G component and B component of all pixels in the background area in R i in the RGB color space The distance of the color histogram of the B component, Represents the color histogram of all pixels in SP i, h in the L component, a component and b component of CIELAB color space and the L component, a component and b component of all pixels in the background area in R i in CIELAB color space The distance of the color histogram of the b component, Indicates the distance between the color histogram of the H component of all pixels in SP i,h in the H component of the HVS color space and the color histogram of the H component of all pixels in the background area in R i in the HVS color space, Represents the distance between the color histogram of the S component of all pixels in SP i,h in the S component of the HVS color space and the color histogram of the S component of all pixels in the background area in R i in the HVS color space, Indicates the distance between the LBP feature statistical histogram of all pixels in SP i, h and the LBP feature statistical histogram of all pixels in the background area in R i , Indicates the distance between the disparity statistical histogram of all pixels in SP i,h and the disparity statistical histogram of all pixels in the background area in R i , where the background area refers to the leftmost, most Right, top, and bottom areas;

①-6、将{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像中的每个区域的对比度特征矢量、通用特征矢量和背景先验特征矢量按顺序进行排列,构成{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像中的每个区域的用于反映视觉显著性的特征矢量,将Ri中的第h个区域SPi,h的用于反映视觉显著性的特征矢量记为Xi,h其中,Xi,h的维数为105,此处符号“[]”为矢量表示符号;①-6. The contrast feature vector, general feature vector and background prior feature vector of each region in the right view image of each stereo image in {L i , R i , d i |1≤i≤N} Arranged in order to form the feature vector used to reflect the visual salience of each region in the right view image of each stereoscopic image in {L i , R i , d i |1≤i≤N}, R The feature vector used to reflect the visual salience of the hth region SP i ,h in i is denoted as Xi ,h , Among them, the dimensions of X i, h are 105, and the symbol “[]” here is a vector representation symbol;

①-7、采用随机森林回归,对{Li,Ri,di|1≤i≤N}中的所有立体图像的右视点图像中的所有区域的用于反映视觉显著性的特征矢量进行训练,并使得经过训练得到的回归函数值与平均眼动值之间的误差最小,得到最优的随机森林回归训练模型,记为f(Dinp),其中,f()为函数表示形式,Dinp表示随机森林回归训练模型的输入矢量;①-7. Use random forest regression to perform feature vectors for reflecting visual salience of all areas in the right viewpoint images of all stereoscopic images in {L i , R i , d i |1≤i≤N} training, and make the error between the regression function value obtained through training and the average eye movement value the smallest, and obtain the optimal random forest regression training model, which is denoted as f(D inp ), where f() is the function representation, D inp represents the input vector of the random forest regression training model;

所述的测试阶段的具体步骤如下:The specific steps of the testing phase are as follows:

②-1、对于任意一副测试立体图像Stest,将Stest的左视点图像、右视点图像、右视差图像对应记为Ltest、Rtest、dtest;然后采用超像素分割技术将Rtest分割成M个互不重叠的区域,将Rtest中的第h个区域记为SPh';其中,M≥1,1≤h≤M;②-1. For any pair of test stereo images S test , record the left viewpoint image, right viewpoint image, and right disparity image of S test as L test , R test , and d test respectively ; then use superpixel segmentation technology to divide R test Divide into M non-overlapping areas, and record the hth area in R test as SP h '; where, M≥1, 1≤h≤M;

②-2、按照步骤①-3至步骤①-6的过程,以相同的操作方式获取Rtest中的每个区域的用于反映视觉显著性的特征矢量,将Rtest中的第h个区域SPh'的用于反映视觉显著性的特征矢量记为Ftest,h;然后根据训练阶段得到的最优的随机森林回归训练模型f(Dinp),将Ftest,h作为最优的随机森林回归训练模型的输入矢量,获取Rtest中的每个区域的三维视觉显著值,将Rtest中的第h个区域SPh'的三维视觉显著值记为S3D,h,S3D,h=f(Ftest,h);再将Rtest中的每个区域的三维视觉显著值作为对应区域中的所有像素点的显著值,从而得到Rtest的三维显著图,记为{S3D(x,y)},其中,此处(x,y)表示Stest中的像素点的坐标位置,1≤x≤W,1≤y≤H,W和H对应表示Stest的宽度和高度,S3D(x,y)表示{S3D(x,y)}中坐标位置为(x,y)的像素点的像素值。②-2. According to the process from step ①-3 to step ①-6, obtain the feature vector used to reflect the visual salience of each region in the R test in the same operation mode, and convert the hth region in the R test to The feature vector used to reflect the visual salience of SP h ' is recorded as F test,h ; then according to the optimal random forest regression training model f(D inp ) obtained in the training stage, F test,h is used as the optimal random The input vector of the forest regression training model, obtain the 3D visual saliency value of each region in the R test , and record the 3D visual saliency value of the hth region SP h ' in the R test as S 3D,h , S 3D,h = f(F test, h ); then take the 3D visual saliency value of each region in the R test as the saliency value of all pixels in the corresponding region, thereby obtaining the 3D saliency map of the R test , denoted as {S 3D ( x, y)}, where (x, y) represents the coordinate position of the pixel in the S test , 1≤x≤W, 1≤y≤H, W and H correspond to the width and height of the S test , S 3D (x, y) represents the pixel value of the pixel at the coordinate position (x, y) in {S 3D (x, y)}.

所述的步骤①-3中Ri中的第h个区域SPi,h的对比度特征矢量的获取过程为:The contrast feature vector of the hth region SP i,h in R i in the step ①-3 The acquisition process is:

a1、计算Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量的均值,记为fi,h,fi,h中的第个元素的值等于Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量中的第个元素的频率响应振幅的均值,其中,fi,h的维数为20, a1. Calculate the mean value of the frequency response feature vectors of all the pixel points in the hth region SP i,h in R i , denoted as f i,h , the first in f i,h The value of the element is equal to the h-th area SP i,h in the frequency response feature vector of all pixels in R i The mean value of the frequency response amplitude of elements, where the dimensions of f i,h are 20,

a2、计算Ri中的第h个区域SPi,h中的所有像素点的颜色特征矢量的均值,记为ci,h c i , h = [ R ‾ i , h , G ‾ i , h , B ‾ i , h , L ‾ i , h , a ‾ i . h , b ‾ i , h , H ‾ i , h , V ‾ i , h , S ‾ i , h ] , 其中,ci,h的维数为9,此处符号“[]”为矢量表示符号,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的R分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的G分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的B分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的L分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的a分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的b分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的H分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的V分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的S分量的颜色值的均值;a2. Calculate the mean value of the color feature vectors of all pixels in the hth area SP i,h in R i , denoted as c i,h , c i , h = [ R ‾ i , h , G ‾ i , h , B ‾ i , h , L ‾ i , h , a ‾ i . h , b ‾ i , h , h ‾ i , h , V ‾ i , h , S ‾ i , h ] , Among them, the dimensions of c i and h are 9, and the symbol “[]” here is a vector representation symbol, Represents the mean value of the color values of the R component of the RGB color space for all pixels in the hth area SP i,h in R i , Represents the mean value of the color values of the G component of the RGB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the B component of the RGB color space for all pixels in the hth area SP i,h in R i , Represents the mean value of the color values of the L component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the a component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the b component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the H component of the HVS color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of all pixels in the hth region SP i,h in R i in the V component of the HVS color space, Represents the mean value of the color values of the S component of the HVS color space for all pixels in the hth region SP i,h in R i ;

a3、计算Ri中的第h个区域SPi,h的视差幅值的均值,记为等于di中与SPi,h对应的区域中的所有像素点的像素值的均值;a3. Calculate the mean value of the parallax magnitude of the hth region SP i, h in R i , denoted as Equal to the mean value of the pixel values of all pixels in the area corresponding to SP i, h in d i ;

a4、将fi,h、ci,h按顺序进行排列,构成Ri中的第h个区域SPi,h的第一特征矢量,记为ui,h其中,ui,h的维数为30,此处符号“[]”为矢量表示符号;a4. Put f i, h , c i, h and Arranged in order to form the first feature vector of the hth region SP i,h in R i , denoted as u i,h , Among them, the dimensions of u i and h are 30, and the symbol “[]” here is a vector representation symbol;

a5、计算Ri中的第h个区域SPi,h的第一特征矢量ui,h与相邻区域的第一特征矢量的距离,记为di,h其中,di,h的维数为30,1≤p≤M,表示Ri中的第h个区域SPi,h的相邻区域的序号的集合,ui,p表示Ri中的第p个区域SPi,p的第一特征矢量,符号“||”为取绝对值符号,P表示Ri中的第h个区域SPi,h的相邻区域的总个数,此处的相邻区域是指Ri中与SPi,h相邻的区域;a5. Calculate the distance between the first feature vector u i,h of the hth area SP i,h in R i and the first feature vector of the adjacent area, denoted as d i,h , Among them, the dimensions of d i, h are 30, 1≤p≤M, Represents the set of serial numbers of the adjacent regions of SP i, h in the hth region in R i , u i, p represents the first feature vector of the pth region SP i, p in R i , symbol "||" In order to take the absolute value sign, P represents the total number of adjacent areas of the hth area SP i, h in R i , where the adjacent area refers to the area adjacent to SP i, h in R i ;

a6、计算Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的H分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的S分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点的LBP特征统计直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点的视差统计直方图,记为其中,的维数为163的维数为163的维数为16,的维数为16,的维数为256,的维数为16;a6. Calculate the color histogram of the R component, G component and B component of all pixels in the hth region SP i,h in R i in the RGB color space, denoted as Calculate the color histogram of the L component, a component and b component of all pixels in the hth area SP i, h in R i in the CIELAB color space, denoted as Calculate the color histogram of the H component of all pixels in the hth region SP i,h in R i in the HVS color space, denoted as Calculate the color histogram of the S component of the HVS color space for all pixels in the hth region SP i,h in R i , denoted as Calculate the LBP feature statistical histogram of all pixels in the hth region SP i,h in R i , denoted as Calculate the disparity statistical histogram of all pixels in the hth area SP i,h in R i , denoted as in, The dimension of is 16 3 , The dimension of is 16 3 , The dimension of is 16, The dimension of is 16, The dimension of is 256, The dimension of is 16;

a7、计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图的距离,记为 a7. Calculate The distance from the color histogram of the R component, G component and B component of all pixels in the adjacent area of the hth area SP i, h in R i in the RGB color space is denoted as

计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the L component, a component and b component of the CIELAB color space to all pixels in the adjacent area of the hth area SP i, h in R i is denoted as

计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点在HVS颜色空间的H分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the H component of the HVS color space to all pixels in the adjacent area of the h-th area SP i, h in R i is denoted as

计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点在HVS颜色空间的S分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the S component of the HVS color space to all pixels in the adjacent area of the hth area SP i,h in R i is denoted as

计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点的LBP特征统计直方图的距离,记为 calculate The distance from the LBP feature statistical histogram of all pixels in the adjacent area of the hth area SP i,h in R i is denoted as

计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点的视差统计直方图的距离,记为 calculate The distance from the disparity statistical histogram of all pixels in the adjacent area of SP i, h in the hth area SP i, h in R i is denoted as

其中,1≤p≤M,表示Ri中的第h个区域SPi,h的相邻区域的序号的集合,P表示Ri中的第h个区域SPi,h的相邻区域的总个数,χ()为求卡方距离函数,表示Ri中的第p个区域SPi,p中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图,表示Ri中的第p个区域SPi,p中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图,表示Ri中的第p个区域SPi,p中的所有像素点在HVS颜色空间的H分量的颜色直方图,表示Ri中的第p个区域SPi,p中的所有像素点在HVS颜色空间的S分量的颜色直方图,表示Ri中的第p个区域SPi,p中的所有像素点的LBP特征统计直方图,表示Ri中的第p个区域SPi,p中的所有像素点的视差统计直方图;Among them, 1≤p≤M, Indicates the set of serial numbers of the adjacent areas of the hth area SP i, h in R i , P indicates the total number of adjacent areas of the hth area SP i, h in R i , χ() is chi-square distance function, Indicates the color histogram of the R component, G component and B component of all pixels in the pth area SP i,p in R i in the RGB color space, Represents the color histogram of the L component, a component and b component of all pixels in the pth area SP i,p in R i in the CIELAB color space, Represents the color histogram of the H component of the HVS color space for all pixels in the p-th region SP i,p in R i , Represents the color histogram of the S component of the HVS color space for all pixels in the p-th region SP i,p in R i , Represents the LBP feature statistical histogram of all pixels in the p-th region SP i,p in R i , Represents the disparity statistical histogram of all pixels in the p-th region SP i, p in R i ;

a8、将di,h按顺序进行排列,构成Ri中的第h个区域SPi,h的对比度特征矢量,记为 X i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , 其中,的维数为36,此处符号“[]”为矢量表示符号。a8, combine d i, h , and Arranged in order to form the contrast feature vector of the hth region SP i,h in R i , denoted as x i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , in, The dimension of is 36, where the symbol “[]” is a vector symbol.

所述的步骤①-4中Ri中的第h个区域SPi,h的通用特征矢量的获取过程为:The general feature vector of the h-th area SP i,h in R i in the step ①-4 The acquisition process is:

b1、计算Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量的方差,记为中的第个元素的值等于Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量中的第个元素的频率响应振幅的方差,其中,的维数为20, b1. Calculate the variance of the frequency response feature vectors of all pixels in the hth region SP i,h in R i , denoted as in the first The value of the element is equal to the h-th area SP i,h in the frequency response feature vector of all pixels in R i The variance of the frequency response amplitude of elements, where, The dimension of is 20,

b2、计算Ri中的第h个区域SPi,h中的所有像素点的颜色特征矢量的方差,记为 c ^ i , h = [ R ^ i , h , G ^ i , h , B ^ i , h , L ^ i , h , a ^ i , h , b ^ i , h , H ^ i , h , V ^ i , h , S ^ i , h ] , 其中,的维数为9,此处符号“[]”为矢量表示符号,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的R分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的G分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的B分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的L分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的a分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的b分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的H分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的V分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的S分量的颜色值的方差;b2. Calculate the variance of the color feature vectors of all pixels in the hth region SP i, h in R i , denoted as c ^ i , h = [ R ^ i , h , G ^ i , h , B ^ i , h , L ^ i , h , a ^ i , h , b ^ i , h , h ^ i , h , V ^ i , h , S ^ i , h ] , in, The dimension of is 9, where the symbol “[]” is a vector representation symbol, Represents the variance of the color value of the R component of the RGB color space for all pixels in the hth area SP i,h in R i , Indicates the variance of the color value of the G component of the RGB color space for all pixels in the hth area SP i,h in R i , Indicates the variance of the color value of the B component of the RGB color space for all pixels in the hth area SP i,h in R i , Represents the variance of the color value of the L component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Represents the variance of the color value of the a component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Represents the variance of the color value of the b component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Represents the variance of the color value of the H component of the HVS color space for all pixels in the hth region SP i,h in R i , Represents the variance of the color value of the V component of the HVS color space for all pixels in the hth region SP i,h in R i , Represents the variance of the color value of the S component of the HVS color space for all pixels in the hth region SP i,h in R i ;

b3、计算Ri中的第h个区域SPi,h的视差幅值的方差,记为等于di中与SPi,h对应的区域中的所有像素点的像素值的方差;b3. Calculate the variance of the parallax magnitude of the hth region SP i in R i , h , denoted as Equal to the variance of the pixel values of all pixels in the area corresponding to SP i, h in d i ;

b4、获取Ri中的第h个区域SPi,h的中心像素点的坐标位置,记为xi,h,其中,xi,h的维数为2;b4. Obtain the coordinate position of the central pixel point of the hth region SP i,h in R i , denoted as x i,h , where the dimension of x i,h is 2;

b5、计算Ri中的第h个区域SPi,h的面积,记为si,hb5. Calculate the area of the hth region SP i,h in R i , and denote it as s i,h ;

b6、将xi,h和si,h按顺序进行排列,构成Ri中的第h个区域SPi,h的通用特征矢量,记为 X i , h a = [ f ^ i , h , c ^ i , h , d ^ i , h , x i , h , s i , h ] , 其中,的维数为33,此处符号“[]”为矢量表示符号。b6, will x i, h and s i, h are arranged in order to form the general feature vector of the hth region SP i, h in R i , denoted as x i , h a = [ f ^ i , h , c ^ i , h , d ^ i , h , x i , h , the s i , h ] , in, The dimension of is 33, where the symbol “[]” is a vector symbol.

所述的步骤①-5中Ri中的第h个区域SPi,h的背景先验特征矢量的获取过程为:The background priori feature vector of the hth region SP i,h in R i in the step ①-5 The acquisition process is:

c1、计算Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量的均值,记为fi,h,fi,h中的第个元素的值等于Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量中的第个元素的频率响应振幅的均值,其中,fi,h的维数为20, c1. Calculate the mean value of the frequency response feature vectors of all pixels in the hth area SP i,h in R i , denoted as f i,h , the first in f i,h The value of the element is equal to the h-th area SP i,h in the frequency response feature vector of all pixels in R i The mean value of the frequency response amplitude of elements, where the dimensions of f i,h are 20,

c2、计算Ri中的第h个区域SPi,h中的所有像素点的颜色特征矢量的均值,记为ci,h c i , h = [ R ‾ i , h , G ‾ i , h , B ‾ i , h , L ‾ i , h , a ‾ i , h , b ‾ i , h , H ‾ i , h , V ‾ i , h , S ‾ i , h ] , 其中,ci,h的维数为9,此处符号“[]”为矢量表示符号,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的R分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的G分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的B分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的L分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的a分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的b分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的H分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的V分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的S分量的颜色值的均值;c2. Calculate the mean value of the color feature vectors of all pixels in the hth area SP i,h in R i , denoted as c i,h , c i , h = [ R ‾ i , h , G ‾ i , h , B ‾ i , h , L ‾ i , h , a ‾ i , h , b ‾ i , h , h ‾ i , h , V ‾ i , h , S ‾ i , h ] , Among them, the dimensions of c i and h are 9, and the symbol “[]” here is a vector representation symbol, Represents the mean value of the color values of the R component of the RGB color space for all pixels in the hth region SP i,h in R i , Represents the mean value of the color values of the G component of the RGB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the B component of the RGB color space for all pixels in the hth area SP i,h in R i , Represents the mean value of the color values of the L component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the a component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the b component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the H component of the HVS color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of all pixels in the hth region SP i,h in R i in the V component of the HVS color space, Represents the mean value of the color values of the S component of the HVS color space for all pixels in the hth region SP i,h in R i ;

c3、计算Ri中的第h个区域SPi,h的视差幅值的均值,记为等于di中与SPi,h对应的区域中的所有像素点的像素值的均值;c3. Calculate the mean value of the parallax magnitude of the hth region SP i, h in R i , denoted as Equal to the mean value of the pixel values of all pixels in the area corresponding to SP i, h in d i ;

c4、将fi,h、ci,h按顺序进行排列,构成Ri中的第h个区域SPi,h的第一特征矢量,记为ui,h其中,ui,h的维数为30,此处符号“[]”为矢量表示符号;c4, put f i, h , c i, h and Arranged in order to form the first feature vector of the hth region SP i,h in R i , denoted as u i,h , Among them, the dimensions of u i and h are 30, and the symbol “[]” here is a vector representation symbol;

c5、计算Ri中的第h个区域SPi,h的第一特征矢量ui,h与背景区域的第一特征矢量的距离,记为ei,h其中,ei,h的维数为30,1≤q≤M,表示Ri中的所有背景区域的序号的集合,ui,q表示Ri中的第q个区域SPi,q的第一特征矢量,符号“||”为取绝对值符号,Q表示Ri中的背景区域的总个数,此处的背景区域是指Ri中位于最左边、最右边、最上边、最下边的区域;c5. Calculate the distance between the first feature vector u i,h of the hth area SP i,h in R i and the first feature vector of the background area, denoted as e i,h , Among them, the dimensions of e i, h are 30, 1≤q≤M, Represents the set of serial numbers of all background regions in R i , u i, q represents the first feature vector of the qth region SP i, q in R i , the symbol "||" is an absolute value symbol, Q represents R The total number of background regions in i , where the background region refers to the regions located in the leftmost, rightmost, uppermost, and lowermost in R i ;

c6、计算Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的H分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的S分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点的LBP特征统计直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点的视差统计直方图,记为其中,的维数为163的维数为163的维数为16,的维数为16,的维数为256,的维数为16;c6. Calculate the color histogram of the R component, G component and B component of all pixels in the hth region SP i,h in R i in the RGB color space, denoted as Calculate the color histogram of the L component, a component and b component of all pixels in the hth area SP i, h in R i in the CIELAB color space, denoted as Calculate the color histogram of the H component of all pixels in the hth region SP i,h in R i in the HVS color space, denoted as Calculate the color histogram of the S component of the HVS color space for all pixels in the hth region SP i,h in R i , denoted as Calculate the LBP feature statistical histogram of all pixels in the hth area SP i,h in R i , denoted as Calculate the disparity statistical histogram of all pixels in the hth area SP i,h in R i , denoted as in, The dimension of is 16 3 , The dimension of is 16 3 , The dimension of is 16, The dimension of is 16, The dimension of is 256, The dimension of is 16;

c7、计算与Ri中的背景区域中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图的距离,记为 c7, calculate The distance from the color histogram of the R component, G component and B component of the RGB color space to all pixels in the background area in R i is denoted as

计算与Ri中的背景区域中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the L component, a component and b component of the CIELAB color space to all pixels in the background area in R i is denoted as

计算与Ri中的背景区域中的所有像素点在HVS颜色空间的H分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the H component of the HVS color space to all pixels in the background area in R i is denoted as

计算与Ri中的背景区域中的所有像素点在HVS颜色空间的S分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the S component of the HVS color space to all pixels in the background area in R i is denoted as

计算与Ri中的背景区域中的所有像素点的LBP特征统计直方图的距离,记为 calculate The distance from the LBP feature statistical histogram of all pixels in the background area in R i is denoted as

计算与Ri中的背景区域中的所有像素点的视差统计直方图的距离,记为 calculate The distance from the disparity statistical histogram of all pixels in the background area in R i is denoted as

其中,1≤q≤M,表示Ri中的所有背景区域的序号的集合,Q表示Ri中的背景区域的总个数,χ()为求卡方距离函数,表示Ri中的第q个区域SPi,q中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图,表示Ri中的第q个区域SPi,q中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图,表示Ri中的第q个区域SPi,q中的所有像素点在HVS颜色空间的H分量的颜色直方图,表示Ri中的第q个区域SPi,q中的所有像素点在HVS颜色空间的S分量的颜色直方图,表示Ri中的第q个区域SPi,q中的所有像素点的LBP特征统计直方图,表示Ri中的第q个区域SPi,q中的所有像素点的视差统计直方图;Among them, 1≤q≤M, Represent the collection of the sequence numbers of all background regions in R i , Q represents the total number of background regions in R i , χ () is the chi-square distance function, Represents the color histogram of the R component, G component and B component of all pixels in the qth area SP i,q in R i in the RGB color space, Represents the color histogram of the L component, a component and b component of all pixels in the qth region SP i, q in R i in the CIELAB color space, Represents the color histogram of the H component of all pixels in the qth region SP i,q in R i in the HVS color space, Represents the color histogram of the S component of the HVS color space for all pixels in the qth region SP i,q in R i , Represents the LBP feature statistical histogram of all pixels in the qth region SP i,q in R i , Represents the disparity statistical histogram of all pixels in the qth region SP i, q in R i ;

c8、将ei,h 按顺序进行排列,构成Ri中的第h个区域SPi,h的背景先验特征矢量,记为 X i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , 其中,的维数为36,此处符号“[]”为矢量表示符号。c8, put e i,h , Arranged in order to form the background prior feature vector of the hth region SP i,h in R i , denoted as x i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , in, The dimension of is 36, where the symbol “[]” is a vector symbol.

所述的Ri中的每个像素点的频率响应特征矢量的获取过程为:The acquisition process of the frequency response feature vector of each pixel in the R i is:

1)-1、采用Gabor滤波器对Ri进行滤波处理,得到Ri中的每个像素点在不同中心频率和不同方向因子下的频率响应振幅,将Ri中坐标位置为(x,y)的像素点在中心频率为ω和方向因子为θ下的频率响应振幅记为G(x,y;ω,θ),其中,此处(x,y)表示{Li,Ri,di|1≤i≤N}中的立体图像中的像素点的坐标位置,1≤x≤W,1≤y≤H,W和H对应表示{Li,Ri,di|1≤i≤N}中的立体图像的宽度和高度,ω表示Gabor滤波器的中心频率,ω∈Φω,θ表示Gabor滤波器的方向因子,θ∈Φθ,Φω表示Gabor滤波器的所有中心频率的集合,Φθ表示Gabor滤波器的所有方向因子的集合;1)-1. Use the Gabor filter to filter R i to obtain the frequency response amplitude of each pixel in R i at different center frequencies and different direction factors, and set the coordinate position in R i as (x, y ) pixel at center frequency ω and direction factor θ is denoted as G(x,y; ω,θ), where (x,y) means {L i ,R i ,d The coordinate position of the pixel in the stereoscopic image in i |1≤i≤N}, 1≤x≤W, 1≤y≤H, W and H correspond to {L i , R i , d i |1≤i ≤ N} in the width and height of the stereo image, ω represents the center frequency of the Gabor filter, ω∈Φ ω , θ represents the direction factor of the Gabor filter, θ∈Φ θ , Φ ω represents all the center frequencies of the Gabor filter The set of , Φ θ represents the set of all direction factors of the Gabor filter;

1)-2、将Ri中的每个像素点在不同中心频率和不同方向因子下的频率响应振幅按顺序进行排列,构成Ri中的每个像素点的频率响应特征矢量,将Ri中坐标位置为(x,y)的像素点的频率响应特征矢量记为fi(x,y),其中,fi(x,y)的维数为20。1)-2. Arrange the frequency response amplitudes of each pixel point in R i in different center frequencies and different direction factors in order to form the frequency response feature vector of each pixel point in R i , and set R i The frequency response feature vector of the pixel at the middle coordinate position (x, y) is recorded as f i (x, y), where the dimension of f i (x, y) is 20.

所述的Ri中的每个像素点的颜色特征矢量的获取过程为:The acquisition process of the color feature vector of each pixel in the R i is:

2)-1、计算Ri中的每个像素点在不同颜色空间的颜色值,将Ri中坐标位置为(x,y)的像素点在RGB颜色空间的R分量、G分量和B分量的颜色值分别记为R(x,y)、G(x,y)和B(x,y),将Ri中坐标位置为(x,y)的像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色值分别记为L(x,y)、a(x,y)和b(x,y),将Ri中坐标位置为(x,y)的像素点在HVS颜色空间的H分量、V分量和S分量的颜色值分别记为H(x,y)、V(x,y)和S(x,y),其中,此处(x,y)表示{Li,Ri,di|1≤i≤N}中的立体图像中的像素点的坐标位置,1≤x≤W,1≤y≤H,W和H对应表示{Li,Ri,di|1≤i≤N}中的立体图像的宽度和高度;2)-1. Calculate the color value of each pixel point in R i in different color spaces, and use the R component, G component and B component of the pixel point whose coordinate position is (x, y) in R i in the RGB color space The color values of are recorded as R( x , y), G(x, y) and B(x, y) respectively, and the L component, The color values of a component and b component are recorded as L(x, y), a(x, y) and b(x, y) respectively, and the pixel point whose coordinate position in R i is (x, y) is in HVS color The color values of the H component, V component and S component of the space are respectively recorded as H(x, y), V(x, y) and S(x, y), where (x, y) here represents {L i , R i , d i |1≤i≤N} in the coordinate position of the pixel in the stereo image, 1≤x≤W, 1≤y≤H, W and H correspond to {L i ,R i ,d Width and height of stereo images in i |1≤i≤N};

2)-2、将Ri中的每个像素点在不同颜色空间的颜色值按顺序进行排列,构成Ri中的每个像素点的颜色特征矢量,将Ri中坐标位置为(x,y)的像素点的颜色特征矢量记为ci(x,y),ci(x,y)=[R(x,y),G(x,y),B(x,y),L(x,y),a(x,y),b(x,y),H(x,y),V(x,y),S(x,y)],其中,ci(x,y)的维数为9,此处符号“[]”为矢量表示符号。2)-2. Arrange the color values of each pixel in R i in different color spaces in order to form the color feature vector of each pixel in R i , and set the coordinate position in R i as (x, y) is recorded as c i (x, y), c i (x, y) = [R(x, y), G(x, y), B(x, y), L (x,y),a(x,y),b(x,y),H(x,y),V(x,y),S(x,y)], where c i (x,y ) has a dimension of 9, where the symbol “[]” is a vector representation symbol.

与现有技术相比,本发明的优点在于:Compared with the prior art, the present invention has the advantages of:

1)本发明方法同时考虑了立体图像的右视点图像中的每个区域的对比度特征矢量、通用特征矢量和背景先验特征矢量,并融合得到立体图像的右视点图像中的每个区域的用于反映视觉显著性的特征矢量,因此本发明方法具有较高的提取准确性和较强的稳定性,并能够较好地反映各种因素的显著变化情况,符合显著语义特征。1) The method of the present invention simultaneously considers the contrast feature vector, general feature vector and background prior feature vector of each region in the right viewpoint image of the stereoscopic image, and fuses to obtain the use of each region in the right viewpoint image of the stereoscopic image. The method of the present invention has high extraction accuracy and strong stability, and can better reflect the significant changes of various factors, and conforms to the significant semantic features.

2)本发明方法通过训练构建用于反映视觉显著性的特征矢量与平均眼动值之间的随机森林回归训练模型,然后利用该随机森林回归训练模型来预测测试立体图像的右视点图像中的每个区域的三维视觉显著值,从而得到测试立体图像的三维显著图,有效地提高了视觉显著值的预测准确性。2) The method of the present invention constructs the random forest regression training model between the feature vector and the average eye movement value for reflecting visual salience by training, then utilizes this random forest regression training model to predict the The 3D visual saliency value of each region is obtained to obtain the 3D saliency map of the test stereo image, which effectively improves the prediction accuracy of the visual saliency value.

附图说明Description of drawings

图1为本发明方法的总体实现框图;Fig. 1 is the overall realization block diagram of the inventive method;

图2a为“Image1”的右视点图像;Figure 2a is the right view image of "Image1";

图2b为“Image1”的右视点图像的真实眼动图;Figure 2b is the real eye movement diagram of the right viewpoint image of "Image1";

图2c为“Image1”的三维显著图;Figure 2c is the 3D saliency map of "Image1";

图3a为“Image2”的右视点图像;Figure 3a is the right view image of "Image2";

图3b为“Image2”的右视点图像的真实眼动图;Figure 3b is the real eye movement diagram of the right viewpoint image of "Image2";

图3c为“Image2”的三维显著图;Figure 3c is the 3D saliency map of "Image2";

图4a为“Image3”的右视点图像;Figure 4a is the right view image of "Image3";

图4b为“Image3”的右视点图像的真实眼动图;Figure 4b is the real eye movement diagram of the right viewpoint image of "Image3";

图4c为“Image3”的三维显著图;Figure 4c is the 3D saliency map of "Image3";

图5a为“Image4”的右视点图像;Figure 5a is the right view image of "Image4";

图5b为“Image4”的右视点图像的真实眼动图;Figure 5b is the real eye movement diagram of the right viewpoint image of "Image4";

图5c为“Image4”的三维显著图;Figure 5c is the 3D saliency map of "Image4";

图6a为“Image5”的右视点图像;Figure 6a is the right view image of "Image5";

图6b为“Image5”的右视点图像的真实眼动图;Figure 6b is the real eye movement diagram of the right viewpoint image of "Image5";

图6c为“Image5”的三维显著图;Figure 6c is the 3D saliency map of "Image5";

图7a为“Image6”的右视点图像;Figure 7a is the right view image of "Image6";

图7b为“Image6”的右视点图像的真实眼动图;Figure 7b is the real eye movement diagram of the right viewpoint image of "Image6";

图7c为“Image6”的三维显著图;Figure 7c is the 3D saliency map of "Image6";

图8a为“Image7”的右视点图像;Figure 8a is the right view image of "Image7";

图8b为“Image7”的右视点图像的真实眼动图;Figure 8b is the real eye movement diagram of the right viewpoint image of "Image7";

图8c为“Image7”的三维显著图。Figure 8c is the 3D saliency map of "Image7".

具体实施方式detailed description

以下结合附图实施例对本发明作进一步详细描述。The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

本发明提出的一种立体图像显著图提取方法,其总体实现框图如图1所示,其包括训练阶段和测试阶段两个过程,训练阶段的具体步骤如下:A kind of three-dimensional image saliency map extracting method that the present invention proposes, its overall realization block diagram as shown in Figure 1, it comprises two processes of training phase and testing phase, and the concrete steps of training phase are as follows:

①-1、将选取的N副各不相同的立体图像以及每幅立体图像的右视差图像构成一个集合,记为{Li,Ri,di|1≤i≤N},其中,N≥1,在本实施例中取N=600,Li表示{Li,Ri,di|1≤i≤N}中的第i幅立体图像的左视点图像,Ri表示{Li,Ri,di|1≤i≤N}中的第i幅立体图像的右视点图像,di表示{Li,Ri,di|1≤i≤N}中的第i幅立体图像的右视差图像。①-1. The selected N sets of different stereoscopic images and the right disparity images of each stereoscopic image form a set, which is recorded as {L i , R i , d i |1≤i≤N}, where N ≥1, N=600 in this embodiment, L i represents the left viewpoint image of the i-th stereoscopic image in {L i , R i , d i |1≤i≤N}, and R i represents {L i ,R i ,d i |1≤i≤N} in the right view image of the i-th stereo image, d i represents the i-th stereo in {L i ,R i ,d i |1≤i≤N} Right disparity image of the image.

在本实施例中,采用新加坡国立大学提供的三维人眼跟踪数据库(NUS 3D-Saliency database)构造训练立体图像集,该立体图像数据库包含600副立体图像以及对应的右视差图像,并给出了每副立体图像的真实眼动图。In this embodiment, the three-dimensional human eye tracking database (NUS 3D-Saliency database) provided by the National University of Singapore (NUS 3D-Saliency database) is used to construct a training stereoscopic image set. The stereoscopic image database contains 600 pairs of stereoscopic images and corresponding right parallax images. Realistic eye-movement maps for each stereoscopic image.

①-2、采用现有的超像素分割技术将{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像分割成M个互不重叠的区域,将Ri中的第h个区域记为SPi,h,可将{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像重新表示为M个区域的集合,将Ri重新表示的M个区域的集合记为{SPi,h};其中,M≥1,在本实施例中取M=400,1≤h≤M。①-2. Use the existing superpixel segmentation technology to segment the right viewpoint image of each stereoscopic image in {L i , R i , d i |1≤i≤N} into M non-overlapping regions, and The h-th region in R i is denoted as SP i,h , and the right viewpoint image of each stereo image in {L i ,R i ,d i |1≤i≤N} can be re-expressed as M regions A set, the set of M regions re-expressed by R i is denoted as {SP i,h }; where M≥1, M=400 in this embodiment, 1≤h≤M.

①-3、计算{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像中的每个区域的对比度特征矢量,将Ri中的第h个区域SPi,h的对比度特征矢量记为 X i , h c = [ d i , h , d i , h 1 , d i , h 2 , d i , h 3 , d i , h 4 , d i , h 5 , d i , h 6 ] , 其中,的维数为36,此处符号“[]”为矢量表示符号,di,h表示SPi,h的第一特征矢量ui,h与相邻区域的第一特征矢量的距离,fi,h的维数为20,fi,h表示SPi,h中的所有像素点的频率响应特征矢量的均值,ci,h的维数为9,ci,h表示SPi,h中的所有像素点的颜色特征矢量的均值,表示SPi,h的视差幅值的均值,表示SPi,h中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图与SPi,h的相邻区域中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图的距离,表示SPi,h中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图与SPi,h的相邻区域中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图的距离,表示SPi,h中的所有像素点在HVS颜色空间的H分量的颜色直方图与SPi,h的相邻区域中的所有像素点在HVS颜色空间的H分量的颜色直方图的距离,表示SPi,h中的所有像素点在HVS颜色空间的S分量的颜色直方图与SPi,h的相邻区域中的所有像素点在HVS颜色空间的S分量的颜色直方图的距离,表示SPi,h中的所有像素点的LBP特征统计直方图与SPi,h的相邻区域中的所有像素点的LBP特征统计直方图的距离,表示SPi,h中的所有像素点的视差统计直方图与SPi,h的相邻区域中的所有像素点的视差统计直方图的距离,此处的相邻区域是指Ri中与SPi,h相邻的区域。①-3. Calculate the contrast feature vector of each area in the right view point image of each stereoscopic image in {L i , R i , d i |1≤i≤N}, and divide the hth area in R i The contrast feature vector of SP i, h is denoted as x i , h c = [ d i , h , d i , h 1 , d i , h 2 , d i , h 3 , d i , h 4 , d i , h 5 , d i , h 6 ] , in, The dimension of is 36, where the symbol “[]” is a vector representation symbol, and d i, h represent the distance between the first feature vector u i, h of SP i, h and the first feature vector of the adjacent area, The dimension of f i, h is 20, f i, h represents the mean value of the frequency response feature vector of all pixels in SP i, h , the dimension of c i, h is 9, c i, h represents SP i, The mean value of the color feature vectors of all pixels in h , Indicates the mean value of the disparity magnitude of SP i,h , Represents the color histogram of all pixels in SP i, h in the R component, G component and B component of the RGB color space and the R component, G component of all pixels in the adjacent area of SP i, h in the RGB color space The distance of the color histogram of component and B component, Represents the color histogram of all pixels in SP i, h in the L component, a component and b component of CIELAB color space and the L component, a of all pixels in the adjacent area of SP i, h in CIELAB color space The distance of the color histogram of component and b component, Indicates the distance between the color histogram of all pixels in SP i, h in the H component of the HVS color space and the color histogram of all pixels in the adjacent area of SP i, h in the H component of the HVS color space, Indicates the distance between the color histogram of all pixels in SP i, h in the S component of the HVS color space and the color histogram of all pixels in the adjacent area of SP i, h in the S component of the HVS color space, Indicates the distance between the LBP feature statistical histogram of all pixels in SP i, h and the LBP feature statistical histogram of all pixels in the adjacent area of SP i, h , Indicates the distance between the disparity statistical histogram of all pixels in SP i, h and the disparity statistical histogram of all pixels in the adjacent area of SP i, h , where the adjacent area refers to the distance between R i and SP The area adjacent to i, h .

在此具体实施例中,步骤①-3中Ri中的第h个区域SPi,h的对比度特征矢量的获取过程为:In this specific embodiment, the contrast feature vector of the h-th region SP i,h in R i in step ①-3 The acquisition process is:

a1、计算Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量的均值,记为fi,h,fi,h中的第个元素的值等于Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量中的第个元素的频率响应振幅的均值,其中,fi,h的维数为20, a1. Calculate the mean value of the frequency response feature vectors of all pixels in the hth area SP i,h in R i , denoted as f i,h , the first in f i,h The value of the element is equal to the h-th area SP i,h in the frequency response feature vector of all pixels in R i The mean value of the frequency response amplitude of elements, where the dimension of f i,h is 20,

a2、计算Ri中的第h个区域SPi,h中的所有像素点的颜色特征矢量的均值,记为ci,h c i , h = [ R ‾ i , h , G ‾ i , h , B ‾ i , h , L ‾ i , h , a ‾ i , h , b ‾ i , h , H ‾ i , h , V ‾ i , h , S ‾ i , h ] , 其中,ci,h的维数为9,此处符号“[]”为矢量表示符号,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的R分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的G分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的B分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的L分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的a分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的b分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的H分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的V分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的S分量的颜色值的均值。a2. Calculate the mean value of the color feature vectors of all pixels in the hth area SP i,h in R i , denoted as c i,h , c i , h = [ R ‾ i , h , G ‾ i , h , B ‾ i , h , L ‾ i , h , a ‾ i , h , b ‾ i , h , h ‾ i , h , V ‾ i , h , S ‾ i , h ] , Among them, the dimensions of c i and h are 9, and the symbol “[]” here is a vector representation symbol, Represents the mean value of the color values of the R component of the RGB color space for all pixels in the hth area SP i,h in R i , Represents the mean value of the color values of the G component of the RGB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the B component of the RGB color space for all pixels in the hth area SP i,h in R i , Represents the mean value of the color values of the L component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the a component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Represents the mean value of the color values of the b component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Represents the mean value of the color values of the H component of the HVS color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of all pixels in the hth region SP i,h in R i in the V component of the HVS color space, Indicates the mean value of the color values of the S component of the HVS color space for all pixels in the hth region SP i,h in R i .

a3、计算Ri中的第h个区域SPi,h的视差幅值的均值,记为等于di中与SPi,h对应的区域中的所有像素点的像素值的均值。a3. Calculate the mean value of the parallax magnitude of the hth region SP i, h in R i , denoted as It is equal to the mean value of the pixel values of all the pixel points in the area corresponding to SP i, h in d i .

a4、将fi,h、ci,h按顺序进行排列,构成Ri中的第h个区域SPi,h的第一特征矢量,记为ui,h其中,ui,h的维数为30,此处符号“[]”为矢量表示符号。a4. Put f i, h , c i, h and Arranged in order to form the first feature vector of the hth area SP i,h in R i , denoted as u i,h , Wherein, the dimensions of u i, h are 30, and the symbol “[]” here is a vector representation symbol.

a5、计算Ri中的第h个区域SPi,h的第一特征矢量ui,h与相邻区域的第一特征矢量的距离,记为di,h其中,di,h的维数为30,1≤p≤M,表示Ri中的第h个区域SPi,h的相邻区域的序号的集合,ui,p表示Ri中的第p个区域SPi,p(SPi,p为SPi,h的相邻区域)的第一特征矢量,符号“||”为取绝对值符号,P表示Ri中的第h个区域SPi,h的相邻区域的总个数,在本实施例中取P=20,此处的相邻区域是指Ri中与SPi,h相邻的区域。a5. Calculate the distance between the first feature vector u i,h of the hth area SP i,h in R i and the first feature vector of the adjacent area, denoted as d i,h , Among them, the dimensions of d i, h are 30, 1≤p≤M, Indicates the set of serial numbers of the adjacent areas of the hth area SP i,h in R i , u i,p indicates the pth area SP i,p in R i (SP i,p is the value of SP i,h Adjacent region), the symbol "||" is an absolute value symbol, and P represents the total number of adjacent regions of the hth region SP i, h in R i , which is taken in this embodiment P=20, the adjacent region here refers to the region adjacent to SP i,h in R i .

a6、计算Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的H分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的S分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点的LBP特征统计直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点的视差统计直方图,记为其中,的维数为163的维数为163的维数为16,的维数为16,的维数为256,的维数为16。a6. Calculate the color histogram of the R component, G component and B component of all pixels in the hth region SP i,h in R i in the RGB color space, denoted as Calculate the color histogram of the L component, a component and b component of all pixels in the hth area SP i, h in R i in the CIELAB color space, denoted as Calculate the color histogram of the H component of all pixels in the hth region SP i,h in R i in the HVS color space, denoted as Calculate the color histogram of the S component of the HVS color space for all pixels in the hth region SP i,h in R i , denoted as Calculate the LBP feature statistical histogram of all pixels in the hth region SP i,h in R i , denoted as Calculate the disparity statistical histogram of all pixels in the hth area SP i,h in R i , denoted as in, The dimension of is 16 3 , The dimension of is 16 3 , The dimension of is 16, The dimension of is 16, The dimension of is 256, The dimension of is 16.

a7、计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图的距离,记为 a7. Calculate The distance from the color histogram of the R component, G component and B component of all pixels in the adjacent area of the hth area SP i, h in R i in the RGB color space is denoted as

计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the L component, a component and b component of the CIELAB color space to all pixels in the adjacent area of the hth area SP i, h in R i is denoted as

计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点在HVS颜色空间的H分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the H component of the HVS color space to all pixels in the adjacent area of the hth area SP i,h in R i is denoted as

计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点在HVS颜色空间的S分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the S component of the HVS color space to all pixels in the adjacent area of the hth area SP i,h in R i is denoted as

计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点的LBP特征统计直方图的距离,记为 calculate The distance from the LBP feature statistical histogram of all pixels in the adjacent area of the hth area SP i,h in R i is denoted as

计算与Ri中的第h个区域SPi,h的相邻区域中的所有像素点的视差统计直方图的距离,记为 calculate The distance from the disparity statistical histogram of all pixels in the adjacent area of SP i, h in the hth area SP i, h in R i is denoted as

其中,1≤p≤M,表示Ri中的第h个区域SPi,h的相邻区域的序号的集合,P表示Ri中的第h个区域SPi,h的相邻区域的总个数,在本实施例中取P=20,χ()为求卡方距离(Chi-distance measure)函数,表示Ri中的第p个区域SPi,p中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图,表示Ri中的第p个区域SPi,p中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图,表示Ri中的第p个区域SPi,p中的所有像素点在HVS颜色空间的H分量的颜色直方图,表示Ri中的第p个区域SPi,p中的所有像素点在HVS颜色空间的S分量的颜色直方图,表示Ri中的第p个区域SPi,p中的所有像素点的LBP特征统计直方图,表示Ri中的第p个区域SPi,p中的所有像素点的视差统计直方图。Among them, 1≤p≤M, Represents the set of the sequence numbers of the adjacent regions of the hth region SP i, h in R i , and P represents the total number of adjacent regions of the hth region SP i, h in R i , in this embodiment Take P=20, χ() is the function of seeking chi-square distance (Chi-distance measure), Represents the color histogram of the R component, G component and B component of all pixels in the pth area SP i,p in R i in the RGB color space, Represents the color histogram of the L component, a component and b component of all pixels in the pth area SP i,p in R i in the CIELAB color space, Represents the color histogram of the H component of the HVS color space for all pixels in the p-th region SP i,p in R i , Represents the color histogram of the S component of the HVS color space for all pixels in the p-th region SP i,p in R i , Represents the LBP feature statistical histogram of all pixels in the p-th region SP i,p in R i , Represents the disparity statistical histogram of all pixels in the p-th region SP i,p in R i .

a8、将di,h按顺序进行排列,构成Ri中的第h个区域SPi,h的对比度特征矢量,记为 X i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , 其中,的维数为36,此处符号“[]”为矢量表示符号。a8, combine d i, h , and Arranged in order to form the contrast feature vector of the hth region SP i,h in R i , denoted as x i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , in, The dimension of is 36, where the symbol “[]” is a vector symbol.

①-4、计算{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像中的每个区域的通用特征矢量,将Ri中的第h个区域SPi,h的通用特征矢量记为 其中,的维数为33,此处符号“[]”为矢量表示符号,的维数为20,表示SPi,h中的所有像素点的频率响应特征矢量的方差,的维数为9,表示SPi,h中的所有像素点的颜色特征矢量的方差,表示SPi,h的视差幅值的方差,xi,h的维数为2,xi,h表示SPi,h的中心像素点的坐标位置,si,h表示SPi,h的面积。①-4. Calculate the general feature vector of each region in the right view point image of each stereoscopic image in {L i , R i , d i |1≤i≤N}, and divide the hth region in R i The general eigenvector of SP i,h is denoted as in, The dimension of is 33, where the symbol “[]” is a vector representation symbol, The dimension of is 20, Represents the variance of the frequency response feature vector of all pixels in SP i,h , The dimension of is 9, Represents the variance of the color feature vectors of all pixels in SP i,h , Represents the variance of the parallax magnitude of SP i, h , the dimension of xi, h is 2, xi, h represents the coordinate position of the center pixel of SP i, h , s i, h represents the area of SP i, h .

在此具体实施例中,步骤①-4中Ri中的第h个区域SPi,h的通用特征矢量的获取过程为:In this specific embodiment, the universal feature vector of the hth region SP i,h in R i in step ①-4 The acquisition process is:

b1、计算Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量的方差,记为中的第个元素的值等于Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量中的第个元素的频率响应振幅的方差,其中,的维数为20, b1. Calculate the variance of the frequency response feature vectors of all pixels in the hth region SP i, h in R i , denoted as in the first The value of the element is equal to the h-th area SP i,h in the frequency response feature vector of all pixels in R i The variance of the frequency response amplitude of elements, where, The dimension of is 20,

b2、计算Ri中的第h个区域SPi,h中的所有像素点的颜色特征矢量的方差,记为 c ^ i , h = [ R ^ i , h , G ^ i , h , B ^ i , h , L ^ i , h , a ^ i , h , b ^ i , h , H ^ i , h , V ^ i , h , S ^ i , h ] , 其中,的维数为9,此处符号“[]”为矢量表示符号,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的R分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的G分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的B分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的L分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的a分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的b分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的H分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的V分量的颜色值的方差,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的S分量的颜色值的方差。b2. Calculate the variance of the color feature vectors of all pixels in the hth region SP i, h in R i , denoted as c ^ i , h = [ R ^ i , h , G ^ i , h , B ^ i , h , L ^ i , h , a ^ i , h , b ^ i , h , h ^ i , h , V ^ i , h , S ^ i , h ] , in, The dimension of is 9, where the symbol “[]” is a vector representation symbol, Represents the variance of the color value of the R component of the RGB color space for all pixels in the hth area SP i,h in R i , Indicates the variance of the color value of the G component of the RGB color space for all pixels in the hth area SP i,h in R i , Indicates the variance of the color value of the B component of the RGB color space for all pixels in the hth area SP i,h in R i , Represents the variance of the color value of the L component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Represents the variance of the color value of the a component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Represents the variance of the color value of the b component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Represents the variance of the color value of the H component of the HVS color space for all pixels in the hth region SP i,h in R i , Represents the variance of the color value of the V component of the HVS color space for all pixels in the hth region SP i,h in R i , Indicates the variance of the color value of the S component of the HVS color space for all pixels in the h-th region SP i,h in R i .

b3、计算Ri中的第h个区域SPi,h的视差幅值的方差,记为等于di中与SPi,h对应的区域中的所有像素点的像素值的方差。b3. Calculate the variance of the parallax magnitude of the hth region SP i, h in R i , denoted as It is equal to the variance of the pixel values of all pixels in the area corresponding to SP i, h in d i .

b4、获取Ri中的第h个区域SPi,h的中心像素点的坐标位置,记为xi,h,其中,xi,h的维数为2。b4. Obtain the coordinate position of the center pixel of the hth area SP i,h in R i , denoted as x i,h , where the dimension of x i,h is 2.

b5、计算Ri中的第h个区域SPi,h的面积,记为si,hb5. Calculate the area of the hth region SP i,h in R i , which is denoted as s i,h .

b6、将xi,h和si,h按顺序进行排列,构成Ri中的第h个区域SPi,h的通用特征矢量,记为 X i , h a = [ f ^ i , h , c ^ i , h , d ^ i , h , x i , h , s i , h ] , 其中,的维数为33,此处符号“[]”为矢量表示符号。b6, will x i, h and s i, h are arranged in order to form the general feature vector of the hth region SP i, h in R i , denoted as x i , h a = [ f ^ i , h , c ^ i , h , d ^ i , h , x i , h , the s i , h ] , in, The dimension of is 33, where the symbol “[]” is a vector symbol.

①-5、计算{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像中的每个区域的背景先验特征矢量,将Ri中的第h个区域SPi,h的背景先验特征矢量记为 X i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , 其中,的维数为36,此处符号“[]”为矢量表示符号,ei,h表示SPi,h的第一特征矢量ui,h与背景区域的第一特征矢量的距离,fi,h的维数为20,fi,h表示SPi,h中的所有像素点的频率响应特征矢量的均值,ci,h的维数为9,ci,h表示SPi,h中的所有像素点的颜色特征矢量的均值,表示SPi,h的视差幅值的均值,表示SPi,h中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图与Ri中的背景区域中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图的距离,表示SPi,h中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图与Ri中的背景区域中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图的距离,表示SPi,h中的所有像素点在HVS颜色空间的H分量的颜色直方图与Ri中的背景区域中的所有像素点在HVS颜色空间的H分量的颜色直方图的距离,表示SPi,h中的所有像素点在HVS颜色空间的S分量的颜色直方图与Ri中的背景区域中的所有像素点在HVS颜色空间的S分量的颜色直方图的距离,表示SPi,h中的所有像素点的LBP特征统计直方图与Ri中的背景区域中的所有像素点的LBP特征统计直方图的距离,表示SPi,h中的所有像素点的视差统计直方图与Ri中的背景区域中的所有像素点的视差统计直方图的距离,此处的背景区域是指Ri中位于最左边、最右边、最上边、最下边的区域。①-5. Calculate the background prior feature vector of each region in the right view point image of each stereoscopic image in {L i , R i , d i |1≤i≤N}, and set the hth in R i The background prior feature vector of a region SP i, h is denoted as x i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , in, The dimension of is 36, where the symbol “[]” is a vector representation symbol, e i,h represents the distance between the first feature vector u i,h of SP i,h and the first feature vector of the background area, The dimension of f i, h is 20, f i, h represents the mean value of the frequency response feature vector of all pixels in SP i, h , the dimension of c i, h is 9, c i, h represents SP i, The mean value of the color feature vectors of all pixels in h , Indicates the mean value of the disparity magnitude of SP i,h , Represents the color histogram of the R component, G component and B component of all pixels in SP i, h in the RGB color space and the R component, G component and B component of all pixels in the background area in R i in the RGB color space The distance of the color histogram of the B component, Represents the color histogram of all pixels in SP i, h in the L component, a component and b component of CIELAB color space and the L component, a component and b component of all pixels in the background area in R i in CIELAB color space The distance of the color histogram of the b component, Indicates the distance between the color histogram of the H component of all pixels in SP i,h in the H component of the HVS color space and the color histogram of the H component of all pixels in the background area in R i in the HVS color space, Represents the distance between the color histogram of the S component of all pixels in SP i,h in the S component of the HVS color space and the color histogram of the S component of all pixels in the background area in R i in the HVS color space, Indicates the distance between the LBP feature statistical histogram of all pixels in SP i, h and the LBP feature statistical histogram of all pixels in the background area in R i , Indicates the distance between the disparity statistical histogram of all pixels in SP i,h and the disparity statistical histogram of all pixels in the background area in R i , where the background area refers to the leftmost, most Right, top, and bottom areas.

在此具体实施例中,步骤①-5中Ri中的第h个区域SPi,h的背景先验特征矢量的获取过程为:In this specific embodiment, the background prior feature vector of the hth region SP i,h in R i in step ①-5 The acquisition process is:

c1、计算Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量的均值,记为fi,h,fi,h中的第个元素的值等于Ri中的第h个区域SPi,h中的所有像素点的频率响应特征矢量中的第个元素的频率响应振幅的均值,其中,fi,h的维数为20, c1. Calculate the mean value of the frequency response feature vectors of all the pixel points in the hth area SP i,h in R i , denoted as f i,h , the first in f i,h The value of the element is equal to the h-th area SP i,h in the frequency response feature vector of all pixels in R i The mean value of the frequency response amplitude of elements, where the dimension of f i,h is 20,

c2、计算Ri中的第h个区域SPi,h中的所有像素点的颜色特征矢量的均值,记为ci,h c i , h = [ R ‾ i , h , G ‾ i , h , B ‾ i , h , L ‾ i , h , a ‾ i , h , b ‾ i , h , H ‾ i , h , V ‾ i , h , S ‾ i , h ] , 其中,ci,h的维数为9,此处符号“[]”为矢量表示符号,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的R分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的G分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的B分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的L分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的a分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的b分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的H分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的V分量的颜色值的均值,表示Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的S分量的颜色值的均值。c2. Calculate the mean value of the color feature vectors of all pixels in the hth area SP i,h in R i , denoted as c i,h , c i , h = [ R ‾ i , h , G ‾ i , h , B ‾ i , h , L ‾ i , h , a ‾ i , h , b ‾ i , h , h ‾ i , h , V ‾ i , h , S ‾ i , h ] , Among them, the dimensions of c i and h are 9, and the symbol “[]” here is a vector representation symbol, Represents the mean value of the color values of the R component of the RGB color space for all pixels in the hth area SP i,h in R i , Represents the mean value of the color values of the G component of the RGB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the B component of the RGB color space for all pixels in the hth area SP i,h in R i , Represents the mean value of the color values of the L component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of the a component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Represents the mean value of the color values of the b component of the CIELAB color space for all pixels in the hth region SP i,h in R i , Represents the mean value of the color values of the H component of the HVS color space for all pixels in the hth region SP i,h in R i , Indicates the mean value of the color values of all pixels in the hth region SP i,h in R i in the V component of the HVS color space, Indicates the mean value of the color values of the S component of the HVS color space for all pixels in the hth region SP i,h in R i .

c3、计算Ri中的第h个区域SPi,h的视差幅值的均值,记为等于di中与SPi,h对应的区域中的所有像素点的像素值的均值。c3. Calculate the mean value of the parallax magnitude of the hth region SP i, h in R i , denoted as It is equal to the mean value of the pixel values of all the pixel points in the area corresponding to SP i, h in d i .

c4、将fi,h、ci,h按顺序进行排列,构成Ri中的第h个区域SPi,h的第一特征矢量,记为ui,h其中,ui,h的维数为30,此处符号“[]”为矢量表示符号;c4, put f i, h , c i, h and Arranged in order to form the first feature vector of the hth region SP i,h in R i , denoted as u i,h , Among them, the dimensions of u i and h are 30, and the symbol “[]” here is a vector representation symbol;

c5、计算Ri中的第h个区域SPi,h的第一特征矢量ui,h与背景区域的第一特征矢量的距离,记为ei,h其中,ei,h的维数为30,1≤q≤M,表示Ri中的所有背景区域的序号的集合,ui,q表示Ri中的第q个区域SPi,q(SPi,q为Ri中的背景区域)的第一特征矢量,符号“||”为取绝对值符号,Q表示Ri中的背景区域的总个数,此处的背景区域是指Ri中位于最左边、最右边、最上边、最下边的区域,即将落在Ri中的最左边、最右边、最上边和最下边的区域作为背景区域。c5. Calculate the distance between the first feature vector u i,h of the hth area SP i,h in R i and the first feature vector of the background area, denoted as e i,h , Among them, the dimension of e i, h is 30, 1≤q≤M, Represents the set of serial numbers of all background regions in R i , u i,q represents the first feature vector of the qth region SP i, q in R i (SP i,q is the background region in R i ), symbol "||" is the absolute value symbol, Q represents the total number of background areas in R i , where the background area refers to the leftmost, rightmost, uppermost, and lowermost areas in The leftmost, rightmost, uppermost and lowermost regions in R i are used as background regions.

c6、计算Ri中的第h个区域SPi,h中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的H分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点在HVS颜色空间的S分量的颜色直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点的LBP特征统计直方图,记为计算Ri中的第h个区域SPi,h中的所有像素点的视差统计直方图,记为其中,的维数为163的维数为163的维数为16,的维数为16,的维数为256,的维数为16。c6. Calculate the color histogram of the R component, G component and B component of all pixels in the hth region SP i,h in R i in the RGB color space, denoted as Calculate the color histogram of the L component, a component and b component of all pixels in the hth area SP i, h in R i in the CIELAB color space, denoted as Calculate the color histogram of the H component of all pixels in the hth region SP i,h in R i in the HVS color space, denoted as Calculate the color histogram of the S component of the HVS color space for all pixels in the hth region SP i,h in R i , denoted as Calculate the LBP feature statistical histogram of all pixels in the hth region SP i,h in R i , denoted as Calculate the disparity statistical histogram of all pixels in the hth area SP i,h in R i , denoted as in, The dimension of is 16 3 , The dimension of is 16 3 , The dimension of is 16, The dimension of is 16, The dimension of is 256, The dimension of is 16.

c7、计算与Ri中的背景区域中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图的距离,记为 c7, calculate The distance from the color histogram of the R component, G component and B component of the RGB color space to all pixels in the background area in R i is denoted as

计算与Ri中的背景区域中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the L component, a component and b component of the CIELAB color space to all pixels in the background area in R i is denoted as

计算与Ri中的背景区域中的所有像素点在HVS颜色空间的H分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the H component of the HVS color space to all pixels in the background area in R i is denoted as

计算与Ri中的背景区域中的所有像素点在HVS颜色空间的S分量的颜色直方图的距离,记为 calculate The distance from the color histogram of the S component of the HVS color space to all pixels in the background area in R i is denoted as

计算与Ri中的背景区域中的所有像素点的LBP特征统计直方图的距离,记为 calculate The distance from the LBP feature statistical histogram of all pixels in the background area in R i is denoted as

计算与Ri中的背景区域中的所有像素点的视差统计直方图的距离,记为 calculate The distance from the disparity statistical histogram of all pixels in the background area in R i is denoted as

其中,1≤q≤M,表示Ri中的所有背景区域的序号的集合,Q表示Ri中的背景区域的总个数,χ()为求卡方距离(Chi-distance measure)函数,表示Ri中的第q个区域SPi,q中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图,表示Ri中的第q个区域SPi,q中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图,表示Ri中的第q个区域SPi,q中的所有像素点在HVS颜色空间的H分量的颜色直方图,表示Ri中的第q个区域SPi,q中的所有像素点在HVS颜色空间的S分量的颜色直方图,表示Ri中的第q个区域SPi,q中的所有像素点的LBP特征统计直方图,表示Ri中的第q个区域SPi,q中的所有像素点的视差统计直方图。Among them, 1≤q≤M, Indicates the collection of the serial numbers of all background regions in R i , Q represents the total number of background regions in R i , χ() is the function of seeking chi-square distance (Chi-distance measure), Represents the color histogram of the R component, G component and B component of all pixels in the qth region SP i,q in R i in the RGB color space, Represents the color histogram of the L component, a component and b component of all pixels in the qth region SP i, q in R i in the CIELAB color space, Represents the color histogram of the H component of all pixels in the qth region SP i,q in R i in the HVS color space, Represents the color histogram of the S component of the HVS color space for all pixels in the qth region SP i,q in R i , Represents the LBP feature statistical histogram of all pixels in the qth region SP i,q in R i , Represents the disparity statistical histogram of all pixels in the qth region SP i,q in R i .

c8、将ei,h按顺序进行排列,构成Ri中的第h个区域SPi,h的背景先验特征矢量,记为 X i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , 其中,的维数为36,此处符号“[]”为矢量表示符号。c8, put e i,h , and Arranged in order to form the background prior feature vector of the hth region SP i,h in R i , denoted as x i , h b = [ e i , h , e i , h 1 , e i , h 2 , e i , h 3 , e i , h 4 , e i , h 5 , e i , h 6 ] , in, The dimension of is 36, where the symbol “[]” is a vector symbol.

①-6、将{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像中的每个区域的对比度特征矢量、通用特征矢量和背景先验特征矢量按顺序进行排列,构成{Li,Ri,di|1≤i≤N}中的每幅立体图像的右视点图像中的每个区域的用于反映视觉显著性的特征矢量,将Ri中的第h个区域SPi,h的用于反映视觉显著性的特征矢量记为Xi,h其中,Xi,h的维数为105,此处符号“[]”为矢量表示符号。①-6. The contrast feature vector, general feature vector and background prior feature vector of each region in the right view image of each stereo image in {L i , R i , d i |1≤i≤N} Arranged in order to form the feature vector used to reflect the visual salience of each region in the right view image of each stereoscopic image in {L i , R i , d i |1≤i≤N}, R The feature vector used to reflect the visual salience of the hth region SP i ,h in i is denoted as Xi ,h , Wherein, the dimensions of X i, h are 105, and the symbol “[]” here is a vector representation symbol.

①-7、采用现有的随机森林回归,对{Li,Ri,di|1≤i≤N}中的所有立体图像的右视点图像中的所有区域的用于反映视觉显著性的特征矢量进行训练,并使得经过训练得到的回归函数值与平均眼动值之间的误差最小,得到最优的随机森林回归训练模型,记为f(Dinp),其中,f()为函数表示形式,Dinp表示随机森林回归训练模型的输入矢量。①-7. Use the existing random forest regression to reflect the visual salience of all regions in the right view point images of all stereoscopic images in {L i , R i , d i |1≤i≤N} The feature vector is trained, and the error between the regression function value obtained through training and the average eye movement value is minimized, and the optimal random forest regression training model is obtained, which is denoted as f(D inp ), where f() is the function Representation, D inp represents the input vector of the random forest regression training model.

测试阶段的具体步骤如下:The specific steps in the testing phase are as follows:

②-1、对于任意一副测试立体图像Stest,将Stest的左视点图像、右视点图像、右视差图像对应记为Ltest、Rtest、dtest;然后采用现有的超像素分割技术将Rtest分割成M个互不重叠的区域,将Rtest中的第h个区域记为SPh',可将Rtest重新表示为M个区域的集合,记为{SPh'};其中,M≥1,在本实施例中取M=400,1≤h≤M。②-1. For any pair of test stereo images S test , record the left viewpoint image, right viewpoint image, and right disparity image of S test as L test , R test , and d test correspondingly; then use the existing superpixel segmentation technology Divide the R test into M non-overlapping areas, record the hth area in the R test as SP h ', and re-express the R test as a collection of M areas, denoted as {SP h '}; where , M≥1, M=400 in this embodiment, 1≤h≤M.

②-2、按照步骤①-3至步骤①-6的过程,以相同的操作方式获取Rtest中的每个区域的用于反映视觉显著性的特征矢量,将Rtest中的第h个区域SPh'的用于反映视觉显著性的特征矢量记为Ftest,h,Ftest,h的获取过程为:计算Rtest中的第h个区域SPh'的对比度特征矢量记为 X h c ′ = [ d h ′ , d h 1 ′ , d h 2 ′ , d h 3 ′ , d h 4 ′ , d h 5 ′ , d h 6 ′ ] , 其中,的维数为36,dh'表示SPh'的第一特征矢量uh'与相邻区域的第一特征矢量的距离,fh'的维数为20,fh'表示SPh'的频率响应特征矢量的均值,ch'的维数为9,ch'表示SPh'的颜色特征矢量的均值,表示SPh'的视差幅值的均值,表示SPh'中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图与SPh'的相邻区域中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图的距离,表示SPh'中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图与SPh'的相邻区域中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图的距离,表示SPh'中的所有像素点在HVS颜色空间的H分量的颜色直方图与SPh'的相邻区域中的所有像素点在HVS颜色空间的H分量的颜色直方图的距离,表示SPh'中的所有像素点在HVS颜色空间的S分量的颜色直方图与SPh'的相邻区域中的所有像素点在HVS颜色空间的S分量的颜色直方图的距离,表示SPh'中的所有像素点的LBP特征统计直方图与SPh'的相邻区域中的所有像素点的LBP特征统计直方图的距离,表示SPh'中的所有像素点的视差统计直方图与SPh'的相邻区域中的所有像素点的视差统计直方图的距离,此处的相邻区域是指与SPh'相邻的区域;计算Rtest中的第h个区域SPh'的通用特征矢量记为 X h a ′ = [ f ^ h ′ , c ^ h ′ , d ^ h ′ , x h ′ , s h ′ ] , 其中,的维数为33,的维数为20,表示SPh'的频率响应特征矢量的方差,的维数为9,表示SPh'的颜色特征矢量的方差,表示SPh'的视差幅值的方差,xh'的维数为2,xh'表示SPh'的中心像素点的坐标位置,sh'表示SPh'的面积;计算Rtest中的第h个区域SPh'的的背景先验特征矢量记为 X h b ′ = [ e h ′ , e h 1 ′ , e h 2 ′ , e h 3 ′ , e h 4 ′ , e h 5 ′ , e h 6 ′ ] , 其中,的维数为36,eh'表示SPh'的第一特征矢量uh'与背景区域的第一特征矢量的距离,表示SPh'中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图与背景区域中的所有像素点在RGB颜色空间的R分量、G分量和B分量的颜色直方图的距离,表示SPh'中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图与背景区域中的所有像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色直方图的距离,表示SPh'中的所有像素点在HVS颜色空间的H分量的颜色直方图与背景区域中的所有像素点在HVS颜色空间的H分量的颜色直方图的距离,表示SPh'中的所有像素点在HVS颜色空间的S分量的颜色直方图与背景区域中的所有像素点在HVS颜色空间的S分量的颜色直方图的距离,表示SPh'中的所有像素点的LBP特征统计直方图与背景区域中的所有像素点的LBP特征统计直方图的距离,表示SPh'中的所有像素点的视差统计直方图与背景区域中的所有像素点的视差统计直方图的距离,此处的背景区域是指位于最左边、最右边、最上边、最下边的区域;将Rtest中的第h个区域SPh'的对比度特征矢量、通用特征矢量和背景先验特征矢量按顺序进行排列,构成Rtest中的第h个区域SPh'的用于反映视觉显著性的特征矢量,记为Ftest,h其中,Ftest,h的维数为105;然后根据训练阶段得到的最优的随机森林回归训练模型f(Dinp),将Ftest,h作为最优的随机森林回归训练模型的输入矢量,获取Rtest中的每个区域的三维视觉显著值,将Rtest中的第h个区域SPh'的三维视觉显著值记为S3D,h,S3D,h=f(Ftest,h);再将Rtest中的每个区域的三维视觉显著值作为对应区域中的所有像素点的显著值,如对于Rtest中的第h个区域SPh',将S3D,h作为SPh'中的所有像素点的显著值,从而得到Rtest的三维显著图,记为{S3D(x,y)},其中,此处(x,y)表示Stest中的像素点的坐标位置,1≤x≤W,1≤y≤H,W和H对应表示Stest的宽度和高度,Stest的宽度与训练阶段选取的立体图像的宽度一致,Stest的高度与训练阶段选取的立体图像的高度一致,S3D(x,y)表示{S3D(x,y)}中坐标位置为(x,y)的像素点的像素值。②-2. According to the process from step ①-3 to step ①-6, obtain the feature vector used to reflect the visual salience of each region in the R test in the same operation mode, and convert the hth region in the R test to The feature vector of SP h ' used to reflect visual salience is recorded as F test,h , and the acquisition process of F test,h is: the contrast feature vector of the hth region SP h ' in R test is calculated as x h c ′ = [ d h ′ , d h 1 ′ , d h 2 ′ , d h 3 ′ , d h 4 ′ , d h 5 ′ , d h 6 ′ ] , in, The dimension of is 36, d h ' represents the distance between the first feature vector u h ' of SP h ' and the first feature vector of the adjacent area, The dimension of f h ' is 20, f h ' represents the mean value of the frequency response feature vector of SP h ', the dimension of c h ' is 9, and c h ' represents the mean value of the color feature vector of SP h ', denote the mean of the disparity magnitudes of SP h ', Represents the color histogram of all pixels in SP h ' in the R component, G component and B component of the RGB color space and the R component, G component and B component of all pixels in the adjacent area of SP h ' in the RGB color space The distance of the color histogram of the B component, Indicates the color histogram of all pixels in SP h ' in the L component, a component and b component of CIELAB color space and the L component, a component and b component of all pixels in the adjacent area of SP h ' in CIELAB color space The distance of the color histogram of the b component, Represents the distance between the color histogram of all pixels in SP h ' in the H component of the HVS color space and the color histogram of all pixels in the adjacent area of SP h ' in the H component of the HVS color space, Represents the distance between the color histogram of the S component of all pixels in the HVS color space in SP h ' and the color histogram of the S component of all pixels in the adjacent area of SP h ' in the HVS color space, Represents the distance between the LBP feature statistical histogram of all pixels in SP h ' and the LBP feature statistical histogram of all pixels in the adjacent area of SP h ', Indicates the distance between the disparity statistical histogram of all pixels in SP h ' and the disparity statistical histogram of all pixels in the adjacent area of SP h ', where the adjacent area refers to the area adjacent to SP h 'region; calculate the general feature vector of the hth region SP h ' in the R test and denote it as x h a ′ = [ f ^ h ′ , c ^ h ′ , d ^ h ′ , x h ′ , the s h ′ ] , in, The dimension of is 33, The dimension of is 20, represents the variance of the frequency response eigenvector of SP h ', The dimension of is 9, represents the variance of the color feature vector of SP h ', Indicates the variance of the parallax magnitude of SP h ', the dimension of x h 'is 2, x h 'indicates the coordinate position of the central pixel of SP h ', s h 'indicates the area of SP h '; calculate the R test in The background prior feature vector of the hth region SP h ' is denoted as x h b ′ = [ e h ′ , e h 1 ′ , e h 2 ′ , e h 3 ′ , e h 4 ′ , e h 5 ′ , e h 6 ′ ] , in, The dimension of is 36, e h ' represents the distance between the first feature vector u h ' of SP h ' and the first feature vector of the background area, Represents the color histogram of all pixels in SP h ' in the R component, G component and B component of the RGB color space and the color histogram of all pixels in the background area in the R component, G component and B component of the RGB color space the distance of the graph, Represents the color histogram of all pixels in SP h ' in the L component, a component and b component of CIELAB color space and the color histogram of all pixels in the background area in CIELAB color space L component, a component and b component the distance of the graph, Represents the distance between the color histogram of the H component of all pixels in SP h ' in the HVS color space and the color histogram of the H component of all pixels in the background area in the HVS color space, Represents the distance between the color histogram of all pixels in SP h ' in the S component of the HVS color space and the color histogram of all pixels in the background area in the S component of the HVS color space, Indicates the distance between the LBP feature statistical histogram of all pixels in SP h ' and the LBP feature statistical histogram of all pixels in the background area, Indicates the distance between the disparity statistical histogram of all pixels in SP h ' and the disparity statistical histogram of all pixels in the background area, where the background area refers to the leftmost, rightmost, uppermost, and lowermost region; the contrast feature vector, general feature vector and background prior feature vector of the hth region SP h ' in the R test are arranged in order to form the hth region SP h ' in the R test to reflect the visual Significant feature vector, denoted as F test,h , Among them, the dimension of F test, h is 105; then according to the optimal random forest regression training model f(D inp ) obtained in the training stage, F test, h is used as the input vector of the optimal random forest regression training model, Obtain the 3D visual saliency value of each region in the R test , and record the 3D visual saliency value of the hth region SP h ' in the R test as S 3D,h , S 3D,h = f(F test,h ) ; Then use the 3D visual saliency value of each region in the R test as the saliency value of all pixels in the corresponding region, such as for the hth region SP h ' in the R test , use S 3D,h as SP h ' The saliency values of all the pixels in the R test , so as to obtain the three-dimensional saliency map of the R test, which is recorded as {S 3D (x,y)}, where (x, y) represents the coordinate position of the pixel in the S test , 1≤x≤W, 1≤y≤H, W and H correspond to the width and height of S test , the width of S test is consistent with the width of the stereo image selected in the training stage, and the height of S test is the same as the stereo image selected in the training stage The heights are consistent, and S 3D (x, y) represents the pixel value of the pixel at the coordinate position (x, y) in {S 3D (x, y)}.

在此具体实施例中,Ri中的每个像素点的频率响应特征矢量的获取过程为:In this specific embodiment, the acquisition process of the frequency response feature vector of each pixel in R i is:

1)-1、采用Gabor滤波器对Ri进行滤波处理,得到Ri中的每个像素点在不同中心频率和不同方向因子下的频率响应振幅,将Ri中坐标位置为(x,y)的像素点在中心频率为ω和方向因子为θ下的频率响应振幅记为G(x,y;ω,θ),其中,此处(x,y)表示{Li,Ri,di|1≤i≤N}中的立体图像中的像素点的坐标位置,1≤x≤W,1≤y≤H,W和H对应表示{Li,Ri,di|1≤i≤N}中的立体图像的宽度和高度,ω表示Gabor滤波器的中心频率,ω∈Φω,θ表示Gabor滤波器的方向因子,θ∈Φθ,Φω表示Gabor滤波器的所有中心频率的集合,在本实施例中Φω={1.74,2.47,3.49,4.93,6.98},Φθ表示Gabor滤波器的所有方向因子的集合,在本实施例中Φθ={0°,90°,180°,270°}。1)-1. Use the Gabor filter to filter R i to obtain the frequency response amplitude of each pixel in R i at different center frequencies and different direction factors, and set the coordinate position in R i as (x, y ) pixel at the center frequency ω and the direction factor θ is denoted as G(x,y; ω,θ), where (x,y) means {L i ,R i ,d The coordinate position of the pixel in the stereoscopic image in i |1≤i≤N}, 1≤x≤W, 1≤y≤H, W and H correspond to {L i , R i , d i |1≤i ≤ N} in the width and height of the stereo image, ω represents the center frequency of the Gabor filter, ω∈Φ ω , θ represents the direction factor of the Gabor filter, θ∈Φ θ , Φ ω represents all the center frequencies of the Gabor filter The set of, in this embodiment Φ ω ={1.74,2.47,3.49,4.93,6.98}, Φ θ represents the set of all direction factors of the Gabor filter, in this embodiment Φ θ ={0°,90° ,180°,270°}.

1)-2、将Ri中的每个像素点在不同中心频率和不同方向因子下的频率响应振幅按顺序进行排列,构成Ri中的每个像素点的频率响应特征矢量,将Ri中坐标位置为(x,y)的像素点的频率响应特征矢量记为fi(x,y),fi(x,y)为由G(x,y;1.74,0°)、G(x,y;2.47,0°)、G(x,y;3.49,0°)、G(x,y;4.93,0°)、G(x,y;6.98,0°)、G(x,y;1.74,90°)、G(x,y;2.47,90°)、G(x,y;3.49,90°)、G(x,y;4.93,90°)、G(x,y;6.98,90°)、G(x,y;1.74,180°)、G(x,y;2.47,180°)、G(x,y;3.49,180°)、G(x,y;4.93,180°)、G(x,y;6.98,180°)、G(x,y;1.74,270°)、G(x,y;2.47,270°)、G(x,y;3.49,270°)、G(x,y;4.93,270°)、G(x,y;6.98,270°)按顺序排列构成的矢量,其中,fi(x,y)的维数为20。1)-2. Arrange the frequency response amplitudes of each pixel point in R i in different center frequencies and different direction factors in order to form the frequency response feature vector of each pixel point in R i , and set R i The frequency response feature vector of the pixel point whose coordinate position is (x, y) is recorded as f i (x, y), and f i (x, y) is defined by G(x, y; 1.74,0°), G( x,y; 2.47,0°), G(x,y; 3.49,0°), G(x,y; 4.93,0°), G(x,y; 6.98,0°), G(x,y; y; 1.74,90°), G(x,y; 2.47,90°), G(x,y; 3.49,90°), G(x,y; 4.93,90°), G(x,y; 6.98,90°), G(x,y; 1.74,180°), G(x,y; 2.47,180°), G(x,y; 3.49,180°), G(x,y; 4.93, 180°), G(x,y; 6.98,180°), G(x,y; 1.74,270°), G(x,y; 2.47,270°), G(x,y; 3.49,270°) ), G(x,y; 4.93,270°), G(x,y; 6.98,270°) are arranged in order to form a vector, where the dimension of f i (x,y) is 20.

在此具体实施例中,Ri中的每个像素点的颜色特征矢量的获取过程为:In this specific embodiment, the acquisition process of the color feature vector of each pixel in R i is:

2)-1、计算Ri中的每个像素点在不同颜色空间的颜色值,将Ri中坐标位置为(x,y)的像素点在RGB颜色空间的R分量、G分量和B分量的颜色值分别记为R(x,y)、G(x,y)和B(x,y),将Ri中坐标位置为(x,y)的像素点在CIELAB颜色空间的L分量、a分量和b分量的颜色值分别记为L(x,y)、a(x,y)和b(x,y),将Ri中坐标位置为(x,y)的像素点在HVS颜色空间的H分量、V分量和S分量的颜色值分别记为H(x,y)、V(x,y)和S(x,y),其中,此处(x,y)表示{Li,Ri,di|1≤i≤N}中的立体图像中的像素点的坐标位置,1≤x≤W,1≤y≤H,W和H对应表示{Li,Ri,di|1≤i≤N}中的立体图像的宽度和高度。2)-1. Calculate the color value of each pixel point in R i in different color spaces, and use the R component, G component and B component of the pixel point whose coordinate position is (x, y) in R i in the RGB color space The color values of are recorded as R( x , y), G(x, y) and B(x, y) respectively, and the L component, The color values of a component and b component are recorded as L(x, y), a(x, y) and b(x, y) respectively, and the pixel point whose coordinate position in R i is (x, y) is in HVS color The color values of the H component, V component and S component of the space are respectively recorded as H(x, y), V(x, y) and S(x, y), where (x, y) here represents {L i , R i , d i |1≤i≤N} in the coordinate position of the pixel in the stereo image, 1≤x≤W, 1≤y≤H, W and H correspond to {L i ,R i ,d Width and height of stereo images in i |1≤i≤N}.

2)-2、将Ri中的每个像素点在不同颜色空间的颜色值按顺序进行排列,构成Ri中的每个像素点的颜色特征矢量,将Ri中坐标位置为(x,y)的像素点的颜色特征矢量记为ci(x,y),ci(x,y)=[R(x,y),G(x,y),B(x,y),L(x,y),a(x,y),b(x,y),H(x,y),V(x,y),S(x,y)],其中,ci(x,y)的维数为9,此处符号“[]”为矢量表示符号。2)-2. Arrange the color values of each pixel in R i in different color spaces in order to form the color feature vector of each pixel in R i , and set the coordinate position in R i as (x, y) is recorded as c i (x, y), c i (x, y) = [R(x, y), G(x, y), B(x, y), L (x,y),a(x,y),b(x,y),H(x,y),V(x,y),S(x,y)], where c i (x,y ) has a dimension of 9, where the symbol “[]” is a vector representation symbol.

以下就利用本发明方法对法国南特大学提供的三维人眼跟踪数据库(3Deye-tracking database)中的Image1、Image2、Image3、Image4和Image5五幅立体图像的三维显著图进行提取。图2a给出了“Image1”的右视点图像、图2b给出了“Image1”的右视点图像的真实眼动图、图2c给出了“Image1”的三维显著图;图3a给出了“Image2”的右视点图像、图3b给出了“Image2”的右视点图像的真实眼动图、图3c给出了“Image2”的三维显著图;图4a给出了“Image3”的右视点图像、图4b给出了“Image3”的右视点图像的真实眼动图、图4c给出了“Image3”的三维显著图;图5a给出了“Image4”的右视点图像、图5b给出了“Image4”的右视点图像的真实眼动图、图5c给出了“Image4”的三维显著图;图6a给出了“Image5”的右视点图像、图6b给出了“Image5”的右视点图像的真实眼动图、图6c给出了“Image5”的三维显著图;图7a给出了“Image6”的右视点图像、图7b给出了“Image6”的右视点图像的真实眼动图、图7c给出了“Image6”的三维显著图;图8a给出了“Image7”的右视点图像、图8b给出了“Image7”的右视点图像的真实眼动图、图8c给出了“Image7”的三维显著图。从图2a至图8c中可以看出,采用本发明方法得到的三维显著图由于考虑了对比度特征、通用特征和背景先验特征,因此能够很好地符合显著语义的特征。Below, the three-dimensional saliency maps of Image1, Image2, Image3, Image4 and Image5 five stereoscopic images in the three-dimensional human eye tracking database (3Deye-tracking database) provided by the University of Nantes in France are extracted by using the method of the present invention. Figure 2a shows the right view image of "Image1", Figure 2b shows the real eye movement image of the right view image of "Image1", Figure 2c shows the 3D saliency map of "Image1"; Figure 3a shows the " The right view image of Image2", Figure 3b shows the real eye movement image of the right view image of "Image2", Figure 3c shows the 3D saliency map of "Image2"; Figure 4a shows the right view image of "Image3" , Figure 4b shows the real eye movement image of the right-viewpoint image of "Image3", and Figure 4c shows the 3D saliency map of "Image3"; Figure 5a shows the right-viewpoint image of "Image4", and Figure 5b shows the The real eye movement image of the right view point image of "Image4", Fig. 5c shows the 3D saliency map of "Image4"; Fig. 6a shows the right view point image of "Image5", and Fig. 6b shows the right view point of "Image5" The real eye movement diagram of the image, Figure 6c shows the 3D saliency map of "Image5"; Figure 7a shows the right viewpoint image of "Image6", and Fig. 7b shows the real eye movement diagram of the right viewpoint image of "Image6" , Figure 7c shows the 3D saliency map of "Image6"; Figure 8a shows the right view image of "Image7", Figure 8b shows the real eye movement image of the right view image of "Image7", and Figure 8c shows The 3D saliency map of "Image7". It can be seen from Figures 2a to 8c that the 3D saliency map obtained by the method of the present invention can well conform to the features of saliency semantics because the contrast feature, general feature and background prior feature are considered.

Claims (6)

1. A method for extracting a three-dimensional image saliency map is characterized by comprising a training stage and a testing stage, wherein the training stage comprises the following specific steps:
① -1, the selected N sets of different stereo images and the right parallax image of each stereo image form a set, which is marked as { Li,Ri,diI is more than or equal to 1 and less than or equal to N, wherein N is more than or equal to 1, and LiRepresents { Li,Ri,diL 1. ltoreq. i. ltoreq.N } left viewpoint image of ith stereoscopic image, RiRepresents { Li,Ri,diI is not less than 1 and not more than N, diRepresents { Li,Ri,diI is not less than 1 and not more than N, and right parallax images of the ith stereo image;
① -2, using superpixel splitting technique to divide { Li,Ri,diI is more than or equal to 1 and less than or equal to N, the right viewpoint image of each stereo image is divided into M non-overlapping areas, R isiThe h-th area in (1) is denoted as SPi,hWherein M is more than or equal to 1, h is more than or equal to 1 and less than or equal to M;
① -3, calculation of { Li,Ri,diI is not less than 1 and not more than N, R is the contrast characteristic vector of each region in the right viewpoint image of each stereoscopic imageiH-th area SP of (1)i,hIs recorded as a contrast feature vector Wherein,has a dimension of 36, here the symbol "[ 2 ]]"is a vector representing a symbol, di,hRepresents SPi,hFirst feature vector u ofi,hThe distance from the first feature vector of the neighboring area,fi,hhas a dimension of 20, fi,hRepresents SPi,hC mean value of the frequency response feature vectors of all the pixels in (1)i,hHas a dimension of 9, ci,hRepresents SPi,hThe mean of the color feature vectors of all the pixel points in (a),represents SPi,hIs measured in the mean value of the parallax amplitude of (c),represents SPi,hColor histograms of R, G and B components of all pixel points in RGB color space and SPi,hThe distances of all pixel points in the adjacent regions of the color histogram of the R component, the G component and the B component of the RGB color space,represents SPi,hThe color histograms of L component, a component and b component of all pixel points in CIELAB color space and SPi,hThe distances of the color histograms of the L component, the a component and the b component of the CIELAB color space of all the pixel points in the neighboring region of (a),represents SPi,hColor histogram and SP of H component of all pixel points in HVS color spacei,hAll pixel points in the neighboring region of (a) are at the distance of the color histogram of the H component of the HVS color space,represents SPi,hColor histogram of S component of all pixel points in HVS color space and SPi,hThe distance of all pixel points in the neighboring region of (a) in the color histogram of the S component of the HVS color space,represents SPi,hLBP feature statistical histogram and SP of all pixel points ini,hThe distance of the statistical histogram of the LBP features of all pixel points in the neighboring region,represents SPi,hThe parallax statistical histogram and SP of all the pixel points ini,hThe distance of the parallax statistical histogram of all the pixel points in the adjacent region, where the adjacent region is RiNeutral SPi,hAn adjacent region;
① -4, calculation of { Li,Ri,diI is not less than 1 and not more than N, R is the general feature vector of each region in the right viewpoint image of each stereoscopic imageiH-th area SP of (1)i,hIs given as Wherein,has a dimension of 33, here the symbol "[ 2 ]]"is a vector representing a symbol and,has a dimension of 20 a and has a high degree of,represents SPi,hThe variance of the frequency response feature vectors of all the pixel points in (a),has a dimension of 9 a and has a high degree of,represents SPi,hThe variance of the color feature vectors of all the pixel points in (a),represents SPi,hOf the parallax amplitude, xi,hHas a dimension of 2, xi,hRepresents SPi,hS of the center pixel pointi,hRepresents SPi,hThe area of (d);
① -5, calculating { Li,Ri,diI 1 ≦ i ≦ N } for each region in the right view image for each stereoscopic imageA background prior feature vector of (2), RiH-th area SP of (1)i,hIs recorded as Wherein,has a dimension of 36, here the symbol "[ 2 ]]"is a vector representing a symbol, ei,hRepresents SPi,hFirst feature vector u ofi,hThe distance from the first feature vector of the background region,fi,hhas a dimension of 20, fi,hRepresents SPi,hC mean value of the frequency response feature vectors of all the pixels in (1)i,hHas a dimension of 9, ci,hRepresents SPi,hThe mean of the color feature vectors of all the pixel points in (a),represents SPi,hIs measured in the mean value of the parallax amplitude of (c),represents SPi,hThe color histogram of R component, G component and B component of all pixel points in RGB color space and RiThe distances of all the pixel points in the background region in the color histogram of the R component, G component and B component of the RGB color space,represents SPi,hThe color histograms of L component, a component and b component and R of all pixel points in CIELAB color spaceiOf all pixels in the background region in CIELAB color spaceThe distances of the color histograms of the L component, the a component and the b component,represents SPi,hColor histogram and R of H component of all pixel points in HVS color spaceiThe distance of all the pixel points in the background region in the color histogram of the H component of the HVS color space,represents SPi,hColor histogram of S component of all pixel points in HVS color space and RiThe distance of all pixel points in the background region in the color histogram of the S component of the HVS color space,represents SPi,hLBP feature statistical histogram and R of all pixel points iniThe distance of the statistical histogram of the LBP features of all the pixels in the background region in (1),represents SPi,hThe parallax statistical histogram and R of all the pixel points iniThe distance of the parallax statistical histogram of all the pixel points in the background region is RiThe areas positioned at the leftmost side, the rightmost side, the uppermost side and the lowermost side;
① -6, will { Li,Ri,diI is more than or equal to 1 and less than or equal to N), the contrast characteristic vector, the general characteristic vector and the background prior characteristic vector of each region in the right viewpoint image of each stereo image are arranged in sequence to form { L ≦i,Ri,diI 1 ≦ i ≦ N } for each region in the right view image of each stereoscopic image, R is a feature vector for reflecting visual saliencyiH-th area SP of (1)i,hThe feature vector for reflecting the visual saliency is marked as Xi,hWherein, Xi,hHas a dimension of 105, here the symbol "[ 2 ]]"is a vector representation symbol;
① -7, using random forest regression, on { L }i,Ri,diI is more than or equal to 1 and less than or equal to N), training the feature vectors for reflecting the visual saliency in all the regions of the right viewpoint images of all the stereo images, minimizing the error between the regression function value obtained through training and the average eye movement value, obtaining the optimal random forest regression training model, and marking the optimal random forest regression training model as f (D)inp) Wherein f () is a functional representation form, DinpRepresenting an input vector of a random forest regression training model;
the specific steps of the test stage are as follows:
② -1, for any one test stereo image StestWill StestThe left viewpoint image, the right viewpoint image, and the right parallax image are expressed as Ltest、Rtest、dtest(ii) a Then adopting super pixel segmentation technique to divide RtestDividing into M non-overlapping regions, and dividing RtestThe h-th area in (1) is denoted as SPh'; wherein M is more than or equal to 1, h is more than or equal to 1 and less than or equal to M;
② -2, following the procedure of step ① -3 through step ① -6, R is obtained in the same manner of operationtestFor each region of (a) to reflect visual saliency, RtestH-th area SP of (1)h' the feature vector for reflecting the visual saliency is denoted as Ftest,h(ii) a Then, training a model f (D) according to the optimal random forest regression obtained in the training stageinp) Will Ftest,hObtaining R as an input vector of an optimal random forest regression training modeltestOf each region of (a), RtestH-th area SP of (1)h' the three-dimensional visual saliency value is denoted as S3D,h,S3D,h=f(Ftest,h) (ii) a Then R is puttestThe three-dimensional visual saliency value of each region in the R-image is taken as the saliency value of all pixel points in the corresponding region, thereby obtaining RtestIII of (2)Dimension saliency map, denoted as { S3D(x, y) }, wherein (x, y) here denotes StestX is more than or equal to 1 and less than or equal to W, y is more than or equal to 1 and less than or equal to H, and W and H correspondingly represent StestWidth and height of (S)3D(x, y) represents S3DAnd the coordinate position in the (x, y) is the pixel value of the pixel point of (x, y).
2. The method for extracting saliency map of stereoscopic images as claimed in claim 1, wherein said step ① -3RiH-th area SP of (1)i,hContrast feature vector ofThe acquisition process comprises the following steps:
a1, calculating RiH-th area SP of (1)i,hThe mean value of the frequency response characteristic vectors of all the pixel points is recorded as fi,h,fi,hTo (1)The value of each element is equal to RiH-th area SP of (1)i,hIn the frequency response feature vector of all the pixel pointsMean of the frequency response amplitudes of the elements, where fi,hHas a dimension of 20 a and has a high degree of,
a2, calculating RiH-th area SP of (1)i,hThe mean value of the color feature vectors of all the pixel points in (1) is marked as ci,hWherein, ci,hHas a dimension of 9, here the symbol "[ 2 ]]"is a vector representing a symbol and,represents RiH-th area SP of (1)i,hThe mean value of the color values of the R components of the RGB color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hThe average of the color values of the G components of the RGB color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hThe average of the color values of the B components of the RGB color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hThe average value of the color values of the L components of all the pixel points in the CIELAB color space,represents RiH-th area SP of (1)i,hThe average value of the color values of the a component of all the pixel points in the CIELAB color space,represents RiH-th area SP of (1)i,hThe average value of the color values of the b components of all the pixel points in the CIELAB color space,represents RiThe h region SP ofi,hThe average of the color values of the H component of the HVS color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hAll the pixel points inThe mean of the color values of the V component in the HVS color space,represents RiH-th area SP of (1)i,hThe mean value of the color values of the S components of all the pixel points in the HVS color space;
a3, calculating RiH-th area SP of (1)i,hIs the mean of the parallax amplitude of Is equal to diNeutral SPi,hThe mean value of the pixel values of all the pixel points in the corresponding region;
a4, mixing fi,h、ci,hAndarranged in sequence to form RiH-th area SP of (1)i,hIs recorded as a first feature vector ofWherein u isi,hHas a dimension of 30, here the symbol "[ 2 ]]"is a vector representation symbol;
a5, calculating RiH-th area SP of (1)i,hFirst feature vector u ofi,hThe distance from the first feature vector of the neighboring region, denoted di,hWherein d isi,hHas a dimension of 30, p is more than or equal to 1 and less than or equal to M,represents RiH-th area SP of (1)i,hU of adjacent areas of the sequence numberi,pRepresents RiP-th area SP in (1)i,pThe symbol "|" is an absolute value symbol, and P represents RiH-th area SP of (1)i,hIs referred to as RiNeutral SPi,hAn adjacent region;
a6, calculating RiH-th area SP of (1)i,hThe color histograms of R, G and B components of all the pixel points in RGB color space are recorded asCalculation of RiH-th area SP of (1)i,hThe color histograms of the L component, the a component and the b component of all the pixel points in the CIELAB color space are recorded asCalculation of RiH-th area SP of (1)i,hThe color histogram of the H component of all the pixel points in the HVS color space is recorded asCalculation of RiH-th area SP of (1)i,hThe color histogram of S component of all pixel points in HVS color space is recorded asCalculation of RiH-th area SP of (1)i,hThe statistical histogram of the LBP characteristics of all the pixel points in (1) is recorded asCalculation of RiH-th area SP of (1)i,hThe parallax statistical histogram of all the pixel points in (1) is recorded asWherein,has dimension of 163Has dimension of 163Has a dimension of 16 a and has a high degree of,has a dimension of 16 a and has a high degree of,has a dimension of 256, and has a high thermal conductivity,has a dimension of 16;
a7, calculatingAnd RiH-th area SP of (1)i,hThe distances of the color histograms of the R component, the G component and the B component of the RGB color space of all the pixel points in the adjacent areas are recorded as
ComputingAnd RiH-th area SP of (1)i,hThe distances of the color histograms of the L component, the a component and the b component of all the pixel points in the adjacent areas in the CIELAB color space are recorded as
ComputingAnd RiH-th area SP of (1)i,hThe distance of the color histogram of the H component of the HVS color space of all the pixel points in the adjacent region is recorded as
ComputingAnd RiH-th area SP of (1)i,hThe distance of the color histogram of the S component of the HVS color space of all the pixel points in the adjacent region is recorded as
ComputingAnd RiH-th area SP of (1)i,hThe distance of the LBP feature statistical histogram of all the pixel points in the adjacent region is recorded as
ComputingAnd RiH-th area SP of (1)i,hThe distance of the parallax statistical histogram of all the pixel points in the adjacent region is recorded as
Wherein p is more than or equal to 1 and less than or equal to M,represents RiH-th area SP of (1)i,hP represents RiH-th area SP of (1)i,hIs the chi () function of the chi-squared distance,represents RiP-th area SP in (1)i,pThe color histograms of the R, G and B components of all the pixel points in the RGB color space,represents RiP-th area SP in (1)i,pThe color histograms of the L component, the a component and the b component of all the pixel points in the CIELAB color space,represents RiP-th area SP in (1)i,pThe color histogram of the H component of the HVS color space for all the pixel points in (a),represents RiP-th area SP in (1)i,pColor of S component of all pixel points in HVS color spaceThe color histogram is a histogram of the color,represents RiP-th area SP in (1)i,pThe statistical histogram of LBP features of all pixel points in (1),represents RiP-th area SP in (1)i,pThe parallax statistics histogram of all the pixel points in the image;
a8, di,hAndarranged in sequence to form RiH-th area SP of (1)i,hIs recorded as the contrast feature vector of Wherein,has a dimension of 36, here the symbol "[ 2 ]]"is a vector representing a symbol.
3. The method for extracting saliency map of stereoscopic images as claimed in claim 2, wherein said step ① -4RiH-th area SP of (1)i,hIs a universal feature vectorThe acquisition process comprises the following steps:
b1, calculating RiH-th area SP of (1)i,hFrequency response characteristics of all pixel points inVariance of vector, noted To (1)The value of each element is equal to RiH-th area SP of (1)i,hIn the frequency response feature vector of all the pixel pointsThe variance of the frequency response amplitude of the individual elements, wherein,has a dimension of 20 a and has a high degree of,
b2, calculating RiH-th area SP of (1)i,hThe variance of the color feature vectors of all the pixel points is recorded as Wherein,has a dimension of 9, here the symbol "[ 2 ]]"is a vector representing a symbol and,represents RiH-th area SP of (1)i,hThe variance of the color values of the R component of the RGB color space for all pixel points in (a),represents RiH-th area SP of (1)i,hThe variance of the color values of the G component of the RGB color space for all pixel points in (a),represents RiH-th area SP of (1)i,hThe variance of the color values of the B component of the RGB color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hThe variance of the color values of the L component of the CIELAB color space for all pixel points in (a),represents RiH-th area SP of (1)i,hThe variance of the color values of the a component of the CIELAB color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hThe variance of the color values of the b-component in CIELAB color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hThe variance of the color values of the H component of the HVS color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hThe variance of the color values of the V component of the HVS color space for all pixel points in (a),represents RiH of (1)Region SPi,hThe variance of the color values of the S component of all the pixel points in the HVS color space;
b3, calculating RiH-th area SP of (1)i,hThe variance of the parallax amplitude of (1), is recorded as Is equal to diNeutral SPi,hVariance of pixel values of all pixel points in the corresponding region;
b4, obtaining RiH-th area SP of (1)i,hThe coordinate position of the central pixel point is marked as xi,hWherein x isi,hHas a dimension of 2;
b5, calculating RiH-th area SP of (1)i,hArea of (d), denoted as si,h
b6, willxi,hAnd si,hArranged in sequence to form RiH-th area SP of (1)i,hIs a universal feature vector of Wherein,has a dimension of 33, here the symbol "[ 2 ]]"is a vector representing a symbol.
4. The method for extracting saliency map of stereoscopic images as claimed in claim 3, wherein said step ① -5RiH-th area SP of (1)i,hBackground prior feature vector ofThe acquisition process comprises the following steps:
c1, calculating RiH-th area SP of (1)i,hThe mean value of the frequency response characteristic vectors of all the pixel points is recorded as fi,h,fi,hTo (1)The value of each element is equal to RiH-th area SP of (1)i,hIn the frequency response feature vector of all the pixel pointsMean of the frequency response amplitudes of the elements, where fi,hHas a dimension of 20 a and has a high degree of,
c2, calculating RiH-th area SP of (1)i,hThe mean value of the color feature vectors of all the pixel points in (1) is marked as ci,hWherein, ci,hHas a dimension of 9, here the symbol "[ 2 ]]"is a vector representing a symbol and,represents RiH-th area SP of (1)i,hThe mean value of the color values of the R components of the RGB color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hThe average of the color values of the G components of the RGB color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hThe average of the color values of the B components of the RGB color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hThe average value of the color values of the L components of all the pixel points in the CIELAB color space,represents RiH-th area SP of (1)i,hThe average value of the color values of the a component of all the pixel points in the CIELAB color space,represents RiH-th area SP of (1)i,hThe average value of the color values of the b components of all the pixel points in the CIELAB color space,represents RiThe h region SP ofi,hThe average of the color values of the H component of the HVS color space for all the pixel points in (a),represents RiH-th area SP of (1)i,hThe average of the color values of the V components of all the pixel points in the HVS color space,represents RiH-th area SP of (1)i,hThe mean value of the color values of the S components of all the pixel points in the HVS color space;
c3, calculating RiH-th area SP of (1)i,hIs the mean of the parallax amplitude of Is equal to diNeutral SPi,hThe mean value of the pixel values of all the pixel points in the corresponding region;
c4, mixing fi,h、ci,hAndarranged in sequence to form RiH-th area SP of (1)i,hFirst feature vector of (1), denoted as ui,hWherein u isi,hHas a dimension of 30, here the symbol "[ 2 ]]"is a vector representation symbol;
c5, calculating RiH-th area SP of (1)i,hFirst feature vector u ofi,hThe distance from the first feature vector of the background region, denoted as ei,hWherein e isi,hHas a dimension of 30, q is more than or equal to 1 and less than or equal to M,represents RiSet of sequence numbers of all background regions in ui,qRepresents RiThe q-th region SP in (1)i,qThe symbol "|" is an absolute value symbol, and Q represents RiThe total number of background regions in (1), where the background region is RiThe areas positioned at the leftmost side, the rightmost side, the uppermost side and the lowermost side;
c6, calculating RiH-th area SP of (1)i,hThe color histograms of R, G and B components of all the pixel points in RGB color space are recorded asCalculation of RiH-th area SP of (1)i,hThe color histograms of the L component, the a component and the b component of all the pixel points in the CIELAB color space are recorded asCalculation of RiH-th area SP of (1)i,hThe color histogram of the H component of all the pixel points in the HVS color space is recorded asCalculation of RiH-th area SP of (1)i,hThe color histogram of S component of all pixel points in HVS color space is recorded asCalculation of RiH-th area SP of (1)i,hThe statistical histogram of the LBP characteristics of all the pixel points in (1) is recorded asCalculation of RiH-th area SP of (1)i,hThe parallax statistical histogram of all the pixel points in (1) is recorded asWherein,has dimension of 163Has dimension of 163Has a dimension of 16 a and has a high degree of,has a dimension of 16 a and has a high degree of,has a dimension of 256, and has a high thermal conductivity,has a dimension of 16;
c7, calculationAnd RiThe distances of the color histograms of the R component, the G component and the B component of the RGB color space of all the pixel points in the background region are recorded as
ComputingAnd RiThe distances of the color histograms of the L component, the a component and the b component of all the pixel points in the background area in the CIELAB color space are recorded as the distances
ComputingAnd RiThe distance of the color histogram of the H component of the HVS color space of all the pixel points in the background area is recorded as
ComputingAnd RiThe distance of the color histogram of the S component of the HVS color space of all the pixel points in the background area is recorded as
ComputingAnd RiThe distance of the LBP feature statistical histogram of all the pixel points in the background area is recorded as
ComputingAnd RiThe distance of the parallax statistical histogram of all the pixel points in the background region is recorded as
Wherein q is more than or equal to 1 and less than or equal to M,represents RiAll ofSet of sequence numbers of background regions, Q denotes RiThe total number of background regions, χ () is a chi-squared distance function,represents RiThe q-th region SP in (1)i,qThe color histograms of the R, G and B components of all the pixel points in the RGB color space,represents RiThe q-th region SP in (1)i,qThe color histograms of the L component, the a component and the b component of all the pixel points in the CIELAB color space,represents RiThe q-th region SP in (1)i,qThe color histogram of the H component of the HVS color space for all the pixel points in (a),represents RiThe q-th region SP in (1)i,qThe color histogram of the S component of the HVS color space for all the pixel points in (a),represents RiThe q-th region SP in (1)i,qThe statistical histogram of LBP features of all pixel points in (1),represents RiThe q-th region SP in (1)i,qThe parallax statistics histogram of all the pixel points in the image;
c8, mixingi,hAndarranged in sequence to form RiH-th area SP of (1)i,hIs recorded as the background prior feature vector of Wherein,has a dimension of 36, here the symbol "[ 2 ]]"is a vector representing a symbol.
5. The method according to any one of claims 2 to 4, wherein R is the number of pixels in the R-mapiThe obtaining process of the frequency response characteristic vector of each pixel point comprises the following steps:
1) -1, using a Gabor filter pair RiFiltering to obtain RiUnder different central frequencies and different direction factors, each pixel point in the signal is subjected to frequency response amplitude RiThe frequency response amplitude of the pixel point with the middle coordinate position (x, y) under the condition that the center frequency is omega and the direction factor is theta is marked as G (x, y; omega, theta), wherein (x, y) represents { L }i,Ri,diI is more than or equal to 1 and less than or equal to N, x is more than or equal to 1 and less than or equal to W, y is more than or equal to 1 and less than or equal to H, and W and H correspondingly represent { L ≦ N }i,Ri,diI 1 ≦ i ≦ N }, ω represents the center frequency of the Gabor filter, ω ∈ Φωθ represents the directional factor of the Gabor filter, θ ∈ Φθ,ΦωRepresenting the set of all centre frequencies of the Gabor filter, phiθRepresents the set of all directional factors of the Gabor filter;
1) -2, reacting RiThe frequency response amplitude of each pixel point in the R-array is arranged in sequence under different central frequencies and different direction factors to form RiThe frequency response feature vector of each pixel point in (a),r is to beiThe frequency response characteristic vector of the pixel point with the middle coordinate position (x, y) is marked as fi(x, y) wherein fiThe dimension of (x, y) is 20.
6. The method as claimed in claim 5, wherein R is a number of termsiThe obtaining process of the color feature vector of each pixel point comprises the following steps:
2) -1, calculating RiThe color value of each pixel point in different color spaces is RiThe color values of the R component, the G component and the B component of the pixel point with the (x, y) middle coordinate position in the RGB color space are respectively recorded as R (x, y), G (x, y) and B (x, y), and R (x, y) is recordediThe color values of the L component, the a component and the b component of the pixel point with the middle coordinate position (x, y) in the CIELAB color space are respectively recorded as L (x, y), a (x, y) and b (x, y), and R is used for indicating the color values of the L component, the a component and the b component in the CIELAB color space as L (x, y), a (x, y) and b (iThe color values of the H component, the V component and the S component of the pixel point with the (x, y) middle coordinate position in the HVS color space are respectively recorded as H (x, y), V (x, y) and S (x, y), wherein (x, y) here represents { L }i,Ri,diI is more than or equal to 1 and less than or equal to N, x is more than or equal to 1 and less than or equal to W, y is more than or equal to 1 and less than or equal to H, and W and H correspondingly represent { L ≦ N }i,Ri,diThe width and height of the stereoscopic image in the condition that i is more than or equal to 1 and less than or equal to N;
2) -2, reacting RiThe color values of each pixel point in the different color spaces are arranged in sequence to form RiThe color feature vector of each pixel point in (1), RiThe color feature vector of the pixel point with the middle coordinate position (x, y) is marked as ci(x,y),ci(x,y)=[R(x,y),G(x,y),B(x,y),L(x,y),a(x,y),b(x,y),H(x,y),V(x,y),S(x,y)]Wherein c isiThe dimension of (x, y) is 9, where the symbol "[ 2 ],"]"is a vector representing a symbol.
CN201510173098.5A 2015-04-14 2015-04-14 A kind of stereo-picture notable figure extracting method Active CN104822058B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510173098.5A CN104822058B (en) 2015-04-14 2015-04-14 A kind of stereo-picture notable figure extracting method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510173098.5A CN104822058B (en) 2015-04-14 2015-04-14 A kind of stereo-picture notable figure extracting method

Publications (2)

Publication Number Publication Date
CN104822058A CN104822058A (en) 2015-08-05
CN104822058B true CN104822058B (en) 2016-08-17

Family

ID=53732209

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510173098.5A Active CN104822058B (en) 2015-04-14 2015-04-14 A kind of stereo-picture notable figure extracting method

Country Status (1)

Country Link
CN (1) CN104822058B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105338335B (en) * 2015-10-15 2017-03-29 宁波大学 A kind of stereo-picture notable figure extracting method
EP3403403B1 (en) * 2016-01-12 2023-06-07 Shanghaitech University Calibration method and apparatus for panoramic stereo video system
CN109167988B (en) * 2018-08-29 2020-06-26 长春理工大学 Stereo image visual comfort evaluation method based on D + W model and contrast
CN109523534A (en) * 2018-11-14 2019-03-26 北京奇艺世纪科技有限公司 A kind of salient region detection method and device of view-based access control model angle

Also Published As

Publication number Publication date
CN104822058A (en) 2015-08-05

Similar Documents

Publication Publication Date Title
US11830230B2 (en) Living body detection method based on facial recognition, and electronic device and storage medium
Li et al. A weighted sparse coding framework for saliency detection
EP3333768A1 (en) Method and apparatus for detecting target
CN106504276A (en) The combinations matches cost algorithms of non local Stereo Matching Algorithm and parallax joint filling algorithm
CN103020265B (en) The method and system of image retrieval
CN104822058B (en) A kind of stereo-picture notable figure extracting method
CN105389554A (en) Live body discrimination method and device based on face recognition
CN101833767B (en) Automatic linear matching method based on gradient and color features in digital image
CN105869173A (en) Stereoscopic vision saliency detection method
CN104574375A (en) Image Saliency Detection Method Combining Color and Depth Information
CN111160478B (en) Hyperspectral target significance detection method based on deep learning
CN104392233B (en) A kind of image saliency map extracting method based on region
CN106780476A (en) A kind of stereo-picture conspicuousness detection method based on human-eye stereoscopic vision characteristic
CN102129694B (en) A method of image salient region detection
CN106651853A (en) Establishment method for 3D saliency model based on prior knowledge and depth weight
CN104408711A (en) Multi-scale region fusion-based salient region detection method
CN107886507A (en) A kind of salient region detecting method based on image background and locus
CN110807379A (en) A semantic recognition method, device, and computer storage medium
WO2023061195A1 (en) Image acquisition model training method and apparatus, image detection method and apparatus, and device
CN106997478A (en) RGB-D image salient target detection method based on salient center prior
CN104243956B (en) A kind of stereo-picture visual saliency map extracting method
CN104331877A (en) Color image edge detection method based on fusion color gradient
CN111325107A (en) Detection model training method and device, electronic equipment and readable storage medium
CN103632372B (en) A kind of video saliency image extraction method
CN101866422A (en) A method of extracting image attention based on image multi-feature fusion

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
EXSB Decision made by sipo to initiate substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20191219

Address after: 313000 Huzhou Youyan Intellectual Property Service Co., Ltd

Patentee after: Huzhou You Yan Intellectual Property Service Co., Ltd.

Address before: 315211 Zhejiang Province, Ningbo Jiangbei District Fenghua Road No. 818

Patentee before: Ningbo University

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20200117

Address after: 215000 room 16, floor 4, building 21, Tengfei Innovation Park, No. 388, Xinping street, Suzhou Industrial Park, Jiangsu Province

Patentee after: ZHONGHE INTERNATIONAL INTELLECTUAL PROPERTY CO., LTD.

Address before: 313000 Huzhou Youyan Intellectual Property Service Co., Ltd

Patentee before: Huzhou You Yan Intellectual Property Service Co., Ltd.