Summary of the invention
The objective of the invention is to defective at the prior art existence, a kind of image adaptive method based on vision significance is proposed, the visual effect distortion that this method can be implemented in the intelligent mobile device under the low resolution and the small screen environment picture drops to minimum, and keep the wherein integrality of saliency object, for the beholder provide with high-resolution display device on duplicate visual effect.
In order to achieve the above object, the scheme of the present invention's employing is as follows.
Dynamic programming often is applied to solve optimization problem, choosing the minimum slit of energy in the image also is that (method of finding out the slit has a variety of a kind of optimization problem, but the slit that diverse ways is found out has different energy, the energy in slit refers to adding up of each pixel energy in this slit), so the present invention adopts dynamic programming to find out these slits.
Above-mentioned image adaptive method based on vision significance is characterized in that it at first being the energy that calculates original image; Next is the relative energy that extracts the saliency object in the original image and strengthen saliency object; Utilize the dynamic programming technology to find out in the image the minimum slit of energy on the vertical direction and horizontal direction then, reject these slits, realize the self-adaptation of image.
Its specific implementation step is:
The energy of A, calculating original image: original color image is converted to gray level image, calculates the gradient of gray level image then, the size of each pixel gradient value is the energy value of each pixel of original image correspondence;
B, extract saliency object and improve its relative energy: original color image is carried out color decompose, then carry out the reorganization of difference color, then to the reorganization after image block and calculate corresponding piece average and piece variance, distinguish the information entropy of computing block average and piece variance at last, determine saliency object according to the consistance of institute's computing information entropy.
C, utilize the dynamic programming technology to find out in the image the minimum slit of energy on the vertical direction and horizontal direction, reject these slits: the analysis on the vertical and horizontal direction of definition image, utilize the dynamic programming technology to find out the slit of local optimum repeatedly, rejecting.
The energy of the calculating original image in the steps A of said method by formula (1) calculates:
Adopt gradient operator as energy function.The original image of supposing input is that (m, n), wherein, m, n be the height and width of corresponding original image respectively for I.
Wherein, the energy (hereinafter to be referred as energygram) of E (I) expression original image, ||,
Represent absolute value sign respectively, image is respectively at x, the partial derivative on the y direction.
Saliency object in the extraction original image among the step B of said method and improve its relative energy and carry out according to the following steps:
Quick variation place in the original image is detected by gradient operator, and energy is represented with the numerical value of 0-255 respectively from low to high, the minimum energy of 0 expression, and 255 expression high-energy are the highest.But be to use gradient operator to be merely able to detect quick variation place, for saliency object, it is inner, and its relative energy value is relatively low owing to change gently, can cause when self-adaptation saliency object destroyed like this.The result that this image adaptive that is not based on vision significance is expected must detect salient region and improve its relative energy, makes picture material being kept perfectly property when self-adaptation.This method detects according to following step and strengthens saliency object.
B1, original color image is carried out colour decompose.
If original image is not that (B:blue) image is converted into the RGB image to RGB for R:red, G:green, carries out color according to formula (2) then and decomposes.
Wherein, r, g, b represent three Color Channel values of original RGB image: redness, and green and blue, R
New, G
New, B
New, Y
NewMonochrome image after expression separates respectively: redness, green, blue and yellow.
B2, the monochrome image after will separating carry out mutual calculus of differences, obtain 6 kinds of difference images altogether.
With the monochrome image R after separating
New, G
New, B
New, Y
NewThe calculus of differences that process is mutual, with the calculus of differences between the Θ presentation video, concrete steps as shown in Equation (3).
RG
Diff, RB
Diff, RY
Diff, GB
Diff, GY
Diff, BY
DiffCorresponding red green respectively, red indigo plant, reddish yellow, turquoise, greenish-yellow and blue yellow difference image.
B3, the piece average of calculating difference image and piece variance and carry out binaryzation.
Each difference image is carried out piecemeal, and (size of each piece is N * M for i, j) expression with Block.Calculate the piece average and the piece variance of difference image then according to formula (4).
σ
I, jAnd μ
I, jRepresent piece Block (i, deviation j) and average, I respectively
I, j(x, y) expression piece Block (i, j) Nei pixel.This method adopts a kind of block-based local quantization method: if σ
I, jAnd μ
I, jRespectively greater than T
σ UpAnd T
μ Up, establish σ
I, j=255, μ
I, j=255; If σ
I, jAnd μ
I, jRespectively less than T
σ LowAnd T
μ Low, establish σ
I, j=0, μ
I, j=0; Calculate the quantity of information that quantizes back average image and offset images, select best difference image, draw conspicuousness figure by the comparison information amount.
According to Shannon information theory principle, calculate the information entropy of each average image and variance image, consistance according to information entropy draws the possibility of each difference image as the conspicuousness image, picks out that image that comprises saliency object and the relative energy that improves saliency object.Shown in the calculating of information entropy (5) formula.
Entropy=-log(P(x)) (5)
Wherein, Entropy represents information entropy, and P (x) represents the probability of corresponding average image of certain difference image or the shared whole total pixel of variance image maximal value pixel.The consistance criterion is: if the relative distance of the quantity of information entropy of average image after quantizing and variance image is near more, the probability as the conspicuousness image of corresponding difference image is big more, finds out the energy that the conspicuousness image strengthens the conspicuousness image then.Last raising the conspicuousness image of relative energy.
Utilize the slit that the dynamic programming technology determines that energy is minimum on the vertical direction and horizontal direction in the image and the step of rejecting the slit as follows among the step C of said method:
The definition in C1, slit.
Following two conditions must strictness be satisfied in vertical direction and the slit on the horizontal direction in this method definition image:
A., for each bar slit, no matter be vertical direction or horizontal direction, only account for a pixel at each row or each row, this expression slit is absolute monotonic;
B. the slit must be 8 connections.The reason that defines these two restrictive conditions is: avoid the bird caging of image in the adaptive process.
Slit on the definition vertical direction is:
Slit on the definition horizontal direction is:
Wherein, Seam represents slit x (i), and y (j) represents the mapping on vertical direction and the horizontal direction respectively, m, and n, || the height of the corresponding original image of difference, wide and absolute value sign, the slit on vertical direction and the horizontal direction is respectively as Fig. 6 a, shown in the 6b.
C2, utilize dynamic programming to find out the minimum slit of energy.
The purpose of finding out the slit is to reject the slit to reach the adaptively changing of image size.For the distortion of image after the MIN minimizing self-adaptation with keep the integrality of saliency object, must determine slit in the image with a kind of method of optimum.Adopt dynamic programming to determine optimum slit in this method.
Dynamic programming on the vertical direction:
Task(i,j)=E(i,j)+S(i,j)+min(Task(i-1,j-1),Task(i-1,j),Task(i-1,j+1)) (8)
Dynamic programming on the horizontal direction:
Task(i,j)=E(i,j)+S(i,j)+min(Task(i-1,j-1),Task(i,j-1),Task(i+1,j-1)) (9)
Task (i, j), E (i, j), S (i, j), (x, y z) represent the image that dynamic programming calculates respectively to min, the energygram picture of original image, the minimum value function of conspicuousness image and three numbers, i ∈ [0, height), and j ∈ [0, width), height, the height and width of the corresponding original image of width.Slit deterministic process with vertical direction is an example: at first go to the capable Task that calculates of M-1 from the 0th according to the dynamic programming method on the vertical direction; Find out the capable minimum value of M-1 then; Date back to the 0th row from M-1 according to the thinking of dynamic programming at last.Shown in Figure 10,11, suppose that matrix shown in Figure 10 is corresponding with a certain image, three red arrows are represented computation sequence (adopting order from top to down in this programme), coloured square represents that the value in this square is calculated.Suppose to calculate the value that the second row secondary series position goes out among Figure 10, according to connectedness and the monotonicity in slit, front (Seam) definition, the value of current position can only with previous row in relevant with the value of its three the most contiguous positions, utilize these three values of having calculated to determine the value of this position, guarantee simultaneously the value calculate be institute might the result of computing method gained in the value of minimum.With reference to Figure 10, the value of utilizing dynamic programming can obtain this position is: 7=2+min (5,8,12), min (a wherein, b, c) minimum value of three numbers is asked in expression, the preliminary energy in the second row secondary series place in " 2 " representing matrix, " 5 ", " 8 " and " 12 " expression relevant position good as calculated value, calculating the value of coming that other method calculates must be worth more than or equal to this.After the value at the second row secondary series place has been calculated, then calculate the value (i.e. second row the 3rd row) of the next position then, calculate last row of last column always.Among Figure 11, carry out the track of dynamic programming operation in the red lines presentation video, just the slit of mentioning repeatedly in this method (Seam).By repeatedly find out and reject these slits, just can reach the purpose of image adaptive.
The self-adaptation of C3, realization image.
In this method, the self-adaptation of image realizes the self-adaptation of image according to the following steps.Because wide self-adaptation and high adaptive approach be identical, so following step mainly is at wide or high self-adaptation (high self-adaptation wide adaptive step identical).If the original image of input is that (m, n), wherein, m, n be the height and width of corresponding original image respectively for I.The size of the target image that need obtain is m ', n '.Δ M=m '-m represents width or the height that needs change;
C31, find out a slit on the vertical direction, represent with Seam according to the method for introducing above;
C32, reject this slit: find out since first row and belong to the residing position of the capable pixel of i among the Seam, wherein, i ∈ [0, m), the pixel on this right, position is moved to the left a unit one by one, delete capable all pixels of n-1.After finishing this step, the width of image just reduces one, becomes m-1, simultaneously Δ M=Δ M-1;
C33, if Δ M=0, just finish; If be not 0, repeat A, the operation of B is till Δ M=0.
By top A, B, the operation of C just can be finished the self-adaptation of picture traverse.Adaptive approach to height also is the same thinking.
Image adaptive method based on vision significance of the present invention has following advantage compared with prior art: this adaptive approach is the minimum slit of energy in the image owing to what reject, so integral image energy loss minimum; Because the relative energy of saliency object is higher, saliency object can not passed in the slit of rejecting, so the result of saliency object and shape can be kept perfectly behind the image adaptive.Therefore, the visual effect distortion that this method can be implemented in the intelligent mobile device under the low resolution and the small screen environment picture drops to minimum, and keep the wherein integrality of saliency object, for the beholder provide with high-resolution display device on duplicate visual effect.
Embodiment
Details are as follows in conjunction with the accompanying drawings for the embodiment of the image adaptive method based on vision significance of the present invention:
With reference to Fig. 1, show the process flow diagram of the image adaptive method based on vision significance of the present invention, can under the environment of low resolution and the small screen, realize self-adaptation at picture material.Be that programming realizes on the PC test platform of 1.66GHz, internal memory 1024M at CPU, provided some results in the processing procedure.Its specific implementation step is:
The energy of A, calculating original image: the original image shown in Fig. 2 a, 2b, 2c, original color image is converted to gray level image, calculate the gradient of gray level image then, the size of each pixel gradient value is the energy value of each pixel of original image correspondence;
B, extract saliency object and improve its relative energy: original color image is carried out color decompose, then carry out the reorganization of difference color, then to the reorganization after image block and calculate corresponding piece average and piece variance, distinguish the information entropy of computing block average and piece variance at last, determine saliency object according to the consistance of institute's computing information entropy;
C, utilize the dynamic programming technology to find out in the image the minimum slit of energy on the vertical direction and horizontal direction, reject these slits: the analysis on the vertical and horizontal direction of definition image, utilize the dynamic programming technology to find out the slit of local optimum repeatedly, reject these slits and reached the purpose that changes the change of image size.
The energy of the calculating original image in the steps A of said method by formula (1) calculates:
This core concept based on the adaptive approach of vision significance is to reject some low-energy pixel (slit is made up of the low-yield pixel that satisfies certain condition) in the original image.How therefore, primary problem is: is energy to know that pixel low, the energy height of that pixel? say that instinctively best way is to remove those " inapparent " pixels.People are when browsing pictures in addition, the easier attraction of border or edge people's attention in the image.Gradient operator (Gradient operator) is based on the local derivative of image function, and is bigger in the fast-changing position of image function (edge), and the effect of gradient operator manifests these positions exactly in image.In this method, adopt gradient operator as energy function.Original image shown in Fig. 2 c, the original image of input be I (m, n), m wherein, n is the height and width of corresponding original image respectively,
Wherein, the energy (hereinafter to be referred as energygram) of E (I) expression original image, ||,
Represent absolute value sign respectively, image is respectively at x, the partial derivative on the y direction.
At first original colorful image is converted to gray level image, then, gray level image utilization (1) formula of gained is carried out the gradient computing, obtain the energy of image, the result as shown in Figure 3.
Saliency object in the extraction original image among the step B of said method and improve its relative energy and carry out according to the following steps:
Quick variation place in the original image is detected by gradient operator, and energy is represented with the numerical value of 0-255 respectively from low to high, the minimum energy of 0 expression, and 255 expression high-energy are the highest.But be to use gradient operator to be merely able to detect quick variation place, for saliency object, it is inner, and its relative energy value is relatively low owing to change gently, can cause when self-adaptation saliency object destroyed like this.The result that this image adaptive that is not based on vision significance is expected must detect salient region and improve its relative energy, makes picture material being kept perfectly property when self-adaptation.This method detects according to following step and strengthens saliency object.
The extraction of saliency object and its energy enhancement process performing step in the image among the step B of said method:
B1, original color image is carried out color decompose;
B2, the image after will decomposing carry out calculus of differences, obtain difference image, and calculated result is shown in Fig. 4 a, 4b, 4c, 4d, 4e, 4f.
B3, calculate the average image and the variance image of each difference image respectively, shown in Fig. 5 a, 5b.
B4 calculates the average image of each difference image and the information entropy of difference image respectively.
B5 is the average information entropy and the variance information entropy of each difference image relatively, draws conspicuousness image and enhancing at last, and the result who draws is shown in Fig. 5 c.
The dynamic programming technology of utilizing among the step C of said method is found out in the image the minimum slit of energy on the vertical direction and horizontal direction, rejects these slits, to finish the performing step of image adaptive:
C1, on the basis of the first, two energy that calculates of step and conspicuousness image, according to (6), the vertical direction that (7) formula defines and the slit of horizontal direction;
C2, utilize the dynamic programming technology to seek slit on vertical direction and the horizontal direction, reject the slit, whole the right, mobile slit or following pixel obtain Fig. 6 a, the 6b image of rejecting the slit;
C3, obtain the size of target image, carry out C1 repeatedly according to practical application, the operation of C2, till satisfying target, adaptive net result is shown in Fig. 7 b.
Can be implemented in the intelligent mobile device under the low resolution and the small screen environment visual effect distortion with picture as mentioned above drops to minimum and keeps the wherein integrality of saliency object.This invention can solve the problem that video effect causes loss in detail and sight to reduce because of resolution descends effectively.According to the program flow diagram of Fig. 1, below provide the example of realization, the type of picture is not subjected to any restriction, can be landscape figure, geometry mechanism map and saliency object figure.Fig. 4 to Fig. 7 has provided corresponding results in the processing procedure.Carry out the explanation of each several part test below in conjunction with program flow diagram.
Test: the present invention mainly be for solve traditional adaptive approach to image manipulation after the problem of the image quality decrease of bringing, compare traditional method, this method can keep the complete integrality of image saliency object when changing the picture size size, this method mainly is divided into three parts.According to process flow diagram shown in Figure 1: the preliminary energy of computed image at first, extract saliency object then and strengthen its energy, utilize dynamic programming to seek and reject the minimum slit of energy in the image at last.The preliminary energygram of the original image of mentioning in the steps A of the corresponding said method of Fig. 3, Fig. 5 c correspondence be figure after extracting the saliency object in the original image and strengthening the saliency object energy, Fig. 7 a is an original image, 7b is the self-adaptation figure that classic method obtains for the adapting to image 7c correspondence that obtains with method of the present invention, can significantly find out: under the situation that changes width, method of the present invention can keep the integrality of saliency object (child is saliency object among this figure), and classic method can cause the distortion of saliency object, and method of the present invention has been brought very big improvement on visual effect.Under low resolution and the small screen environment, very big content distortion has appearred in geometry figure and saliency object figure as can be seen from Figure 9, and the distortion of landscape figure is less relatively.