CN106331680A - Method and system for 2D-to-3D adaptive cloud unloading on handset - Google Patents

Abstract

The present invention discloses a method and a system for 2D-to-3D adaptive cloud unloading on a handset. The method comprises a first step of inputting a frame of 2D monocular image, and dividing the image into N equal image blocks; a second step of classifying the image blocks according to perspective view, linear view and common view; a third step of according to the category of the classified views, calculating algorithm complexity of depth estimation of each image block; a fourth step of substituting the algorithm complexity of each image block into a cloud unloading dynamic resource allocation model, and optimizing to obtain an allocation result; and a fifth step of according to the allocation result obtained in the fourth step, performing depth estimation at the handset end and the cloud end, so as to generate a depth graph. The cloud unloading dynamic resource allocation model is established, so that the cloud computing-based method and system for 2D-to-3D adaptive unloading on the handset are formed. Complex computing at the handset end is unloaded to the cloud end, so that the stored resource of the handset end is released, the handset processing speed is improved, and the handset power consumption is lowered, further, a high-quality depth graph which is reasonable in depth estimation and has high running speed is obtained.

Description

A kind of mobile phone terminal 2D turns 3D self adaptation cloud discharging method and system

Technical field

The present invention relates to computer vision and digital image processing field, turn 3D self adaptation particularly to a kind of mobile phone terminal 2D The method and system of cloud unloading.

Background technology

In the last few years, along with updating of mobile device technology and gradually popularizing of mobile phone, people came into intelligence In the epoch of energy mobile phone, user increasingly gets used to using mobile device end to replace legacy equipment to watch picture and video.Intelligence The development of the hardware handles equipment such as mobile phone polycaryon processor memory GPU so that from strength to strength, mobile phone is relevant should for the function of mobile phone With the most constantly upgrading, mobile phone 3D-TV is one of its Main way.But the shooting of 3D video is complicated, the post-production cycle The factors such as length result in its resource and compare shortage, seriously govern the development of mobile terminal 3D video.By original substantial amounts of 2D Resource conversion becomes 3D to be the effective way solving this problem.

It is estimation of Depth and virtual view synthesis that 2D turns two committed steps in 3D technology.Estimation of Depth be from a width or Efficiently extracting depth information in multiple image, the depth map of reconstruction may be used for 3D modeling, virtual perspective renders, video editing Etc. many aspects.In high-quality depth map image to be embodied, each point is at the correct depth of space corresponding point, also wants energy Process picture noise, the degree of depth of low texture region and regional occlusion problem accurately.As the basis of numerous application, the degree of depth is estimated The effect of meter also functions to vital effect in stereoscopic vision.

It is limited that the depth estimation algorithm of current existing 2D image adapts to scene, and stereoeffect is bad, and complexity is high. The development of the technology that 2D based on mobile phone terminal turns 3D is more immature, and its principal element limited has: 2D turns 3D to be needed the biggest Storage resources, challenge beyond doubt the hugest for this mobile phone terminal very limited amount of to storage volume；Next to that the process of mobile phone Speed is very limited, is difficult to meet 2D and turns the real-time demand of 3D；And, either video conversion or the broadcasting of video all can Consume substantial amounts of electricity, even if technology of filling is dodged in a lot of mobile phone support, but or user can be brought inconvenience.

Summary of the invention

Present invention is primarily targeted at, overcome the deficiencies in the prior art, it is provided that a kind of mobile phone terminal 2D turns 3D self adaptation cloud The method and system of unloading, is unloaded to high in the clouds by the calculating that mobile phone terminal is complicated, thus discharges the storage resource of mobile phone terminal, improves hands The processing speed of machine, reduces the power consumption of mobile phone.

The invention provides a kind of method that mobile phone terminal 2D turns the unloading of 3D self adaptation cloud, it is characterised in that include walking as follows Rapid:

A1. input a frame 2D monocular image, image is divided into N number of image block；

A2. image block is classified, be categorized as prospective view, linear view and general view；

A3. basis has divided the classification of view, calculates the algorithm complex of each image block estimation of Depth respectively；

A4. by the algorithm complex of calculated for A3 each image block, high in the clouds unloading Dynamic Resource Allocation for Multimedia model is substituted into, Optimization obtains allocation result；

A5. the allocation result obtained according to A4 optimization, carries out estimation of Depth at mobile phone terminal and high in the clouds respectively, generates the degree of depth Figure.

Preferably, described step A2 image block classification comprises the steps:

A201. each image block is converted into HSI space by rgb space, calculates the pixel value in HSI space, according to setting Threshold classification go out prospective view and non-prospective view；

A202. non-prospective view is carried out vanishing point detection, if vanishing point being detected, being then linear view, being otherwise general Intervisibility figure.

Preferred further, the sorting technique of described step A201 prospective view and non-prospective view is: calculates image and sits Mark (x, y) place HSI spatial pixel values H (x, y), S (x, y), I (x, y), if 100 < H (x, y) < 180 and 100 < S (and x, y) < 255, then Sky (x, y)=1；If 50 < H (x, y) < 100 and 100 < S (and x, y) < 255, then Ground (x, y)=1；IfIf Amount is more than the threshold value set, then it is prospective view, is otherwise non-distant view View.

Preferred further, described step A202 vanishing point is detected as: utilize the edge that Canny operator calculates image, profit Edge line detection is carried out, according to the breakpoint detection vanishing point of straight line with Hough.

Further preferred, the detection formula of described vanishing point detection is:

Wherein, (x₀,y₀) it is the image block vanishing point in image plane, (ρ_i,θ_i) it is the image block point (x that corresponds to image plane_i, y_i) at polar coordinate, W_iFor corresponding weight.

Preferably, described step A3 calculates algorithm complex to include the following:

A301. for prospective view, image block is split with k-means algorithm, near with the execution time of k-means algorithm Algorithm complex like estimation of Depth；

A302. for general view, detecting image block with graph cut algorithm, the region that can detect is front Scape, is otherwise background, foreground and background calculates the execution time of graph cut algorithm respectively, with holding of graph cut algorithm The algorithm complex that row time approximate depth is estimated；

A303. for linear view, detect image block with the segmentation of k-means algorithm and vanishing point simultaneously, calculate it respectively and calculate The execution time of method, take the algorithm complex that bigger execution time approximate depth is estimated.

The algorithm complex of estimation of Depth depends on the complexity of each algorithm, and during the execution of the complexity of algorithm and algorithm Between the relation that is proportionate, therefore the algorithm complex can estimated by the execution time approximate depth of this algorithm.

Preferred further, the formula that the algorithm complex of described prospective view image block is expressed as:

C₁≈(8+12α_i) WH/S, formula (2)

The formula that the algorithm complex of general view image block is expressed as:

C₃≈β_iWH/S, formula (3)

The formula that the algorithm complex of linear view image block is expressed as:

C₂≈max(γ,8+12α_i) WH/S, formula (4)

Wherein,

β_iIterations for graph cut algorithm；C₁、C₂、C₃It is respectively prospective view, linear view, general view figure Algorithm complex as block；W and H is respectively width and the height of current image block；S is the area for normalized image block, S =87296；α_iFor the number of clusters divided in k-means algorithm, O_iFor the number of contours of Guan Bi in image, γ is that the image to S size enters The time of row vanishing point detection.

Preferably, described step A4 high in the clouds unloading Dynamic Resource Allocation for Multimedia model is by minimizing the power consumption of mobile phone terminal, optimum Change and obtain allocation result.

First, when not having high in the clouds to unload, the power consumption of mobile phone terminal can be defined as:

Wherein, P_cAnd P_trIt is respectively the mobile phone power consumption when calculating and data transmit, C_allThe instruction number of required algorithm.f For the processing speed of mobile phone terminal, unit is orders per second.D is the size of the data of high in the clouds and mobile phone terminal transmission, and B is band Wide.

When there being high in the clouds to unload, the power consumption of mobile phone terminal is:

Wherein, P_iPower consumption when leaving unused for mobile phone, S is the calculating speed in high in the clouds, C_cAnd C_mRespectively it is assigned to high in the clouds and hands The algorithm complex of machine end.

For minimizing the power consumption of mobile phone terminal, it is considered to the meter between mobile phone terminal and the computing capability in high in the clouds and mobile phone and high in the clouds Calculate speed, according to the algorithm complex of estimation of Depth, propose one and turn the based on high in the clouds unloading dynamic of 3D for mobile phone terminal 2D Resource allocator model.

Preferred further, the expression formula of described high in the clouds unloading Dynamic Resource Allocation for Multimedia model is:

Wherein,

n_c1,n_c2And n_c3Respectively it is unloaded to the number of the prospective view in high in the clouds, linear view and general view.

n_all1,n_all2And n_all3It is respectively prospective view, linear view and total number of general view.

WithRelationship expression such as:Constraints represents and is unloaded to high in the clouds When the quantity of block should should unload lower than without using cloud less than the power consumption of the sum of block and the mobile phone using cloud unloading Power consumption.

The algorithm that the overall algorithm complexity of high in the clouds and mobile phone terminal is respectively the different classes of each image block distributed above is complicated Degree sum:

C_c=(n_c1,n_c2,n_c3)×(C₁,C₂,C₃)^T, formula (12)

C_m=(n_m1,n_m2,n_m3)×(C₁,C₂,C₃)^T, formula (13)

n_m1,n_m2And n_m3Respectively it is unloaded to the number of the block of the prospective view of mobile phone terminal, linear view and general view；

C₁,C₂And C₃It is respectively when the image of corresponding prospective view, linear view and general view carries out estimation of Depth Computation complexity.

By optimization formula (9), can obtainIn the value of each variable, i.e. correspond to unload respectively The number of image block to prospective view, linear view and the general view in high in the clouds.

The present invention also provides for a kind of mobile phone terminal 2D and turns the system of 3D self adaptation cloud unloading, and this system includes image division mould Block, image block classification module, complicated dynamic behaviour module, Dynamic Resource Allocation for Multimedia model module, depth estimation module；Image division mould Block is for by the division of 2D monocular image；Image block classification, for the classification of image block, is that distant view regards by image block classification module Figure, linear view and general view；Complicated dynamic behaviour module is for calculating the algorithm complex of each image block；Dynamic resource divides Join the model module distribution for optimization high in the clouds unloading dynamic resource；Depth estimation module is used for mobile phone terminal or the image in high in the clouds Block estimation of Depth, generates depth map.

The invention have the benefit that by setting up high in the clouds unloading Dynamic Resource Allocation for Multimedia model, formed based on cloud computing Mobile phone terminal 2D turns self adaptation discharging method and the system of 3D, the calculating that mobile phone terminal is complicated is unloaded to high in the clouds, thus discharges mobile phone The storage resource of end, improves the processing speed of mobile phone, reduces the power consumption of mobile phone.Can be obtained by this depth estimation method and system Obtain estimation of Depth reasonable, the speed of service efficient high-quality depth map.

Embodiments of the invention also have following beneficial effect: detected with vanishing point by the calculating of HIS space pixel value, More accurately image block can be classified；Different classes of image block, by calculating with k-means algorithm, graph cut respectively Image block is detected by the detection of method, vanishing point, can improve the efficiency of algorithm；By consider the computing capability in mobile phone terminal and high in the clouds with And the transfer rate between mobile phone and high in the clouds, propose to minimize the Dynamic Resource Allocation for Multimedia model of the high in the clouds unloading of mobile phone terminal power consumption, The power consumption of mobile phone can be preferably minimized.

Accompanying drawing explanation

Fig. 1 is the method schematic diagram that embodiment of the present invention mobile phone terminal 2D turns the unloading of 3D self adaptation cloud.

Fig. 2 is the input picture of the embodiment of the present invention, and Fig. 2 a is Plain, and Fig. 2 b is high mountain, and Fig. 2 c is highway, Fig. 2 d For train, Fig. 2 e is sandy beach, and Fig. 2 f is butterfly.

Fig. 3 is the power consumption schematic diagram that the embodiment of the present invention is saved under heterogeneous networks wideband scenarios.

Fig. 4 is the depth map generated, and Fig. 4 a is Plain, and Fig. 4 b is high mountain, and Fig. 4 c is highway, and Fig. 4 d is train, figure 4e is sandy beach, and Fig. 4 f is butterfly.

Detailed description of the invention

Below in conjunction with detailed description of the invention and compare accompanying drawing the present invention is described in further detail, it should be emphasised that, That the description below is merely exemplary rather than in order to limit the scope of the present invention and application thereof.

The mobile phone terminal 2D of the present embodiment turns the method flow diagram of 3D self adaptation cloud unloading as shown in Figure 1.

A1. input a frame 2D monocular image, image is divided into N number of image block；

A2. each image block is converted into HSI space by rgb space, calculates the pixel value in HSI space, according to set Threshold classification goes out prospective view and non-prospective view；Non-prospective view is carried out vanishing point detection, utilizes Canny operator to calculate and publish picture The edge of picture, utilizes Hough to carry out edge line detection, according to the breakpoint detection vanishing point of straight line if vanishing point being detected, then For linear view, it it is otherwise general view；

A3. basis has divided the classification of view, calculates the algorithm complex of each image block estimation of Depth respectively |: for Prospective view, splits image block with k-means algorithm, the algorithm estimated by the execution time approximate depth of k-means algorithm Complexity；For general view, detecting image block with graph cut algorithm, the region that can detect is prospect, otherwise For background, respectively foreground and background is calculated the execution time of graph cut algorithm, with the execution time of graph cut algorithm Carry out the algorithm complex that approximate depth is estimated；For linear view, detect image with the segmentation of k-means algorithm and vanishing point simultaneously Block, calculates the execution time of its algorithm respectively, takes the algorithm complex that bigger execution time approximate depth is estimated.

A4. by the algorithm complex of calculated for A3 each image block, substitution high in the clouds unloading Dynamic Resource Allocation for Multimedia model:

Allocation result is obtained by minimizing the power consumption optimum of mobile phone terminal；

A5. the allocation result obtained according to A4 optimization, carries out estimation of Depth at mobile phone terminal or high in the clouds respectively, generates the degree of depth Figure.

Depth map mobile phone terminal and high in the clouds generated respectively carries out depth integration, and synthesis 3D viewpoint also shows 3D view.

The test platform data HP iPAQ PDA of this experiment, each data parameters is P_cFor 0.9W, P_iFor 0.3W, P_trFor 1.3W。

Originally test image is as in figure 2 it is shown, Fig. 2 a is Plain, and Fig. 2 b is high mountain, and Fig. 2 c is highway, and Fig. 2 d is fire Car, Fig. 2 e is sandy beach, and Fig. 2 f is butterfly.After being divided into 9 pieces, all kinds of view image block numbers that every width figure is corresponding are as shown in table 1, n_all1,n_all2And n_all3It is respectively prospective view, linear view and total number of general view.

The number of table 1 all kinds of view image block

When test network bandwidth is respectively 0.5Mbps, 1.5Mbps, 2.5Mbps, 3.5Mbps and 4.5Mbps, high in the clouds is unloaded The each data parameters carried and the energy of saving are as shown in table 2 below.The curve chart of table 2 correspondence is as shown in Figure 3.The degree of depth that experiment generates As shown in Figure 4, Fig. 4 a is Plain to figure, and Fig. 4 b is high mountain, and Fig. 4 c is highway, and Fig. 4 d is train, and Fig. 4 e is sandy beach, and Fig. 4 f is Butterfly.

Table 2 high in the clouds unloading Dynamic Resource Allocation for Multimedia result and the energy of saving

In table 2, n_c1,n_c2And n_c3Number to the different masses should being unloaded on high in the clouds under transmission bandwidth, saved respectively Energy is compared with when not having cloud to unload, the power consumption percentage ratio saved when system has cloud to unload.

Claims (10)

1. the method that a mobile phone terminal 2D turns the unloading of 3D self adaptation cloud, it is characterised in that comprise the steps:

A1. input a frame 2D monocular image, image is divided into N number of image block；

A2. image block is classified, be categorized as prospective view, linear view and general view；

A3. basis has divided the classification of view, calculates the algorithm complex of each image block estimation of Depth respectively；

A4. by the algorithm complex of calculated for A3 each image block, high in the clouds unloading Dynamic Resource Allocation for Multimedia model is substituted into, optimum Change and obtain allocation result；

A5. the allocation result obtained according to A4 optimization, carries out estimation of Depth at mobile phone terminal and high in the clouds respectively, generates depth map.

2. the method for claim 1, it is characterised in that described step A2 image block classification comprises the steps:

A201. each image block is converted into HSI space by rgb space, calculates the pixel value in HSI space, according to the threshold set Value sorts out prospective view and non-prospective view；

A202. non-prospective view being carried out vanishing point detection, if vanishing point being detected, being then linear view, otherwise for commonly to regard Figure.

3. method as claimed in claim 2, it is characterised in that described step A201 prospective view and the classification of non-prospective view Method is: calculate image coordinate (x, y) place HSI spatial pixel values H (x, y), S (x, y), I (x, y), if 100 < H (x, y) < 180 and 100 < S (x, y) < 255, then Sky (x, y)=1；If 50 < H (x, y) < 100 and 100 < S (and x, y) < 255, then Ground (x, Y)=1；IfIf Amount is more than the threshold value set, then it is prospective view, is otherwise Non-prospective view.

4. method as claimed in claim 2, it is characterised in that described step A202 vanishing point is detected as: utilize Canny operator meter Calculate the edge of image, utilize Hough to carry out edge line detection, according to the breakpoint detection vanishing point of straight line.

5. method as claimed in claim 4, it is characterised in that the detection formula of vanishing point detection is:

$\underset{x_0,y_0}{\min }\underset{i=1}{\overset{n}{\&Sigma;}}{W}_i{({\mathrm{\&rho;}}_i-x_0{\mathrm{cos\&theta;}}_i-y_0{\mathrm{sin\&theta;}}_i)}^2;$

Wherein, (x₀,y₀) it is the image block vanishing point in image plane, (ρ_i,θ_i) it is the image block point (x that corresponds to image plane_i,y_i) Polar coordinate, W_iFor corresponding weight.

6. the method for claim 1, it is characterised in that calculate algorithm complex in described step A3,

Comprise the steps:

A301. for prospective view, split image block with k-means algorithm, approximate deeply with the execution time of k-means algorithm The algorithm complex that degree is estimated；

A302. for general view, detecting image block with graph cut algorithm, the region that can detect is prospect, no It is then background, respectively foreground and background is calculated the execution time of graph cut algorithm, during with the execution of graph cut algorithm Between come approximate depth estimate algorithm complex；

A303. for linear view, detect image block with the segmentation of k-means algorithm and vanishing point simultaneously, calculate its algorithm respectively The execution time, take the algorithm complex that bigger execution time approximate depth is estimated.

7. method as claimed in claim 6, it is characterised in that the algorithm complex of described prospective view image block be expressed as Under formula: C₁≈(8+12α_i) WH/S, the formula that the algorithm complex of general view image block is expressed as: C₃≈β_iWH/ S, the formula that the algorithm complex of linear view image block is expressed as: C₂≈max(γ,8+12α_i)WH/S,

Wherein,β_iIterations for graph cut algorithm；

C₁、C₂、C₃It is respectively prospective view, linear view, the algorithm complex of general view image block,

W and H is respectively width and the height of current image block, and S is the area for normalized image block, S=87296, α_iFor k- The number of clusters divided in means algorithm, O_iFor the number of contours of Guan Bi in image, γ is that the image to S size carries out vanishing point detection Time.

8. the method for claim 1, it is characterised in that described step A4 high in the clouds unloading Dynamic Resource Allocation for Multimedia model passes through Minimizing the power consumption of mobile phone terminal, optimization obtains allocation result.

9. method as claimed in claim 8, it is characterised in that the expression formula of described high in the clouds unloading Dynamic Resource Allocation for Multimedia model For:

$n_c=\mathrm{argmin}(P_c\&times;\frac{C_m}{f}+P_i\&times;\frac{C_c}{s}+P_{tr}\&times;\frac{D}{B}),;$

Wherein, C_m=(n_m1,n_m2,n_m3)×(C₁,C₂,C₃)^T,C_c=(n_c1,n_c2,n_c3)×(C₁,C₂,C₃)^T,

P_cFor the mobile phone power consumption when calculating, P_iPower consumption when leaving unused for mobile phone, P_trFor the mobile phone power consumption when data transmit；C_m For being assigned to the algorithm complex of mobile phone terminal, C_cFor being assigned to the algorithm complex in high in the clouds；F is the processing speed of mobile phone terminal, and S is The calculating speed in high in the clouds, D is the size of the data of high in the clouds and mobile phone terminal transmission, and B is bandwidth；n_c1,n_c2And n_c3Respectively it is unloaded to The number of the prospective view in high in the clouds, linear view and general view；n_all1,n_all2And n_all3It is respectively prospective view, linear view Total number with general view.

10. a mobile phone terminal 2D turns the system that 3D self adaptation cloud unloads, it is characterised in that system includes image division module, figure As block sort module, complicated dynamic behaviour module, Dynamic Resource Allocation for Multimedia model module, depth estimation module；Image division module is used In the division by 2D monocular image；Image block classification, for the classification of image block, is prospective view, line by image block classification module Property view and general view；Complicated dynamic behaviour module is for calculating the algorithm complex of each image block；Dynamic Resource Allocation for Multimedia mould Pattern block is for the distribution of optimization high in the clouds unloading dynamic resource；The depth estimation module image block for mobile phone terminal or high in the clouds is deep Degree is estimated.