CN103248890A - Source data acquisition and compression method for wireless projection - Google Patents

Abstract

The invention discloses a source data acquisition and compression method for wireless projection, which belongs to the field of wireless projection. The method can quickly and efficiently capture and compress screen images, transmits the compressed screen images to a service terminal for display via a wireless network, and finally achieves the function of wireless projection. The scheme adopts screenshot based on mirror image driving and an optimization algorithm based on H.264. Compared with the traditional screenshot GDI and DirectX, the method has the advantages of low occupancy of CPU, high quality of images, high efficiency of screenshot and the like. Aiming to the characteristic of the large interframe relativity of the screen images, the video compression algorithm based on the H.264 is improved, and on the basis, layered codes based on ROI (Region of Interest) is introduced, and as a result, the data transmission capacity is lower and the quality is higher.

Description

A kind of information source data acquisition and compression method for wireless display

Technical field

The present invention relates to a kind of implementation method of wireless display technology, in particular, the present invention relates generally to a kind of information source data acquisition and compression method for wireless display.

Background technology

In optical projection system, information source is exactly screen picture.The acquisition technique of on-screen data mainly contains GDI screenshotss and DirectX screenshotss.GDI screenshotss mechanism be with " desktop also is a window, and desktop also has a window handle " this simple general knowledge for the basis, if we have obtained the device context (DC) of desktop, just can utilize the content that copies it in the DC of our establishment.The handle that obtains window can be realized with function G etDeskWindow () function.In like manner DC also can obtain by handle HWnd.Concrete implementation method is: 1, obtain the desktop handle by GetDesktopWindow () function; 2, obtain desktop DC with GetDC (); 3, bitmap and the DC of establishment and screen DC compatibility, and this bitmap is selected among the DC of establishment; 4, when you are ready to grab when screen, to compatible DC, finishing and grabbing the compatible bitmap in screen back is exactly the screen content of grabbing screen with regard to the content of copy window DC; 5, discharge the object of creating at last.DirectX is the content that catches current screen by visit foreground buffering.Because the content that the foreground buffering has been preserved current screen.Each DirectX program comprises a region of memory that is known as buffering, has wherein preserved the video memory content relevant with this program.DirectX provides an interface function GetFrontBuffer () method, and it receives the IDirect3DSurface pointer to object and does parameter, and copies the foreground buffer content to Surface.IDirect3DSurface obtains with CreateImageSurface ().We just can directly preserve into picture the file of bmp form by calling D3DXSaveSurfaceToFile () method then.

And the defective of above-mentioned screenshotss scheme is bigger.By the GDI screenshotss, this method major defect is that the efficient of screenshotss is not high, and the CPU occupation rate is big, and data volume is big etc.In addition, owing to utilize the words of this mode screenshotss can catch information less than video pictures, this is to come video playback rapidly owing to tend to call DirectDraw during video playback.And the screenshotss efficient of DirectX is compared the GDI screenshotss and is wanted high, and this optimizes mechanism by the DirectX bottom and determines.Generally speaking, one big little be the bitmap file of 600*400, adopt 32 looks, if adopt the GDI sectional drawing approximately to need time about 110ms, adopt the DirectX sectional drawing then to need to see Table (1) about 80-100ms.Obviously top two kinds of methods are not enough for the exigent shadow casting technique of real-time.

The compression of view data is modal to be the jpeg2000 algorithm.It is that (Joint Photographic Experts Group JPEG) just is devoted to develop new still image compression standard beginning in 1997, and became new international standard, i.e. ISO/IEC15444-1 in 2000 JPEG (joint photographic experts group).This algorithm does not have to adopt the compression algorithm based on dct transform, but adopts the advanced compress technique based on wavelet transform.The embedded coding that blocks optimized in being based on that the main algorithm of the entropy of jpeg2000 coding adopts in addition

(EBCOT), the jpeg standard of comparing, picture compression efficiency has on average improved 30%.The cataloged procedure of Jpeg2000 may be summarized to be: 1, image is carried out chroma space; 2, image being cut into rectangle and connecing piece, is that unit carries out wavelet transformation with the splicing block in each component; 3, the wavelet coefficient that conversion is obtained quantizes and divides, and the final unit of division is code block; 4, the coefficient in the code-aiming block carries out bit plane scanning, implements simultaneously based on contextual entropy coding; 5, support code stream tissue and the form encapsulation of gradable decoding, insert necessary error correction information.But the defective of j peg2000 is this algorithm only to a width of cloth picture compression, and in the process of projection, needing constantly to intercept the client screen image compresses, and the former frame of a two field picture is likely similar with back one frame, and Here it is, and frame-to-frame correlation is strong.If adopt the jpeg2000 algorithm, it is not high that each two field picture is compressed obvious efficient.This method is utilized the strong characteristics of screen picture frame-to-frame correlation, introduces and H.264 optimizes algorithm, reduces the data volume after compressing.

Summary of the invention

One of purpose of the present invention is at above-mentioned deficiency, a kind of information source data acquisition and compression method for wireless display is provided, how to solve in the prior art the more efficient real-time screen picture data of obtaining with expectation, and how under the condition that does not influence picture quality, carry out in the frame to screen picture and the interframe compression, make technical problems such as volume of transmitted data reduces greatly.

For solving above-mentioned technical problem, the present invention by the following technical solutions:

A kind of information source data acquisition and compression method for wireless display provided by the present invention, described method comprises the steps:

Steps A, FTP client FTP send drawing command, and video card hardware driving and mirror image drive and receive this order simultaneously; Wherein the video card hardware driving is made mapping operation after with command execution, and shows at display; Mirror image drives and make mapping operation in the unit region of memory, and revises the data in this unit region of memory;

Step B, FTP client FTP save as bitmap format according to the unit interval with the view data of buffer memory in the unit region of memory in the steps A, namely obtain the consecutive image frame data of screen picture;

Step C, the successive image frame transfer of data that obtains among the step B to encoder is H.264 carried out compressed encoding, and then formation video flowing, wireless aps by ICP/IP protocol and service end is sent to service end and carries out decode operation again, shows in service end at last.

As preferably, further technical scheme is: it is operand that described mirror image drives with the unit region of memory, and draws the image identical with the video card hardware driving simultaneously in the unit region of memory.

Further technical scheme is: among the described step C H.264 encoder the step that the consecutive image frame data carry out compressed encoding is comprised

Estimation utilizes reference frame to estimate the moving displacement amount of current encoded frame;

Motion compensation calculates image after the estimation and the difference of original image, then motion-compensated values is sent to decoding end;

With estimation and compensation result coding, adopt BMA, utilize mean-square error criteria to weigh the piece coupling; Its formula is

$\mathrm{MSE}(x,y)=\frac{1}{\mathrm{MN}}\underset{m=1}{\overset{M}{\mathrm{\Σ}}}\underset{n=1}{\overset{N}{\mathrm{\Σ}}}[f_k(m,n)-f_{k-1}(m+x,n+y){]}^2$

In the following formula, f _k(m n) is the brightness of present frame, f _k-1 (m+x n+y) is the brightness of reference frame, the relevance function of two pieces be R (x, y)=1/MSE (x, y).

Further technical scheme is again: also comprise the ROI hierarchical coding among the described step C in the compressed encoding step, the step of described ROI hierarchical coding comprises

Calculate the ROI zone, and carry out the sheet group and divide, thereby obtain the ROI model;

Revise the H.264 variable-length encoding part of encoder, can be based on realizing on the flexible macro-block order characteristic encoding characteristics that ROI preferentially strengthens coding, remaining non-area-of-interest NROI then handles by basic coding;

Outputting video streams, this video flowing are the video after area-of-interest strengthens the coding effect.

Compared with prior art, one of beneficial effect of the present invention is: the characteristics that taken full advantage of the screen picture frame-to-frame correlation, mainly optimized the interframe compression algorithm, the image that collects is carried out compressing in interframe and the frame, and introduce area-of-interest (ROI) hierarchical coding based on H.264, rather than as jpeg2000 to doing compression in the frame.So just significantly reduce data quantity transmitted, and can not influence whole video quality, made projected picture become more smooth.A kind of information source data acquisition and compression method for wireless display provided by the present invention is simple in structure simultaneously, can be used as collection, compression and the transmission of doing image information source data in all kinds of wireless display equipment, and range of application is wide.

Description of drawings

The installation sectional drawing that Fig. 1 drives for the mirror image that is used for one embodiment of the invention;

Fig. 2 is for being used for the realization flow block diagram of explanation one embodiment of the invention.

Embodiment

The present invention is further elaborated below in conjunction with accompanying drawing.

One embodiment of the present of invention are a kind of information source data acquisition and compression methods for wireless display, and described method comprises the steps:

Steps A, FTP client FTP send drawing command, and video card hardware driving and mirror image drive and receive this order simultaneously; Wherein the video card hardware driving is made mapping operation after with command execution, and shows at display; Mirror image drives and make mapping operation in the unit region of memory, and revises the data in this unit region of memory;

Step B, FTP client FTP save as bitmap format according to the unit interval with the view data of buffer memory in the unit region of memory in the steps A, namely obtain the consecutive image frame data of screen picture;

Step C, the successive image frame transfer of data that obtains among the step B to encoder is H.264 carried out compressed encoding, and then formation video flowing, wireless aps by ICP/IP protocol and service end is sent to service end and carries out decode operation again, shows in service end at last.

With reference to shown in Figure 1, according to another embodiment of the present invention, this is for installing the sectional drawing after mirror image drives.The mirror image here drives title and is called MirageDriver, can view in equipment manager.Intel (R) HD Graphics then is that the core video card of CPU inside drives, and it is responsible for showing output, has hardware device.And the mirror image driving does not have actual physical equipment, and it saves as operand in one, and all operations that video card drives is all carried out one time.When the user when screen is drawn a line, mirror image drives also the correspondence position in that piece internal memory and draws a line, if the data in this piece internal memory are saved as bitmap, just obtains with the same image of current screen.

With reference to shown in Figure 2, the figure shows realization block diagram of the present invention.When FTP client FTP sends drawing command, video card hardware driving and mirror image drive and receive order simultaneously, the video card hardware driving then will be made mapping operation after the command execution, show at display, mirror image then is to do mapping operation in a memory field after driving the reception order, revises the data in this piece internal memory, and this moment, we only needed timer of setting, just the data in this piece internal memory are preserved into bitmap format at set intervals, the image that obtains so is exactly screen picture.Then we deliver to the H.264 encoder of optimizing to this a succession of image frame data and encode, and the video flowing of the coding formation afterwards wireless aps by ICP/IP protocol and service end again is sent to service end and carries out decode operation, carries out display operation at last.

H.264 be joint video team (JoiniVideoTeam, JVT) the common up-to-date international video encoding standard of being set up by ITU-T and ISO/IEC cooperation of releasing.H.264 how codec and encoding and decoding of indefinite regulation to realize, but stipulated the sentence structure of the video bit stream behind the coding and the coding/decoding method of this bit stream.Being implemented under this framework of each encoder should be able to intercommunication, has bigger flexibility in realization, and is conducive to vie each other.The present invention does optimization on original H.264 encryption algorithm basis, is mainly reflected in the interframe redundancy optimization and handles.

The interframe redundancy of screen picture is stronger, and the step of its interframe compressed encoding is: 1, estimation, so-called estimation are to refer to utilize reference frame to estimate the moving displacement amount of current encoded frame; 2, motion compensation refers to calculate image after the estimation and the difference of original image, then motion-compensated values is sent to decoding end; 3, with estimation and compensation result coding.Realize estimation and compensation employing BMA usually, can utilize mean-square error criteria to weigh the piece coupling.Its formula is

$\mathrm{MSE}(x,y)=\frac{1}{\mathrm{MN}}\underset{m=1}{\overset{M}{\mathrm{\Σ}}}\underset{n=1}{\overset{N}{\mathrm{\Σ}}}[f_k(m,n)-f_{k-1}(m+x,n+y){]}^2$

F wherein _k(m, the n) brightness of expression present frame, f _K-1(m+x n+y) then represents the brightness of reference frame.So the relevance function of these two pieces be R (x, y)=1/MSE (x, y).If R (x, y) more big, represent that then these two piece matchings are more strong.

H.264 will need the image encoded frame to be divided into 7 kinds of pieces, be respectively seven kinds of 16*16,16*8,8*16,8*8,8*4,4*8,4*4.So just can describe with maximum 16 vectors, estimation that like this can more accurate record block, but seem bigger with the words data volume of 16 vectors descriptions.Consider that the screen picture frame-to-frame correlation is strong, the present invention adopts 16*16,16*8, four kinds of block methods of 8*16,8*8, needs 4 vectors to describe at most, has significantly reduced data volume and computing time.The consequence of bringing like this be before the coded frame estimation is not optimized accurately, but the particularity of combining screen image can not influence people's visual experience substantially.H.264 support the multi-reference frame coding in addition, for example the B two field picture can use the forward frame of reconstruction and back to predict that to frame we reduce the number of multi-reference frame here, only uses front and back totally two two field pictures, because the picture frame correlation of screen is strong, be easy to match suitable piece.Complexity and the memory space of codec have been reduced so greatly.

The present invention has introduced encoding region of interest (ROI) technology on hierarchical coding.Area-of-interest refers to people to the interested part of piece image.Utilize the ROI compress technique can strengthen the picture quality of area-of-interest, reduce the picture quality of non-area-of-interest.Like this can be so that image data amount reduces and does not influence the observation quality of integral body.Because H.264 supporting flexible macro-block order characteristic, the present invention to utilize a kind of algorithm to calculate area-of-interest is assigned to it in sheet group then and encodes.The movable information of supposing macro block i is MV _i=| x _i|+| y _i|, then its mean motion information is

If average complexity is

Wherein

Standard deviation for macro block i.Motion and complexity information can be represented with variable WT so, and

α wherein ₁And α ₂Represented weight coefficient respectively, and both sums are 1.Here defined a threshold value beta, when the value of WT during greater than β, then this macro block is judged to area-of-interest (ROI), otherwise is judged to non-area-of-interest.And then ROI and NROI be assigned to respectively in the different Slice group.Model selection prospect and the background mode of group so just are easy to image is divided into foreground area and background area, carry out hierarchical coding at last.Its roughly flow process be: 1, calculate the ROI zone, and carry out the sheet group and divide, obtain the ROI model; 2, revise the variable-length encoding part of encoder, can be based on realizing on the flexible macro-block order characteristic encoding characteristics that ROI preferentially strengthens coding, remaining non-area-of-interest NROI then handles by basic coding.3, output video, this video flowing are exactly the video after area-of-interest strengthens the coding effect.This method makes the compression ratio of image increase, and also makes the subjective quality of image be improved simultaneously.

The present invention does optimization from both direction to collection and the compression of information source data.At first, adopt the screenshotss that utilize mirror image to drive in the technical the present invention of screenshotss, made screenshotss efficient improve greatly.The words that the screen picture of general 1024*768 adopts mirror image to drive screenshotss only need about 40-60ms, and as shown in table 1 below, compare traditional GDI and DirectX screenshotss have improved efficient.The shortcoming that can not intercept in progress video image in addition during the GDI screenshotss also can drive to be resolved by mirror image.Secondly, after collecting the information source data, the present invention does not use the image compression algorithm of traditional jpeg2000, but adopt optimized based on H.264 video compression algorithm.This algorithm has taken full advantage of the characteristics of screen picture frame-to-frame correlation, mainly optimized the interframe compression algorithm, the image that collects is carried out compressing in interframe and the frame, and introduces area-of-interest (ROI) hierarchical coding based on H.264, rather than as jpeg2000 to doing compression in the frame.So just significantly reduced data quantity transmitted, and can not influence whole video quality, this will make projected picture become more smooth.

Table 1

Screenshotss efficient	The GDI screenshotss	The DirectX screenshotss	Mirror image drives screenshotss
				Time (ms)	110	80-100	40-60

Except above-mentioned, also need to prove " embodiment ", " another embodiment " that speak of in this manual, " embodiment " etc., refer to concrete feature, structure or the characteristics described in conjunction with this embodiment and be included among at least one embodiment that the application's generality describes.A plurality of local appearance statement of the same race is not necessarily to refer to same embodiment in specification.Furthermore, when describing a concrete feature, structure or characteristics in conjunction with arbitrary embodiment, what advocate is to realize that in conjunction with other embodiment this feature, structure or characteristics also fall within the scope of the invention.

Although invention has been described with reference to a plurality of explanatory embodiment of the present invention here, but, should be appreciated that those skilled in the art can design a lot of other modification and execution modes, these are revised and execution mode will drop within the disclosed principle scope and spirit of the application.More particularly, in the scope of, accompanying drawing open in the application and claim, can carry out multiple modification and improvement to building block and/or the layout of subject combination layout.Except modification that building block and/or layout are carried out with improving, to those skilled in the art, other purposes also will be tangible.

Claims (4)

1. information source data acquisition and a compression method that is used for wireless display is characterized in that: described method

Comprise the steps:

Steps A, FTP client FTP send drawing command, and video card hardware driving and mirror image drive and receive this order simultaneously; Wherein the video card hardware driving is made mapping operation after with command execution, and shows at display; Mirror image drives and make mapping operation in the unit region of memory, and revises the data in this unit region of memory;

Step B, FTP client FTP save as bitmap format according to the unit interval with the view data of buffer memory in the unit region of memory in the steps A, namely obtain the consecutive image frame data of screen picture;

Step C, the successive image frame transfer of data that obtains among the step B to encoder is H.264 carried out compressed encoding, and then formation video flowing, wireless aps by ICP/IP protocol and service end is sent to service end and carries out decode operation again, shows in service end at last.

2. information source data acquisition and compression method for wireless display according to claim 1 is characterized in that: it is operand that described mirror image drives with the unit region of memory, and draws the image identical with the video card hardware driving simultaneously in the unit region of memory.

3. information source data acquisition and compression method for wireless display according to claim 1 is characterized in that: among the described step C H.264 encoder the step that the consecutive image frame data carry out compressed encoding is comprised

Estimation utilizes reference frame to estimate the moving displacement amount of current encoded frame;

Motion compensation calculates image after the estimation and the difference of original image, then motion-compensated values is sent to decoding end;

With estimation and compensation result coding, adopt BMA, utilize mean-square error criteria to weigh the piece coupling; Its formula is

$\mathrm{MSE}(x,y)=\frac{1}{\mathrm{MN}}\underset{m=1}{\overset{M}{\mathrm{\Σ}}}\underset{n=1}{\overset{N}{\mathrm{\Σ}}}[f_k(m,n)-f_{k-1}(m+x,n+y){]}^2$

In the following formula, f _k(m n) is the brightness of present frame, f _K-1(m+x n+y) is the brightness of reference frame, the relevance function of two pieces be R (x, y)=1/MSE (x, y).

4. according to claim 1 or 3 described information source data acquisition and compression methods for wireless display, it is characterized in that: also comprise the ROI hierarchical coding among the described step C in the compressed encoding step, the step of described ROI hierarchical coding comprises

Calculate the ROI zone, and carry out the sheet group and divide, thereby obtain the ROI model;

Revise the H.264 variable-length encoding part of encoder, can be based on realizing on the flexible macro-block order characteristic encoding characteristics that ROI preferentially strengthens coding, remaining non-area-of-interest NROI then handles by basic coding;

Outputting video streams, this video flowing are the video after area-of-interest strengthens the coding effect.

Images