CN104349142A - Layered representation-based unmanned plane video adaptive transmission method - Google Patents
Layered representation-based unmanned plane video adaptive transmission method Download PDFInfo
- Publication number
- CN104349142A CN104349142A CN201410608007.1A CN201410608007A CN104349142A CN 104349142 A CN104349142 A CN 104349142A CN 201410608007 A CN201410608007 A CN 201410608007A CN 104349142 A CN104349142 A CN 104349142A
- Authority
- CN
- China
- Prior art keywords
- layer
- image
- video
- compression
- background
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention discloses a layered representation-based unmanned plane video adaptive transmission method. A device for realizing the method comprises a layered structure for video representation, wherein the layered structure sequentially comprises a background layer, an object layer and an enhancement layer from bottom to top. According to different requirements, a corresponding layered structure is selected to transmit a video, and the problem that the movement of an independent moving object in the video is vague and cannot be recognized after the independent moving object is compressed is solved by dividing the moving object. A clear panoramic picture can be provided, or a video sequence with relatively high quality can be provided. The output precision is improved, the compression time is greatly shortened, and the working efficiency is improved to a great extent.
Description
Technical field
The present invention relates to screen transmission method, be specifically related to a kind of UAV Video adaptive transmission method of expressing based on layering.
Background technology
Unmanned plane (Unmanned Aerial Vehicles, UAV) be the focus that in the world today, military issue weapons develops, and will likely become the leading force of 21 century air fighting, be widely used in [1] such as battle reconnaissance, supervision, border patrol, electronic reconnaissance, target localization, recognition and trackings.In addition, unmanned plane also can be used for the exploration mapping, highway tour, place monitoring, floods supervision etc. of civilian aspect.Just at present, the effect of most of unmanned plane plays premised on Airborne Surveillance, Airborne Control ability, and this will depend on video, the collection of image, compression and transmission [2] [3] [4].Through the application & development of decades, the image delivering system of unmanned plane relative maturity and stable, but video becomes according to amount the bottleneck that unmanned plane information transmits because of its googol.From basic comprising, UAV Video transmission system and civilian Video transmission system basically identical, be all made up of (as Fig. 1) video acquisition module, video processing module, video compressing module, video transmission module, video decompression module, analysis module, video display module.
However, still the technology of relative maturity in civilian Video transmission system cannot be directly applied to UAV Video transmission system, to be both all exist marked difference to main cause in the environment of the performance of hardware device, the restriction of transmission channel and practical application etc., different to the requirement of indices when causing system, make a concrete analysis of as follows:
1, the requirement of compression performance: the flying height of unmanned plane is not from hundreds of rice to a few km etc., namely shooting time subject and video camera distant, the scouting region that single-frame images captures is wider, therefore, in order to the situation in region is scouted in description detailed as much as possible, unmanned plane aerial photography video needs to have higher resolution.The raising of resolution not only represents and reduces the holding capacity of compression artefacts, more represents the increase of data volume.But meanwhile, the transmission channel of unmanned plane is the narrower wireless channel of bandwidth, and this just means, UAV Video transmission system is more harsh to the requirement of video compression performance, should ensure that low distortion rate reaches high compression ratio again.
2, the requirement of computation complexity: on the one hand, because the payload of unmanned plane is limited, the hardware device entrained by it has strict requirement on volume and weight, and this is directly presented as the restriction to data storage capacities and disposal ability; On the other hand, in the application such as Long-distance Control, pay special attention to the ageing of video information, this is not only and claims to video transmission rate, claims especially to video processing speed.Comprehensive above two aspect factors are known, and UAV Video transmission system requires fast and simply processes video, namely requires low computation complexity.
3, the requirement of reliability: in Unmanned Aerial Vehicle Data transmitting procedure, the impact that not only can be subject to the factors such as landform, atural object and air because of wireless signal causes the fluctuation of bandwidth, also may be subject to external attack and interception, these all can cause damage or the loss of video data.Therefore, UAV Video transmission system also must have the design strengthening reliability and fault-tolerance.
4, the requirement of flexibility: except some requirement above, UAV Video transmission system also should have the ability of the various emergency situations of reply flexibly, as adjusted data transmit-receive speed to adapt to the fluctuation of bandwidth in good time; The consumption of storage resources is dealt with in rational minimizing data backup; The selection of intelligence sends data and carrys out response terminal request etc.
In sum, the requirement of UAV Video transmission system requires more more comprehensively high than civilian Video transmission system, not only will take into account compression performance and computation complexity, also will have higher reliability and flexibility.
At present, the UAV Video transmission system of extensive use mainly utilizes measured Video Codec to carry out compressed video, and the most frequently used has M-JPEG and MPEG-2 [5].Wherein, the performance of MPEG-2 [6] is comparatively superior, and it have employed the key technologies such as motion compensation, DCT and quantification, and its compression ratio can reach more than one percent, and has better error resilience performance when there is data-bag lost.However, MPEG-2 still can not meet the rigors of UAV Video transmission system completely.Citing, a frame per second is that 25 frames are per second, resolution is standard high definition (1920x1080), the data volume of the digital video of 8bit pixel is 622Mbit/s, and the actual bandwidth that unmanned plane can be used for video signal transmission generally only has a few Mbit/s, and MPEG-2 is difficult to while ensuring the quality of products, reach so high compression ratio.Just saw over the past two years, new video compression coding standard H.264/AVC [7] more and more obtains the accreditation of industrial quarters with the performance of its brilliance, also occupying certain status [8] [9] in unmanned plane research field progressively.But, H.264/AVC carry out video compression even if use, retain the resolution of video if want and do not cause visible distortion when compressing, often also the data volume of original 622Mbit/s can only be dropped to 8 ~ 15Mbit/s.
Except the encoder of standard, also Video processing and the compression algorithm of a large amount of nonstandardized techniques is had, they utilize the feature of UAV Video, realize higher compression efficiency [10] ~ [18] by utilizing the technology such as the detection of overall motion estimation, global motion compensation, area-of-interest, image mosaic.In addition, consider the task feature of unmanned plane, document [19] [20] give chapter and verse the feature of video to detect area-of-interest, and the importance in this region is fully taken into account when compressing, different processing methods is adopted to area-of-interest and regions of non-interest, this method while the quality ensureing area-of-interest, can reduce the video data volume greatly.But these methods are all one-sidely solve subproblem, but cannot in compression performance, complexity, obtain good combination property between reliability and flexibility.In order to obtain globally optimal solution when this many-side mutually restricts and affects, document [21] gives a conceptual solution, analyzes theoretically and how under resource constraint, to meet the many-sided demand of user.
Existing correlation technique
1：Video adaptation:Concepts,technologies,and open issues[21]
This article gives video self adaptation solution, analyzes theoretically and how under resource constraint, to meet the many-sided demand of user, thus in compression performance, complexity, obtain good combination property between reliability and flexibility.Shown in Fig. 2 is the conceptual frame of video Adaptable System, an i.e. given entity (Entity), just by determining self adaptation operating space (Adaptation Space), relation between resource space (Resource Space) and user satisfaction space (Utility Space) three, concrete self adaptation solution can be designed.In figure, in resource space, shade cube represents the resource constraint in current application environment, and under the prerequisite meeting this constraint, can there is multiple self adaptation solution, these schemes constitute the self adaptation operational set meeting this resource constraint jointly.And video self adaptation problem to be solved is: select operation with the greatest satisfiction by user in the self adaptation operation set meeting given resource constraint.
This article proposes the conceptual framework of video self adaptation, and for video Adaptive Transmission provides good resolving ideas, but based on the framework of such theoretical property, in embody rule environment, how to play good effect still needs more deep exploration and research.
2: sequence images of unmanned aerial vehicle compression method research [12]
This article, according to the movement characteristic of unmanned plane, is obtained the overlapping region between sequence image, movement sequence image is transformed into still image, then completes the compression of stitching image with class EBCOT algorithm.The experimental result display of compression, the method is in compression efficiency and be all better than the video compression standard such as H.264 running time, can meet the transmission demand of unmanned plane image preferably.
This article has only excavated the larger feature in UAV Video overlapping region, movement sequence image is transformed into still image compress, but have ignored in UAV Video and there is independently moving object, and in practical application these independently moving object may be the target should paid close attention in video just.Therefore, although this method can obtain good compression performance, in video content expression, there is shortcoming, on the one hand, the Subjective video quality loss after its compression is heavier; On the other hand, the moving object in video produces motion blur upon compression, cannot identification.
List of references:
[1] Wen Xianqiao, Li Ying. " application prospect from the development unmanned plane future war of American UAV ". modern defense technology .2003 in October
[2]Andreas Birk,Burkhard Wiggerich,Heiko Bülow,Max Pfingsthorn,
Schwertfeger“Safety,Security,and Rescue Missions with an Unmanned Aerial Vehicle(UAV)”Journal of Intelligent&Robotic Systems(28January 2011),pp.1-20
[3]G.Belloni,M.Feroli,Antonio Ficola,Stefano Pagnottelli,and Paolo Valigi,“A COTS-Based Mini Unmanned Aerial Vehicle(SR-H3)for Security,Environmental Monitoring and Surveillance Operations:Design and Test”,EUROS,Vol.44Springer (2008),p.73-82.
[4]Yu-chi Liu；Qiong-hai Dai；“Vision Aided Unmanned Aerial Vehicle Autonomy:An Overview”,2010 3rd International Congress on Image and Signal Processing(CISP),16-18Oct.2010,pp.417-421
[5]Schaphorst,R.A.；“An overview of video compression in the tactical battlefield”,Military Communications Conference IEEE,11-14Oct 1993,pp.843-847vol.3
[6]ITU-T,ISO/IEC.1994.“Generic coding of moving pictures and associated audio information—part 2:video”.ITU-T Rec.H.262|ISO/IEC 13818-2(MPEG-2 Video).
[7]ITU-T,ISO/IEC.2005.“Advanced video coding for generic audiovisual services”.ITU-T Rec.H.264|ISO/IEC IS 14496-10.
[8]Xiao lin Chen,Shan cong Zhang,and Jie Liu,“Design of UAV video compression system based on H.264encoding algorithm”,EMEITIEEE(2011),p.2619-2622.
[9]Bennett,B.,Dee,C.and Ngugen,M.H.；"Operational concepts of MPEG-4 H.264for tactical DoD applications"；MILCOM 2005,October 17--20 Atlantic City,NJ,
[10]M.Bhaskaranand and J.D.Gibson,“Low-complexity video encoding for UAV reconnaissance and surveillance,”in Military Communications Conference(MILCOM),2011,pp.1633–1638.
[11] Cui Maihui, Zhou Jianjun, Chen Chao. the Compression Transmission Technology of UAV Video Intelligence. telecom technology .2007 February
[12] field inscription on ancient bronze objects, Xie Qingpeng, Tan Yihua, Liu Jian. sequence images of unmanned aerial vehicle compression method is studied. Central China University of Science and Technology's journal, in December, 2005
[13]M.
and R.Thoma.“Image segmentation based on object oriented mapping parameter estimation”.Signal Processing,15(3):315–334,1988.Multidimensional Signal Processing.
[14]A.Krutz,A.Glantz,and T.Sikora.“Recent advances in video coding using static background models”.In Picture Coding Symposium(PCS),2010,pages 462–465,dec.2010.
[15]S.Yahyanejad,D.Wischounig-Strucl,M.Quaritsch,and B.Rinner.Incremental mosaicking of images from autonomous,small-scale uavs.In Advanced Video and Signal Based Surveillance(AVSS),2010 Seventh IEEE International Conference on,pages 329–336,29 2010-sept.1 2010.
[16]Z.Zhu and H.Tang.Content-based dynamic 3d mosaics.In Computer Vision and Pattern Recognition Workshop,2006.CVPRW’06.Conference on,page 169,june 2006.
[17]S.Yahyanejad,D.Wischounig-Strucl,M.Quaritsch,and B.Rinner.Incremental mosaicking of images from autonomous,small-scale uavs.In Advanced Video and Signal Based Surveillance(AVSS),2010Seventh IEEE International Conference on,pages 329–336,29 2010-sept.1 2010.
[18] Zhu Yunfang, Ye Xiuqing, Gu Weikang. the splicing technology of panorama drawing of video sequence. Journal of Image and Graphics, Vol.11, No.8, Aug., 2006
[19]H.Cheng and J.Wus.Adaptive region of interest estimation for aerial surveillance video.In Image Processing,2005.ICIP 2005.IEEE International Conference on,volume 3,pages III–860–3,sept.2005.
[20]N.Doulamis,A.Doulamis,D.Kalogeras,and S.Kollias.Low bit-rate coding of image sequences using adaptive regions of interest.Circuits and Systems for Video Technology,IEEE Transactions on,8(8):928–934,dec 1998.
[21]Chang SF,Vetro A.“Video adaptation:Concepts,technologies,and open issues”.Proc.IEEE,93:148-158.2005
Summary of the invention
Goal of the invention: in order to solve in prior art, the shortcoming that unmanned plane load is limited, Bandwidth-Constrained, application complexity, traditional Video processing and compression method all exist compression performance and computation complexity can not get both, the problem of its demand can not be met fully, the invention provides a kind of UAV Video adaptive transmission method of expressing based on layering, solve the deficiencies in the prior art.
Technical scheme: a kind of UAV Video adaptive transmission method of expressing based on layering, is characterized in that, the method comprises the hierarchy that video is expressed; Described hierarchy comprises background layer, destination layer and enhancement layer from the bottom to top successively;
Described background layer is: one being comprised all image mosaic in the image sets of multiple image becomes a width panorama Background, and the mode of recycling Image Coding is compressed; Be not divided into k sublayer according to compression effectiveness, be followed successively by 0 from the bottom to top, 1,2 ... k-1;
Described destination layer is: each two field picture only comprises the target area of each two field picture in original video, adopts independent inter-frame prediction method to compress; Be not divided into m sublayer according to compression effectiveness, be followed successively by 0 from the bottom to top, 1,2 ... m-1;
Described enhancement layer is: a frame complete image of the corresponding original video of each two field picture; Be not divided into n sublayer according to compression effectiveness, be followed successively by 0 from the bottom to top, 1,2 ... n-1;
The mode of production and transfer layer of structure selects one from following five kinds of combinations: background layer; Destination layer; Background layer and destination layer; Background layer and enhancement layer; Background layer, destination layer and enhancement layer;
Further, comprise splicing and the compression of background layer image, specifically comprise the following steps:
2.1), image sequence splicing: splice according to the order of sequence remaining N-1 two field picture in image sets, concatenation comprises Feature point correspondence, virtual borderlines and Images uniting;
2.1.1), Feature point correspondence: determine in image to be spliced and the point one to one of the characteristic point in benchmark image, utilize existing algorithm extract minutiae, recycling matching algorithm obtains corresponding relation; Or the point that directly given range search is the highest with the characteristic point similarity of benchmark image in image to be spliced;
2.1.2), virtual borderlines: the position relationship corresponding according to characteristic of correspondence point, utilizes existing model to calculate the coordinate transform of image relative datum image to be spliced; The transformation parameter of gained is compressed, stores and transmitted;
2.1.3), Images uniting: according to step 2.3.2) coordinate conversion relation, by the evolution of each pixel in image to be spliced to benchmark image coordinate system, and its pixel value is copied to correspondence position, if overlapping with the pixel of benchmark image, then get the average of two two field picture correspondence position pixels or choose one of them value.
2.2), image compression: adopt conventional images compression algorithm to carry out the compression of quality scalability to splicing result, form k background layer sublayer; Wherein, background layer 0 is the compressed image of minimum quality, and background layer 1 ~ background layer k-1 compresses the classification refinement of the compression residual error of background layer 0; The quality of background layer is improved by the superposition of multiple background layer sublayer; The setting of level and the quality of each level are arranged according to the change of the network bandwidth.
Further, the way selection background layer of production and transfer layer of structure, an image is N frame, specifically comprises the following steps:
3.1), benchmark image is chosen: get any frame in image sets, using the coordinate system of benchmark image as the coordinate system of stitching image;
3.2), benchmark image feature point extraction: utilize existing algorithm extract minutiae;
3.3), image sequence splicing: carry out step 2.1);
3.4), image compression: carry out step 2.2).By the selection to different layers, effectively reduce calculation step, high-precision picture quality can be reached simultaneously.
Further, when there is self-movement target clearly in screen captured by unmanned plane, the way selection background layer of production and transfer layer of structure and destination layer; Specifically comprise the following steps:
4.1) moving Object Segmentation: by analyzing whole shooting picture, distinguishes global motion and local motion, extracts self-movement target further, and records moving target exact position in the corresponding frame;
4.2) background layer generates: removed from image by moving target, only retain background information, carry out step 2.1) and step 2.2) completing splicing and the compression of background layer, generation has the background layer of different quality sublayer.
4.3) destination layer generates: to the independent compressed encoding of moving target, adopt existing algorithm or set up motion model separately to target, the motion estimation module after modeling in estimation model parameter alternate algorithm;
Destination layer is carried out to the compression of quality scalability, destination layer 0 is the compressed image of minimum quality, destination layer 1 ~ destination layer m-1 compresses the classification refinement of the compression residual error of destination layer 0, improved the quality of destination layer by superposition, the setting of level and the quality of each level are arranged according to the change of the network bandwidth.By the differentiation to self-movement target, improve the present situation to self-movement target video integrality deficiency in prior art.
Further, the way selection background layer of production and transfer layer of structure and enhancement layer, or when selection background layer, destination layer and enhancement layer; Specifically comprise the following steps:
5.1) design video expression structure according to the actual requirements, select generation background layer/background layer and destination layer according to structure composition;
5.2) enhancement layer generates: utilize background layer and/or destination layer to predict, or adopt the interframe of this layer or infra-frame prediction to carry out absolute coding; Enhancement layer is carried out to the compression of quality scalability: enhancement layer 0 is the compressed image of minimum quality, enhancement layer 1 ~ enhancement layer n-1 compresses the classification refinement of the compression residual error of enhancement layer 0, is improved the quality of enhancement layer by superposition; The setting of level and the quality of each level can be arranged according to the change of the network bandwidth.
Further, between each sublayer of the same layer of hierarchy, redundancy is reduced by inter-layer prediction.
Further, redundancy is reduced by inter-layer prediction between the different layers of hierarchy.
Beneficial effect: according to the characteristic sum demand of UAV Video, devise a kind of video expression-form of novel hierarchy, namely expressed by hierarchy and image is operated, this expression, compared with original video or conventional compression code stream, has higher compression efficiency, more flexibly organizational form and operation-interface more easily.On the one hand, based on the video compression of image mosaic and Target Segmentation, can compression time be reduced greatly and ensure compression quality, greatly enhance operating efficiency; On the other hand, by the transmission structure that adaptive selection is different, reasonably transmitting according to practical application restriction of dynamic, avoids loss of data and congested.
Accompanying drawing explanation
Fig. 1 is Video transmission system flow chart
Fig. 2 is video self adaptation conceptual frame
Fig. 3 is UAV Video Adaptive Transmission system flow chart
Fig. 4 is that the video of hierarchy expresses schematic diagram
Fig. 5 is unmanned plane emulation sectional drawing
Fig. 6 is video-splicing panorama sketch of taking photo by plane
Fig. 7 is that the compression performance of simulation video compares
Fig. 8 is actual video-splicing design sketch of taking photo by plane
Fig. 9 is that the compression performance of actual video of taking photo by plane compares
Figure 10 is the figure that takes photo by plane that there is self-movement target
Figure 11 is background and moving Object Segmentation effect schematic diagram
Figure 12 generates based on the background layer of image mosaic
The video expression structure of Figure 13 background layer+destination layer
The video expression structure of Figure 14 background layer+destination layer+enhancement layer
Embodiment
Below in conjunction with accompanying drawing the present invention done and further explain.
In view of the load of unmanned plane is limited, the feature such as Bandwidth-Constrained, application be complicated, all there is the shortcoming that compression performance and computation complexity can not get both in traditional Video processing and compression method, can not meet its demand fully.The present invention solves the problem existing for UAV Video transmission for employing video adaptive technique, the video of hierarchy is adopted to express the content and the feature that describe video, and on this basis different compression methods is adopted to the video area of different content, thus under the prerequisite of the constraints and user's request that meet UAV Video transmission, realize efficiently, fast, reliably, transmission of video flexibly.
First technical scheme of the present invention proposes a kind of video Adaptive Transmission system for unmanned plane, the flow chart of this system as shown in Figure 3, the difference of itself and traditional UAV Video transmission system is mainly four modules of bold box in figure, is respectively video and expresses generation module, video self adaptation operational module, resource constraint module and user satisfaction module.In practical operation, with traditional UAV Video transmission method unlike, not directly to the compressed encoding that UAV Video carries out, but utilize a kind of there is higher flexibility, the expression-form of scalability and ease for operation is described video and portrays, this expression-form can contain the video of different quality, different resolution, different compression ratio.During transmission, meet the compressed video of application demand further according to the constraint of transmission conditions and the adaptive extraction of the demand of user.
The present invention, according to the characteristic sum demand of UAV Video, designs a kind of video expression-form of novel hierarchy, and this expression, compared with original video or conventional compression code stream, has less data volume, more flexibly organizational form and operation-interface easily.The concrete structure that this video is expressed as shown in Figure 4, is an image sets unit for 16 two field pictures in figure, adopts the form of layering to describe video.Wherein, background layer is lowermost layer, adopts the mode of image mosaic that all image mosaic in an image sets are got up to become a width panorama Background, and the mode of recycling Image Coding is compressed, k the sublayer that background layer also can be divided into compression effectiveness not wait; Be destination layer on background layer, each two field picture of this layer only comprises the target area of each two field picture in original video, and this layer can adopt independent inter-frame prediction method to compress, and equally, destination layer also can be divided into m sublayer according to the difference of compression effectiveness; Be enhancement layer on destination layer, a frame complete image of the corresponding original video of each two field picture of enhancement layer, and be divided into n sublayer according to the difference of compression effectiveness.In this hierarchy, what embody to enhancement layer from background layer to destination layer is the increase of video content again, and from the 0th to kth-1/m-1/n-1 sublayer in each layer, increasing progressively of quality that corresponding is or resolution.Inter-layer prediction can be carried out between each sublayer of same layer to reduce redundancy, and prediction can be carried out between different layers reduce redundancy further, also can not carry out prediction to ensure independence.
In the transmission of video application of reality, can limit according to the bandwidth of reality, delay requirements etc., select the part level in production and transfer structure flexibly, as: background layer or destination layer or background layer+destination layer or background layer+enhancement layer or background layer+destination layer+enhancement layer.And wherein every one deck also can optionally arrange sublayer number.
The frame of video of unmanned plane shooting often has the overlapping of larger area with between frame, and between two frames, difference major embodiment is global motion, as Fig. 5 institute.Video frame rate is 30 frames/second, and the image in figure is 9 adjacent frames, and the Duplication between them reaches more than 90%, there is comparatively simple global translation motion between frame and frame.
If spliced according to overlapping region coupling by the image of taking photo by plane in video, width panorama sketch in a big way can be obtained.Figure 6 shows that the panorama sketch after 30 two field picture splicings in 1 second video of taking photo by plane, have larger overlapping region between consecutive frame in this 30 two field picture as seen.
If compress spliced image, not only can reduce compression time and can data volume be reduced.The simulation video of above figure is example, common CP is compressed, according to H.264 to 30 frames adopt I-P-P-P forms compress required for time be 26.003s, and to 30 two field pictures splice be re-used as piece image compress required for time be 2.466s.The compression performance that Fig. 7 gives two kinds of methods compares, and can find out that the video quality difference after two kinds of method compressions is less, and recompress the data volume after can reducing compression further after splicing.
What more than experiment adopted is simulation video, translational motion mainly overall between consecutive frame, can absolutely prove, carry out splicing recompression to multi-frame video and can improve the Time and place efficiency of compression under the prerequisite not affecting compression quality, the unmanned plane being highly suitable for calculating and transfer resource is limited transmission of video of taking photo by plane is applied.
Take photo by plane in video in reality, due to the complexity of global motion, spliced image has certain mass loss, as shown in Figure 8, Figure 9.Video quality after adopting the method may affect compression to a certain extent, but the Time and place efficiency of its compression still accounts for main advantage.Wherein, H.264 employing carries out compression required time is 24.488s, and adopt the compression method required time based on splicing to be 1.32s, compression time reduces more than 90%.And compression performance mainly embodies advantage when low bit-rate.
All suppose to take photo by plane in above experiment and there is not the object of self-movement in video, and in actual photographed, also there will be the situation having self-movement target, as shown in Figure 10, wherein black vehicle is moving object, its embody motion in video and global motion inconsistent.
For the video that there is self-movement object, adopt the compression method based on splicing can lose the information of moving object, therefore, need by self-movement Objective extraction out, to compress separately.As shown in figure 11, left figure is background splicing effect figure, and right figure is the self-movement object be partitioned into.Background layer is adopted and tests identical method compress with above, adopt traditional inter prediction to compress to independently moving object.Compression time used is 3.014s, and the 27.455s compared H.264 still reduces more than 90%.Compression performance is slightly poorer than H.264, but when subjective quality is suitable, code check declines to some extent.
Embodiment one:
As shown in figure 12, far away in unmanned plane voyage, when flying height is high, transmission bandwidth is narrower and time delay is comparatively large, and traditional video compression standard algorithm cannot provide subjective quality acceptable video image, and can increase time delay further because compressing complexity height.Now, the extreme example of this programme can be adopted, i.e. a coding transmission background layer.This layer adopts the mode of image mosaic that all image mosaic in an image sets are got up to become a width panorama Background, and the mode of recycling still image coding is compressed, k (k >=1) the individual sublayer that background layer also can be divided into compression effectiveness not wait.
Specific implementation step following (establishing an image sets to be made up of N frame):
Step 1) benchmark image chooses: benchmark image is the image processed at first in splicing, and follow-up image projects to the coordinate system of this image according to mapping relations, namely using the coordinate system of benchmark image as the coordinate system of stitching image.Benchmark image can get any frame in image sets, and ordinary circumstance gets the first frame or last frame or key frame.
Step 2) benchmark image feature point extraction: existing algorithm can be utilized as Harris operator, Moravec operator, Plessey operator, Forstner operator, SUSAN operator, SIFT algorithm etc.
Step 3) image sequence splicing: this step is mainly spliced in a certain order to remaining N-1 two field picture in image sets, and concatenation can be divided into Feature point correspondence, virtual borderlines, Images uniting three parts.
3.1) Feature point correspondence: determine in image to be spliced and the point one to one of the characteristic point in benchmark image, concrete grammar: can first adopt and step one same procedure extract minutiae, recycling matching algorithm obtains corresponding relation; Or the point that directly given range search is the highest with the characteristic point similarity of benchmark image in image to be spliced.
3.2) virtual borderlines: the position relationship right according to characteristic of correspondence point, calculates the coordinate transform of image relative datum image to be spliced, can adopt translation model, affine model, bilinear model or perspective projection model.The transformation parameter that this step obtains can carry out compressing, store and transmitting.
3.3) Images uniting: the transformation relation of trying to achieve according to upper step, by the evolution of each pixel in image to be spliced to benchmark image coordinate system, and copies to correspondence position by its pixel value.If overlapping with the pixel of benchmark image, then get the average of two two field picture correspondence position pixels or choose one of them value according to judgment criteria.
Step 4) image compression: adopt conventional images compression algorithm to carry out the compression of quality scalability to splicing result, as shown in Figure 5, background layer 0 is the compressed image of minimum quality, background layer 1 ~ k-1 compresses the classification refinement of the compression residual error of background layer 0, can be improved the quality of background layer by superposition.The setting of level and the quality of each level can be arranged according to the change of the network bandwidth.
The video of this embodiment expresses the panorama sketch that not only can provide shooting environmental, can also reduce from panorama sketch according to the transformation parameter extracted in step 3 take the photograph the original appearance of video.
Embodiment two:
When unmanned plane during flying height is lower, institute is taken the photograph in video exists self-movement target clearly, only adopt the method for image mosaic to carry out compression and cannot react self-movement clarification of objective in video really, and may because the change in location of moving target makes to there is expendable pixel loss in stitching image to blocking of background.Now, the scheme that background layer and destination layer can be adopted to combine is expressed video content and compresses.As shown in figure 12, moving target is split from background, only background layer is spliced to form to background image, and moving target carries out separately expressing and compressing.
Specific implementation step is as follows:
Step 1) moving Object Segmentation: because UAV Video video background of taking photo by plane also exists motion, therefore needing by analyzing whole sports ground, distinguishing global motion and local motion, extract self-movement target further.And record moving target exact position in the corresponding frame.
Step 2) background layer generation: moving target is removed from image, only retains background information, and adopt the mode of example one to splice background layer, compress, generate the background layer with different quality sublayer.
Step 3) destination layer generation: namely to the independent compressed encoding of moving target.This step can adopt canonical algorithm; Or separately motion model is set up to target, and estimation model parameter, replace the motion estimation module in canonical algorithm.Destination layer also can carry out the compression of quality scalability, and as shown in Figure 6, destination layer 0 is the compressed image of minimum quality, and destination layer 1 ~ m-1 compresses the classification refinement of the compression residual error of destination layer 0, can be improved the quality of destination layer by superposition.The setting of level and the quality of each level can be arranged according to the change of the network bandwidth.
Based on above video expression structure, the panorama sketch of shooting environmental can be provided; Or the tracing of the movement figure of moving target in panorama sketch; Or panorama sketch is split to the video sequence obtaining background, then be the position in corresponding frame according to moving target, the original appearance of reduction capture video.
Embodiment three:
Due to the existence of the error of calculation, background splicing and target following all may cause the deviation of actual position information, make the video rebuild there is irremediable distortion.And in actual applications, also there is the application higher to precise requirements.If under the condition that transmission bandwidth or memory device allow, more complicated compressed format can be adopted.Now, the scheme that background layer, destination layer and enhancement layer can be adopted to combine is expressed video content and is compressed.As shown in Figure 4, enhancement layer directly carries out predictive coding to frame of video, and can generate different quality layers according to the condition of reality.In this embodiment, destination layer is optional generation, and actual compound mode can be background layer+destination layer+enhancement layer; Or background layer+enhancement layer.Enhancement layer can utilize the information of background layer and/or destination layer to carry out inter-layer prediction, also can only adopt the interframe of this layer or infra-frame prediction to encode.
Specific implementation step is as follows:
Step one: design video expression structure according to the actual requirements, utilizes method generation background layer or the background layer+destination layer of embodiment one or embodiment two according to structure composition.
Step 2: enhancement layer generates: background layer and/or destination layer can be utilized to predict, also can only adopt the interframe of this layer or infra-frame prediction to carry out absolute coding.Enhancement layer also can carry out the compression of quality scalability, and as shown in Figure 7, enhancement layer 0 is the compressed image of minimum quality, and enhancement layer 1 ~ n-1 compresses the classification refinement of the compression residual error of enhancement layer 0, can be improved the quality of enhancement layer by superposition.The setting of level and the quality of each level can be arranged according to the change of the network bandwidth.
Based on above video expression structure, the panorama sketch of shooting environmental can be provided; Or the motion tracking video of pinpoint target; Or more complete and video sequence that is better quality.
The above is only the preferred embodiment of the present invention; it should be pointed out that for those skilled in the art, under the premise without departing from the principles of the invention; can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.
Claims (7)
1. based on the UAV Video adaptive transmission method that layering is expressed, it is characterized in that, the method comprises the hierarchy that video is expressed; Described hierarchy comprises background layer, destination layer and enhancement layer from the bottom to top successively;
Described background layer is: one being comprised all image mosaic in the image sets of multiple image becomes a width panorama Background, and the mode of recycling Image Coding is compressed; Be not divided into k sublayer according to compression effectiveness, be followed successively by 0 from the bottom to top, 1,2 ... k-1;
Described destination layer is: each two field picture only comprises the target area of each two field picture in original video, adopts independent inter-frame prediction method to compress; Be not divided into m sublayer according to compression effectiveness, be followed successively by 0 from the bottom to top, 1,2 ... m-1;
Described enhancement layer is: a frame complete image of the corresponding original video of each two field picture; Be not divided into n sublayer according to compression effectiveness, be followed successively by 0 from the bottom to top, 1,2 ... n-1;
The mode of production and transfer layer of structure selects one from following five kinds of combinations: background layer; Destination layer; Background layer and destination layer; Background layer and enhancement layer; Background layer, destination layer and enhancement layer.
2. as claimed in claim 1 a kind of based on layering express UAV Video adaptive transmission method, it is characterized in that, comprise splicing and the compression of background layer image, specifically comprise the following steps:
2.1), image sequence splicing: select a two field picture as benchmark image, remaining N-1 two field picture in image sets is spliced with benchmark image according to the order of sequence, finally obtain the panorama sketch that a width is made up of N two field picture, concatenation comprises Feature point correspondence, virtual borderlines and Images uniting;
2.1.1), Feature point correspondence: determine in image to be spliced and the point one to one of the characteristic point in benchmark image, utilize existing algorithm extract minutiae, recycling matching algorithm obtains corresponding relation; Or the point that directly given range search is the highest with the characteristic point similarity of benchmark image in image to be spliced;
2.1.2), virtual borderlines: the position relationship corresponding according to characteristic of correspondence point, utilizes existing model to calculate the coordinate transform of image relative datum image to be spliced; The transformation parameter of gained is compressed, stores and transmitted;
2.1.3), Images uniting: according to step 2.3.2) coordinate conversion relation, by the evolution of each pixel in image to be spliced to benchmark image coordinate system, and its pixel value is copied to correspondence position, if overlapping with the pixel of benchmark image, then get the average of two two field picture correspondence position pixels or choose one of them value.
2.2), image compression: adopt conventional images compression algorithm to carry out the compression of quality scalability to splicing result, form k background layer sublayer; Wherein, background layer 0 is the compressed image of minimum quality, and background layer 1 ~ background layer k-1 compresses the classification refinement of the compression residual error of background layer 0; The quality of background layer is improved by the superposition of multiple background layer sublayer; The setting of level and the quality of each level are arranged according to the change of the network bandwidth.
3. a kind of UAV Video adaptive transmission method of expressing based on layering as claimed in claim 2, it is characterized in that, during the way selection background layer of production and transfer layer of structure, an image is N frame, specifically comprises the following steps:
3.1), benchmark image is chosen: get any frame in image sets, using the coordinate system of benchmark image as the coordinate system of stitching image;
3.2), benchmark image feature point extraction: utilize existing algorithm extract minutiae;
3.3), image sequence splicing: carry out step 2.1);
3.4), image compression: carry out step 2.2).
4. as claimed in claim 2 a kind of based on layering express UAV Video adaptive transmission method, it is characterized in that, when there is self-movement target clearly in screen captured by unmanned plane, the way selection background layer of production and transfer layer of structure and destination layer; Specifically comprise the following steps:
4.1) moving Object Segmentation: by analyzing whole shooting picture, distinguishes global motion and local motion, extracts self-movement target further, and records moving target exact position in the corresponding frame;
4.2) background layer generates: removed from image by moving target, only retain background information, carry out step 2.1) and step 2.2) completing splicing and the compression of background layer, generation has the background layer of different quality sublayer.
4.3) destination layer generates: to the independent compressed encoding of moving target, adopt existing algorithm or set up motion model separately to target, the motion estimation module after modeling in estimation model parameter alternate algorithm;
Destination layer is carried out to the compression of quality scalability, destination layer 0 is the compressed image of minimum quality, destination layer 1 ~ destination layer m-1 compresses the classification refinement of the compression residual error of destination layer 0, improved the quality of destination layer by superposition, the setting of level and the quality of each level are arranged according to the change of the network bandwidth.
5. as claimed in claim 4 a kind of based on layering express UAV Video adaptive transmission method, it is characterized in that, the way selection background layer of production and transfer layer of structure and enhancement layer, or select background layer, destination layer and enhancement layer time; Specifically comprise the following steps:
5.1) design video expression structure according to the actual requirements, select generation background layer/background layer and destination layer according to structure composition;
5.2) enhancement layer generates: utilize background layer and/or destination layer to predict, or adopt the interframe of this layer or infra-frame prediction to carry out absolute coding; Enhancement layer is carried out to the compression of quality scalability: enhancement layer 0 is the compressed image of minimum quality, enhancement layer 1 ~ enhancement layer n-1 compresses the classification refinement of the compression residual error of enhancement layer 0, is improved the quality of enhancement layer by superposition; The setting of level and the quality of each level can be arranged according to the change of the network bandwidth.
6. as claimed in claim 1 a kind of based on layering express UAV Video adaptive transmission method, it is characterized in that, between each sublayer of the same layer of described hierarchy, reduce redundancy by inter-layer prediction.
7. as claimed in claim 2 a kind of based on layering express UAV Video adaptive transmission method, it is characterized in that, between the different layers of described hierarchy, reduce redundancy by inter-layer prediction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410608007.1A CN104349142B (en) | 2014-11-03 | 2014-11-03 | A kind of UAV Video adaptive transmission method based on layering expression |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410608007.1A CN104349142B (en) | 2014-11-03 | 2014-11-03 | A kind of UAV Video adaptive transmission method based on layering expression |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104349142A true CN104349142A (en) | 2015-02-11 |
| CN104349142B CN104349142B (en) | 2018-07-06 |
Family
ID=52503834
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410608007.1A Active CN104349142B (en) | 2014-11-03 | 2014-11-03 | A kind of UAV Video adaptive transmission method based on layering expression |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104349142B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107124559A (en) * | 2017-06-19 | 2017-09-01 | 成都优孚达信息技术有限公司 | A kind of communication data compression method |
| CN107197275A (en) * | 2017-06-15 | 2017-09-22 | 武汉大学深圳研究院 | Cruise UAV Video compression method based on long-range Large Scale Background template |
| CN107409233A (en) * | 2015-03-05 | 2017-11-28 | 索尼公司 | Image processing apparatus and image processing method |
| CN108496370A (en) * | 2017-03-30 | 2018-09-04 | 深圳市大疆创新科技有限公司 | Video transmission method, method of reseptance, system and unmanned vehicle |
| CN108496369A (en) * | 2017-03-30 | 2018-09-04 | 深圳市大疆创新科技有限公司 | Transmission of video, method of reseptance, system, equipment and unmanned vehicle |
| CN109089097A (en) * | 2018-08-28 | 2018-12-25 | 恒信东方文化股份有限公司 | A kind of object of focus choosing method based on VR image procossing |
| CN109584158A (en) * | 2018-10-25 | 2019-04-05 | 成都城电电力工程设计有限公司 | A kind of distant view photograph automatic splicing method |
| CN109600617A (en) * | 2018-12-19 | 2019-04-09 | 北京东土科技股份有限公司 | Coding, retransmission method, device, equipment and the storage medium of video data |
| CN109740505A (en) * | 2018-12-29 | 2019-05-10 | 成都视观天下科技有限公司 | A kind of training data generation method, device and computer equipment |
| CN113490026A (en) * | 2021-05-11 | 2021-10-08 | 浙江加我网络科技有限公司 | Film and television level remote synchronous film-passing control method |
| CN114331843A (en) * | 2021-12-28 | 2022-04-12 | 苏州思卡信息系统有限公司 | Image splicing method based on gray level histogram |
| WO2022088200A1 (en) * | 2020-11-02 | 2022-05-05 | 深圳市大疆创新科技有限公司 | Video transmission method and apparatus, and mobile platform and computer-readable storage medium |
| CN116723333A (en) * | 2023-08-02 | 2023-09-08 | 清华大学 | Layered video coding method, device and product based on semantic information |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103347185A (en) * | 2013-06-28 | 2013-10-09 | 北京航空航天大学 | Unmanned aerial vehicle scouting image synthesis compressed encoding method based on selective block transformation |
| CN103679674A (en) * | 2013-11-29 | 2014-03-26 | 航天恒星科技有限公司 | Method and system for splicing images of unmanned aircrafts in real time |
| CN103870598A (en) * | 2014-04-02 | 2014-06-18 | 北京航空航天大学 | Unmanned aerial vehicle surveillance video information extracting and layered cataloguing method |
-
2014
- 2014-11-03 CN CN201410608007.1A patent/CN104349142B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103347185A (en) * | 2013-06-28 | 2013-10-09 | 北京航空航天大学 | Unmanned aerial vehicle scouting image synthesis compressed encoding method based on selective block transformation |
| CN103679674A (en) * | 2013-11-29 | 2014-03-26 | 航天恒星科技有限公司 | Method and system for splicing images of unmanned aircrafts in real time |
| CN103870598A (en) * | 2014-04-02 | 2014-06-18 | 北京航空航天大学 | Unmanned aerial vehicle surveillance video information extracting and layered cataloguing method |
Non-Patent Citations (2)
| Title |
|---|
| MICHAEL TEUTSCH等: "《Detection, Segmentation and Tracking of Moving Objects in UAV Videos》", 《2012 IEEE NINETH INTERNATIONAL CONFERENCE ON ADVANCED VIDEO AND SIGNAL BASED SURVEILLANCE》 * |
| SAEED YAHYANEJAD等: "《Incremental Mosaicking of Images from Autonomous, Samll-Scale UAVs》", 《2010 SEVENTH IEEE INTERNATIONAL CONFERENCE ON ADVANCED VIDEO AND SIGNAL BASED SURVEILLANCE》 * |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107409233B (en) * | 2015-03-05 | 2020-04-14 | 索尼公司 | Image processing apparatus, image processing method, and program |
| CN107409233A (en) * | 2015-03-05 | 2017-11-28 | 索尼公司 | Image processing apparatus and image processing method |
| CN108496370A (en) * | 2017-03-30 | 2018-09-04 | 深圳市大疆创新科技有限公司 | Video transmission method, method of reseptance, system and unmanned vehicle |
| CN108496369A (en) * | 2017-03-30 | 2018-09-04 | 深圳市大疆创新科技有限公司 | Transmission of video, method of reseptance, system, equipment and unmanned vehicle |
| WO2018176303A1 (en) * | 2017-03-30 | 2018-10-04 | 深圳市大疆创新科技有限公司 | Video transmitting and receiving method, system, and device, and unmanned aerial vehicle |
| CN107197275A (en) * | 2017-06-15 | 2017-09-22 | 武汉大学深圳研究院 | Cruise UAV Video compression method based on long-range Large Scale Background template |
| CN107124559B (en) * | 2017-06-19 | 2019-08-13 | 成都优孚达信息技术有限公司 | A kind of communication data compression method |
| CN107124559A (en) * | 2017-06-19 | 2017-09-01 | 成都优孚达信息技术有限公司 | A kind of communication data compression method |
| CN109089097A (en) * | 2018-08-28 | 2018-12-25 | 恒信东方文化股份有限公司 | A kind of object of focus choosing method based on VR image procossing |
| CN109584158A (en) * | 2018-10-25 | 2019-04-05 | 成都城电电力工程设计有限公司 | A kind of distant view photograph automatic splicing method |
| CN109600617A (en) * | 2018-12-19 | 2019-04-09 | 北京东土科技股份有限公司 | Coding, retransmission method, device, equipment and the storage medium of video data |
| CN109740505A (en) * | 2018-12-29 | 2019-05-10 | 成都视观天下科技有限公司 | A kind of training data generation method, device and computer equipment |
| CN109740505B (en) * | 2018-12-29 | 2021-06-18 | 成都视观天下科技有限公司 | Training data generation method and device and computer equipment |
| WO2022088200A1 (en) * | 2020-11-02 | 2022-05-05 | 深圳市大疆创新科技有限公司 | Video transmission method and apparatus, and mobile platform and computer-readable storage medium |
| CN113490026A (en) * | 2021-05-11 | 2021-10-08 | 浙江加我网络科技有限公司 | Film and television level remote synchronous film-passing control method |
| CN113490026B (en) * | 2021-05-11 | 2023-05-05 | 浙江斐越数字科技有限公司 | Video-level remote synchronous film-passing control method |
| CN114331843A (en) * | 2021-12-28 | 2022-04-12 | 苏州思卡信息系统有限公司 | Image splicing method based on gray level histogram |
| CN114331843B (en) * | 2021-12-28 | 2022-10-04 | 苏州思卡信息系统有限公司 | Image splicing method based on gray level histogram |
| CN116723333A (en) * | 2023-08-02 | 2023-09-08 | 清华大学 | Layered video coding method, device and product based on semantic information |
| CN116723333B (en) * | 2023-08-02 | 2023-10-31 | 清华大学 | Layered video coding method, device and product based on semantic information |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104349142B (en) | 2018-07-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104349142A (en) | Layered representation-based unmanned plane video adaptive transmission method | |
| JP4991712B2 (en) | Digital image processing apparatus, processing method, and computer program product | |
| US8351685B2 (en) | Device and method for estimating depth map, and method for generating intermediate image and method for encoding multi-view video using the same | |
| WO2018118159A1 (en) | Sphere projected motion estimation/compensation and mode decision | |
| CN114079779B (en) | Image processing method, intelligent terminal and storage medium | |
| CN105190688A (en) | Method and apparatus for viewing images | |
| Liu et al. | Three-dimensional point-cloud plus patches: Towards model-based image coding in the cloud | |
| KR20160032137A (en) | Feature-based image set compression | |
| CN101521823B (en) | Spatial correlation panoramic data compressing method | |
| WO2018151978A1 (en) | Processing of equirectangular object data to compensate for distortion by spherical projections | |
| US20110129012A1 (en) | Video Data Compression | |
| Gong et al. | An image-sequence compressing algorithm based on homography transformation for unmanned aerial vehicle | |
| JP2017207818A (en) | Image processing apparatus, image processing method and program | |
| Zhang et al. | An inter-image redundancy measure for image set compression | |
| Vishwanath et al. | Motion compensated prediction for translational camera motion in spherical video coding | |
| Angelino et al. | Sensor aided h. 264 video encoder for uav applications | |
| Anandan et al. | Video as an image data source: efficient representations and applications | |
| Wu et al. | 3D mesh based inter-image prediction for image set compression | |
| US11665330B2 (en) | Dynamic-baseline imaging array with real-time spatial data capture and fusion | |
| Angelino et al. | A sensor aided H. 264 encoder tested on aerial imagery for SFM | |
| KR100737808B1 (en) | Method for efficiently compressing 2d multi-view images | |
| Milani | Compression of multiple user photo galleries | |
| Sehli et al. | WeLDCFNet: Convolutional Neural Network based on Wedgelet Filters and Learnt Deep Correlation Features for depth maps features extraction | |
| Belyaev et al. | Drone HDR infrared video coding via aerial map prediction | |
| Chen et al. | Multisource surveillance video data coding with hierarchical knowledge library |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |