WO2019047059A1 - 无线数据传输方法和设备 - Google Patents

无线数据传输方法和设备 Download PDF

Info

Publication number
WO2019047059A1
WO2019047059A1 PCT/CN2017/100701 CN2017100701W WO2019047059A1 WO 2019047059 A1 WO2019047059 A1 WO 2019047059A1 CN 2017100701 W CN2017100701 W CN 2017100701W WO 2019047059 A1 WO2019047059 A1 WO 2019047059A1
Authority
WO
WIPO (PCT)
Prior art keywords
group
frame
frames
processor
encoded
Prior art date
Application number
PCT/CN2017/100701
Other languages
English (en)
French (fr)
Inventor
朱磊
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to EP17924248.2A priority Critical patent/EP3681156B1/en
Priority to CN201780006922.5A priority patent/CN108521870B/zh
Priority to PCT/CN2017/100701 priority patent/WO2019047059A1/zh
Priority to CN202011468184.6A priority patent/CN112672152A/zh
Publication of WO2019047059A1 publication Critical patent/WO2019047059A1/zh
Priority to US16/727,428 priority patent/US20200177913A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/587—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64C—AEROPLANES; HELICOPTERS
    • B64C39/00—Aircraft not otherwise provided for
    • B64C39/02—Aircraft not otherwise provided for characterised by special use
    • B64C39/024—Aircraft not otherwise provided for characterised by special use of the remote controlled vehicle type, i.e. RPV
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146—Data rate or code amount at the encoder output
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H04N19/39—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability involving multiple description coding [MDC], i.e. with separate layers being structured as independently decodable descriptions of input picture data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams, manipulating MPEG-4 scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/238—Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams
    • H04N21/2383—Channel coding or modulation of digital bit-stream, e.g. QPSK modulation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2101/00—UAVs specially adapted for particular uses or applications
    • B64U2101/30—UAVs specially adapted for particular uses or applications for imaging, photography or videography

Definitions

  • the present disclosure relates to the field of wireless communication technologies, and more particularly, to a wireless data transmission method and apparatus.
  • Low-latency map transmission (especially for drone applications) requires that the transmission time of each frame be controlled within a certain range to avoid large fluctuations. Usually, the decoding display at the receiving end will be stuck. E.g,
  • the code stream coded at this time is stable. If the channel suddenly changes, it will also cause transmission delay jitter. For example, assuming that the bandwidth of the channel is reduced by a factor of two, it means that the transmission delay is increased by a factor of two.
  • the average rate of a period of time is controlled by a rate control algorithm to be stable at a given target code rate, thereby ensuring an average overall delay jitter range for frames of several frames or for a period of time. Smaller.
  • the present disclosure proposes to decompose the original image video sequence in a time-decomposed manner, and then separately perform coded transmission and independent fault tolerance for each of the decomposed sub-sequences.
  • each image frame in the sub-decomposed sub-sequence combined selection transmission is performed according to the size of each sub-sequence code stream and the channel conditions at the time.
  • the receiving end can perform decoding reconstruction according to the received correct code stream. For the image of the frame that is not received or erroneous, linear interpolation is performed using the correctly received frame, thereby obtaining the final complete reconstructed image.
  • a method for processing video data includes: performing time sampling on an image sequence to form a plurality of sub-sequences; separately encoding the plurality of sub-sequences to form a plurality of encoded sub-sequences; and determining a size of each frame and a channel bandwidth according to the encoded plurality of sub-sequences , select the frame to transfer.
  • the oldest frames are found from each of the encoded subsequences and the frames are combined to form a group, which is sequentially repeated to form a plurality of groups. Then, based on the size of each frame in each group and the channel bandwidth, the frames to be transmitted in each group are selected.
  • the frames to be transmitted in the group can be selected according to the following conditions:
  • S represents the total code stream size of the selected frame in the group
  • T represents the channel bandwidth
  • the frames to be transmitted in the group can be selected according to the following conditions:
  • S represents the total code stream size of the selected frame in the group
  • T represents the channel bandwidth
  • D represents the tolerance threshold
  • a drone including an imaging device, a processor, and a transmission unit.
  • the imaging device is configured to capture a sequence of images.
  • the processor is configured to time downsample the sequence of images to form a plurality of subsequences.
  • the processor is further configured to separately encode the plurality of subsequences to form the encoded plurality of subsequences.
  • the processor is further configured to select a frame to transmit based on the size of each of the plurality of sub-sequences encoded and the channel bandwidth.
  • the transmission unit is configured to transmit the selected frame.
  • a computer readable storage medium storing a computer program.
  • the computer program is run by the at least one processor, causing the at least one processor to perform the following operations: time down sampling the image sequence to form a plurality of subsequences; The sequences are separately encoded to form a plurality of encoded subsequences; and the frame to be transmitted is selected according to the size of each frame in the encoded plurality of subsequences and the channel bandwidth.
  • delay jitter ie, instability of transmission time
  • real-time source channel mismatch can be reduced at the frame level.
  • FIG. 1 is a flow chart showing a method in accordance with one embodiment of the present disclosure.
  • FIG. 2 is a schematic diagram showing temporal downsampling of a sequence of images, in accordance with one embodiment of the present disclosure.
  • FIG. 3 is a block diagram showing a drone according to an embodiment of the present disclosure.
  • FIG. 4 is a block diagram showing a computer readable storage medium in accordance with one embodiment of the present disclosure.
  • FIG. 1 is a flow chart showing a method 10 in accordance with one embodiment of the present disclosure.
  • step S110 an image sequence including a plurality of frames is time-sampled to form a plurality of sub-sequences.
  • FIG. 2 is a schematic diagram showing temporal downsampling of a sequence of images, in accordance with one embodiment of the present disclosure.
  • the original image sequence (P0, P1, ... P7, 7) is decomposed into 4 sub-sequences.
  • the first subsequence includes frames P0, P4, P8, ...
  • the second subsequence includes frames P1, P5, P9, ...
  • the third subsequence includes frames P2, P6, P10, ..., fourth.
  • the subsequences contain frames P3, P7, P11....
  • four sub-video sequences whose temporal resolution is 1/4 of the original video sequence are obtained.
  • the four sub-sequences shown in FIG. 2 are only one specific example.
  • the present disclosure is not limited to dividing the original image sequence into four sub-sequences, but may be divided into a greater or lesser number of sub-sequences according to actual needs.
  • the original image sequence (P0, P1, ...) can be Decomposed into 6 subsequences.
  • the first subsequence includes frames P0, P6, P12, ...
  • the second subsequence includes frames P1, P7, P13, ...
  • the third subsequence includes frames P2, P8, P14, ..., fourth.
  • the subsequences include frames P3, P9, P15, ..., the fifth subsequence includes frames P4, P10, P16, ..., and the sixth subsequence includes frames P5, P11, P17, ....
  • the original image sequence (P0, P1, 7) can also be decomposed into two sub-sequences, in which case the first sub-sequence contains frames P0, P2, P4, ..., the second sub-sequence
  • the sequence contains frames P1, P3, P5....
  • step S120 the plurality of subsequences obtained at step S110 are separately encoded (i.e., compressed) to form a plurality of encoded subsequences (encoded code streams). For example, for each subsequence, a corresponding encoder can be used for encoding. The outputs of the plurality of encoders used are aggregated to form an encoded code stream.
  • the encoded code stream includes the encoded individual frames.
  • step S130 the encoded frame to be transmitted is selected based on the size of each frame (encoded frame) and the channel bandwidth of the encoded plurality of subsequences.
  • the current frame code stream of the four sub-sequences in the group may be performed according to the size of the current frame code stream and the real-time channel estimation value of each sub-sequence in each group (G0, G1, ). Combine transmission to achieve real-time matching with the wireless channel.
  • the code stream sizes of the four encoded frames P0, P1, P2, and P3 in the group G0 are S0, S1, S2, and S3, respectively.
  • the estimated value of the current channel bandwidth ie, the amount of transmittable data at the current group G0 time
  • T the estimated value of the current channel bandwidth
  • the value of T may be predefined (eg, based on historical values) or may be calculated using a channel bandwidth estimator.
  • the transmission and reception states of the current four sub-sequences are error-free. Then,
  • the code stream including the four encoded frames P0, P1, P2, and P3 may be completely transmitted.
  • the selection is made from S0, S1, S2, and S3 so that the total size of the combined code streams is closest to T.
  • the code stream containing as many number of encoded frames as possible is selected on the premise that the total code stream size after the combination is kept closest to T.
  • select send includes The code stream of the frames P0 and P2 after the code.
  • select send includes The code stream of the frames P0 and P2 after the code.
  • S0+S1 ⁇ T, S0+S2+S3 ⁇ T is satisfied, and the size of S0+S1 is equivalent to the size of S0+S2+S3, it is preferable to transmit the frames including P0, P2, and P3 after encoding. Code stream.
  • the combined data size should be less than T.
  • D is the tolerance threshold
  • S is the total size of the selected encoded frame.
  • the code stream containing as many number of encoded frames as possible is selected on the premise that the condition is satisfied.
  • the code stream of each subsequence in the group can also be received in units of groups. For example, when one or more of the frames P0, P1, P2, P3 in the group G0 are correctly received, the original image of the time position at which it is located can be restored using the correctly received sub-sequence image, without applying the error-producing sub- sequence. Conversely, for a subsequence that has an error, the original image of its corresponding temporal position can be recovered by linearly weighted interpolation using the correctly received reconstruction sequence, thereby producing a final reconstructed image sequence.
  • the time-dimensional linear interpolation can be performed by other correctly received frame images, thereby obtaining a reconstructed image of the time position at which it is located. In this way, delay jitter due to real-time source channel mismatch can be reduced at the frame level, and the fault tolerance is improved.
  • FIG. 3 is a block diagram showing a drone 30 in accordance with one embodiment of the present disclosure.
  • the drone 30 includes an imaging device 310, a processor 320, and a transmission unit 330.
  • Imaging device 310 is configured to capture a sequence of images comprising a plurality of frames.
  • imaging device 310 can include one or more cameras distributed over the drone.
  • the processor 320 is configured to perform an operation on a sequence of images comprising a plurality of frames captured by the imaging device 310. Specifically, the processor 320 performs time downsampling on the captured image sequence including a plurality of frames to form a plurality of sub-sequences. The processor 320 also separately encodes the plurality of subsequences formed to form a plurality of encoded subsequences. In addition, the processor 320 also selects the encoded frame to be transmitted according to the size of each encoded frame in the encoded plurality of subsequences and the estimated value of the current channel bandwidth.
  • the processor 320 can find the oldest time from each of the encoded subsequences. Frames and combine these encoded frames to form a group. The processor 320 repeats the operations in sequence to form a plurality of groups. And, the processor 320 selects the encoded frame to be transmitted in each group according to the size of each encoded frame in each group and the estimated value of the current channel bandwidth.
  • processor 320 can select the encoded frame to be transmitted in the group according to the following conditions:
  • processor 320 selects as many of the encoded frames as possible in each group for transmission.
  • the processor 320 may select the encoded frame to be transmitted in the group according to the following conditions:
  • processor 320 selects as many of the encoded frames as possible in each group for transmission.
  • Transmission unit 330 is configured to transmit frames selected by processor 320.
  • transmission unit 330 can include a wireless communication module that employs a variety of wireless communication technologies (eg, cellular communication, Bluetooth, WiFi, etc.).
  • embodiments of the present disclosure may be implemented by means of a computer program product.
  • the computer program product can be a computer readable storage medium.
  • a computer program is stored on a computer readable storage medium, and when executed on a computing device, related operations can be performed to implement the above-described aspects of the present disclosure.
  • FIG. 4 is a block diagram showing a computer readable storage medium 40 in accordance with one embodiment of the present disclosure.
  • computer readable storage medium 40 includes a program 410.
  • the program 410 when executed by the at least one processor, causes the at least one processor to perform time-down sampling of an image sequence including a plurality of frames to form a plurality of sub-sequences; separately encoding the plurality of sub-sequences to form an encoded code And a plurality of sub-sequences; and selecting the encoded frame to be transmitted according to the size of each of the encoded plurality of sub-sequences and the estimated value of the current channel bandwidth.
  • At least one processor is caused Performing the following operations: finding the oldest frame from each of the encoded subsequences, and combining the encoded frames to form a group; repeating the operations in sequence to form a plurality of groups; and according to each group The size of each encoded frame and the estimated value of the current channel bandwidth are selected, and the encoded frame to be transmitted in each group is selected.
  • Examples of computer readable storage media 40 in embodiments of the present disclosure include, but are not limited to, semiconductor storage media, optical storage media, magnetic storage media, or any other computer readable storage media.
  • Such an arrangement of the present disclosure is typically provided as software, code, and/or other data structures, such as one or more, that are arranged or encoded on a computer readable medium such as an optical medium (eg, CD-ROM), floppy disk, or hard disk.
  • a computer readable medium such as an optical medium (eg, CD-ROM), floppy disk, or hard disk.
  • Software or firmware or such a configuration may be installed on the computing device such that one or more processors in the computing device perform the technical solutions described in the embodiments of the present disclosure.
  • each functional module or individual feature of the device used in each of the above embodiments may be implemented or executed by circuitry, typically one or more integrated circuits.
  • Circuitry designed to perform the various functions described in this specification can include general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs) or general purpose integrated circuits, field programmable gate arrays (FPGAs), or others.
  • a general purpose processor may be a microprocessor, or the processor may be an existing processor, controller, microcontroller, or state machine.
  • the above general purpose processor or each circuit may be configured by a digital circuit or may be configured by a logic circuit.
  • the present disclosure may also use integrated circuits obtained using the advanced technology.
  • the program running on the device may be a program that causes a computer to implement the functions of the embodiments of the present invention by controlling a central processing unit (CPU).
  • the program or information processed by the program may be temporarily stored in a volatile memory (such as a random access memory RAM), a hard disk drive (HDD), a non-volatile memory (such as a flash memory), or other memory system.
  • a program for realizing the functions of the embodiments of the present invention can be recorded on a computer readable recording medium.
  • the corresponding functions can be realized by causing a computer system to read programs recorded on the recording medium and execute the programs.
  • the so-called "computer system” herein may be a computer system embedded in the device, and may include an operating system or hardware (such as a peripheral device).

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

提供了一种用于处理视频数据的方法。该方法包括：对图像序列进行时间下采样，形成多个子序列；对多个子序列分别进行编码，形成编码后的多个子序列；以及根据编码后的多个子序列中的各个帧的大小以及信道带宽，选择要传输的帧。还提供了一种无人机和计算机可读存储介质。

Description

无线数据传输方法和设备 技术领域
本公开涉及无线通信技术领域，更具体地，本公开涉及一种无线数据传输方法和设备。
背景技术
当前，在无线信道或者信道带宽发生实时变化时，如何稳定地传输视频，是研究和应用的热点话题。对于此类信道特征的视频传输来说，存在一定的问题。首先，信源随时间变化，而且信道也随时间变化。特别地，影响无线信道的条件很多，例如收发设备距离、相对位置、是否存在遮挡、即时电磁环境干扰等等。此外，信源和信道两者相互独立地变化，难以预测，这给信源编码和信道带宽的实时匹配造成困难。
低延时图传(尤其是无人机应用)要求每一帧的传输时间都能控制在一定范围内以避免大幅波动。通常，接收端的解码显示会发生卡顿现象。例如，
(1)当信道稳定时，如果摄像机突然出现运动，或者摄像机的取景框内的物体发生大幅运动，都可能导致编码得到的码流的大小有突然的变化。举例来说，假设码流增加两倍，则意味着传输延时会增大两倍。
(2)当信源稳定时，此时编码的码流大小平稳。如果信道突然发生变化，也会造成传输延时抖动。举例来说，假设信道的带宽减小两倍，则意味着传输延时会增大两倍。
目前，诸如H.263、H.264、H.265、MPEG4等视频编码标准得以广泛地使用。此外，通过码率控制算法来控制一段时间(若干帧)的平均码率，使其能够稳定在某个给定目标码率上，从而保证若干帧或一段时间的帧的平均总体延时抖动范围较小。
然而，现有方案仅仅旨在控制一个帧组的平均总体延时，无法解决由于动态的信源信道失配而导致的帧级别的延时抖动。
发明内容
为了解决以上问题中的至少一部分，本公开提出采用基于时间分解的方式对原始图像视频序列进行分解，然后对分解后的每个子序列分别进行编码传输和独立容错。
在本公开中，对于每一个时间分解后的子序列中的图像帧，根据每个子序列码流大小以及当时的信道条件，进行组合选择发送。接收端可以根据接收到的正确的码流进行解码重建。对于未收到或者出错的帧所在的图像，利用正确接收到的帧进行线性插值，由此获得最终的完整的重建图像。
根据本公开的一个方面，提供了一种用于处理视频数据的方法。该方法包括：对图像序列进行时间下采样，形成多个子序列；对多个子序列分别进行编码，形成编码后的多个子序列；以及根据编码后的多个子序列中的各个帧的大小以及信道带宽，选择要传输的帧。
在一个实施例中，从每一个编码后的子序列中找出时间上最早的帧，并将这些帧进行组合以形成一个组，依次重复该操作以形成多个组。然后，根据每个组中各个帧的大小以及信道带宽，选择每个组中要传输的帧。
在一个实施例中，可以根据以下条件来选择组中要传输的帧：
S≤T
其中，S表示组中所选择的帧的总码流大小，T表示信道带宽。
在另一个实施例中，可以根据以下条件来选择组中要传输的帧：
T-D≤S≤T+D
其中，S表示组中所选择的帧的总码流大小，T表示信道带宽，D表示容忍门限。
优选地，选择每个组中尽可能多的数量的帧进行传输。
根据本公开的另一个方面，提供了一种无人机，包括成像装置、处理器和传输单元。成像装置被配置为捕获图像序列。处理器被配置为对图像序列进行时间下采样，形成多个子序列。处理器还被配置为对多个子序列分别进行编码，形成编码后的多个子序列。处理器还被配置为根据编码后的多个子序列中的各个帧的大小以及信道带宽，选择要传输的帧。传输单元被配置为传输所选择的帧。
根据本公开的另一个方面，提供了一种计算机可读存储介质，存储有计算机程序。当计算机程序由至少一个处理器运行时，使至少一个处理器执行以下操作：对图像序列进行时间下采样，形成多个子序列；对多个子 序列分别进行编码，形成编码后的多个子序列；以及根据编码后的多个子序列中的各个帧的大小以及信道带宽，选择要传输的帧。
采用本公开的技术方案，能够在帧级别上减少由于实时信源信道失配所造成的延时抖动(即传输时间的不稳定)。
附图说明
通过下文结合附图的详细描述，本公开的上述和其它特征将会变得更加明显，其中：
图1是示出了根据本公开一个实施例的方法的流程图。
图2是示出了根据本公开一个实施例的对图像序列进行时间下采样的示意图。
图3是示出了根据本公开一个实施例的无人机的框图。
图4是示出了根据本公开一个实施例的计算机可读存储介质的框图。
具体实施方式
下面结合附图和具体实施方式对本公开进行详细阐述。应当注意，本公开不应局限于下文所述的具体实施方式。另外，为了简便起见，省略了对与本公开没有直接关联的公知技术的详细描述，以防止对本公开的理解造成混淆。
图1是示出了根据本公开一个实施例的方法10的流程图。
如图1所示，在步骤S110，对包括多个帧的图像序列进行时间下采样，形成多个子序列。
图2是示出了根据本公开一个实施例的对图像序列进行时间下采样的示意图。在图2所示的示例中，将原始图像序列(P0，P1，...P7，...)分解为4个子序列。其中，第一个子序列包含帧P0、P4、P8...，第二个子序列包含帧P1、P5、P9...，第三个子序列包含帧P2、P6、P10...，第四个子序列包含帧P3、P7、P11...。由此，获得了四个时间分辨率为原始视频序列的1/4的子视频序列。
需要说明的是，图2中所示的4个子序列仅仅是一个具体示例。本公开不限于将原始图像序列分为4个子序列，而是可以根据实际需求划分为更多或更少数量的子序列。例如，可以将原始图像序列(P0，P1，......) 分解为6个子序列。其中，第一个子序列包含帧P0、P6、P12...，第二个子序列包含帧P1、P7、P13...，第三个子序列包含帧P2、P8、P14...，第四个子序列包含帧P3、P9、P15...，第五个子序列包含帧P4、P10、P16...，第六个子序列包含帧P5、P11、P17...。由此，获得了六个时间分辨率为原始视频序列的1/6的子视频序列。同理，也可以将原始图像序列(P0，P1，......)分解为2个子序列，在此情况下第一个子序列包含帧P0、P2、P4...，第二个子序列包含帧P1、P3、P5...。
回到图1，在步骤S120，对步骤S110处获得的多个子序列分别进行编码(即压缩)，形成编码后的多个子序列(编码后的码流)。例如，针对每一个子序列，可以采用相应的一个编码器进行编码。对所采用的多个编码器的输出进行汇总，形成编码后的码流。编码后的码流包括编码后的各个帧。
接下来，在步骤S130，根据编码后的多个子序列中的各个帧(经过编码的帧)的大小以及信道带宽，选择要传输的经过编码的帧。
根据一个实施例，在选择要传输的帧时，考虑以组(即图2中所示的G0、G1…)为单位进行传输。仍以图2为例，可以根据每个组(G0、G1…)中的每个子序列当前帧码流的大小以及实时的信道估计值，对该组中的四个子序列的当前帧码流进行组合发送，以实现与无线信道的实时匹配。
下面，通过一个具体示例详细描述如何根据帧大小以及信道带宽来选择要传输的帧。
假定组G0中的四个编码后的帧P0、P1、P2和P3的码流大小分别为S0、S1、S2、S3。另外，假定当前信道带宽的估计值(即当前组G0时刻的可发送数据量)为T。T的值可以是预先定义的(例如根据历史值来得到)，也可以是采用信道带宽估计器通过计算得到的。进一步地，假定当前四个子序列的收发状态均无差错。那么，
(1)如果S0+S1+S2+S3＜＝T，或者场景对于延时无要求，则可以完整地发送包括四个编码后的帧P0、P1、P2和P3的码流。
(2)否则，从S0、S1、S2、S3中进行选择，使得组合后的码流总大小最接近T。优选地，在保持组合后的码流总大小最接近T的前提下，选择包含尽可能多的个数的编码后的帧的码流。
例如，在此场景下，如果满足S0+S1＜S0+S2＜T，则选择发送包括编 码后的帧P0和P2的码流。备选地，如果满足S0+S1＜T，S0+S2+S3＜T，而且S0+S1的大小与S0+S2+S3的大小相当，则优选发送包括编码后的帧P0、P2和P3的码流。
(3)对于延时要求严格的应用场景，组合后的数据量大小应小于T。而对于延时抖动有一定容忍能力的应用场景，则以组合后的码流的数据量大小满足T-D＜＝S＜＝T+D为条件来选择要传输的编码后的帧。其中，D是容忍门限，而S是所选择的编码后的帧的总大小。优选地，在满足该条件的前提下，选择包含尽可能多的个数的编码后的帧的码流。
在接收端，可以同样以组为单位来接收组中的每一个子序列的码流。例如，当组G0中的帧P0、P1、P2、P3中有一个或者多个被接收正确时，可以使用正确接收的子序列图像恢复其所在时间位置的原始图像，而不适用发生错误的子序列。相反，对于发生错误的子序列，可以使用正确接收的重建序列通过线性加权内插的方式恢复其对应时间位置的原始图像，由此产生最终的重建图像序列。
根据本实施例，即使已发送的某帧图像中有任何一个数据块发生错误，也可以通过其他已正确接收的帧图像进行时间维度线性内插，从而获得其所在时间位置的重建图像。这样，能够在帧级别上减少由于实时信源信道失配所造成的延时抖动，提高了容错率。
本公开的技术方案可以应用于无人机中。图3是示出了根据本公开一个实施例的无人机30的框图。如图3所示，无人机30包括成像装置310、处理器320以及传输单元330。
成像装置310被配置为捕获包括多个帧的图像序列。例如，成像装置310可以包括分布在无人机上的一个或更多个摄像机。
处理器320被配置为对成像装置310所捕获的包括多个帧的图像序列执行操作。具体地，处理器320对所捕获的包括多个帧的图像序列进行时间下采样，以形成多个子序列。处理器320还对所形成的多个子序列分别进行编码，形成编码后的多个子序列。此外，处理器320还根据编码后的多个子序列中的各个编码后的帧的大小以及当前信道带宽的估计值，选择要传输的编码后的帧。
例如，处理器320可以从每一个编码后的子序列中找出时间上最早的 帧，并将这些编码后的帧进行组合以形成一个组。处理器320依次重复该操作以形成多个组。并且，处理器320根据每个组中各个编码后的帧的大小以及当前信道带宽的估计值，选择每个组中要传输的编码后的帧。
例如，处理器320可以根据以下条件来选择组中要传输的编码后的帧：
S≤T
其中，S表示组中所选择的编码后的帧的总码流大小，T表示信道带宽。优选地，处理器320选择每个组中尽可能多的数量的编码后的帧进行传输。
备选地，处理器320可以根据以下条件来选择组中要传输的编码后的帧：
T-D≤S≤T+D
其中，S表示组中所选择的编码后的帧的总码流大小，T表示信道带宽，D表示容忍门限。优选地，处理器320选择每个组中尽可能多的数量的编码后的帧进行传输。
传输单元330被配置为传输由处理器320所选择的帧。例如，传输单元330可以包括采用多种无线通信技术(例如蜂窝通信、蓝牙、WiFi等)的无线通信模块。
采用根据本公开的实施例的无人机，在执行图像传输任务时，能够在帧级别上减少由于实时信源信道失配所造成的延时抖动，从而提高了容错率。
此外，本公开的实施例可以借助于计算机程序产品来实现。例如，该计算机程序产品可以是计算机可读存储介质。计算机可读存储介质上存储有计算机程序，当在计算设备上执行该计算机程序时，能够执行相关的操作以实现本公开的上述技术方案。
例如，图4是示出了根据本公开一个实施例的计算机可读存储介质40的框图。如图4所示，计算机可读存储介质40包括程序410。该程序410在由至少一个处理器运行时，使得至少一个处理器执行以下操作：对包括多个帧的图像序列进行时间下采样，形成多个子序列；对多个子序列分别进行编码，形成编码后的多个子序列；以及根据编码后的多个子序列中的各个编码后的帧的大小以及当前信道带宽的估计值，选择要传输的编码后的帧。
备选地，当程序410由至少一个处理器运行时，使得至少一个处理器 执行以下操作：从每一个编码后的子序列中找出时间上最早的帧，并将这些编码后的帧进行组合以形成一个组；依次重复该操作以形成多个组；以及根据每个组中各个编码后的帧的大小以及当前信道带宽的估计值，选择每个组中要传输的编码后的帧。
本领域技术人员可以理解，本公开的实施例中的计算机可读存储介质40的示例包括但不限于：半导体存储介质、光学存储介质、磁性存储介质、或任何其他的计算机可读存储介质。
上文已经结合优选实施例对本公开的方法和涉及的设备进行了描述。本领域技术人员可以理解，上面示出的方法仅是示例性的。本公开的方法并不局限于上面示出的步骤和顺序。
应该理解，本公开的上述实施例可以通过软件、硬件或者软件和硬件两者的结合来实现。本公开的这种设置典型地提供为设置或编码在例如光介质(例如CD-ROM)、软盘或硬盘等的计算机可读介质上的软件、代码和/或其他数据结构、或者诸如一个或多个ROM或RAM或PROM芯片上的固件或微代码的其他介质、或一个或多个模块中的可下载的软件图像、共享数据库等。软件或固件或这种配置可安装在计算设备上，以使得计算设备中的一个或多个处理器执行本公开实施例所描述的技术方案。
此外，上述每个实施例中所使用的设备的每个功能模块或各个特征可以由电路实现或执行，所述电路通常为一个或多个集成电路。设计用于执行本说明书中所描述的各个功能的电路可以包括通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)或通用集成电路、现场可编程门阵列(FPGA)或其他可编程逻辑器件、分立的门或晶体管逻辑、或分立的硬件组件、或以上器件的任意组合。通用处理器可以是微处理器，或者所述处理器可以是现有的处理器、控制器、微控制器或状态机。上述通用处理器或每个电路可以由数字电路配置，或者可以由逻辑电路配置。此外，当由于半导体技术的进步，出现了能够替代目前的集成电路的先进技术时，本公开也可以使用利用该先进技术得到的集成电路。
运行在根据本发明的设备上的程序可以是通过控制中央处理单元(CPU)来使计算机实现本发明的实施例功能的程序。该程序或由该程序处理的信息可以临时存储在易失性存储器(如随机存取存储器RAM)、硬盘驱动器(HDD)、非易失性存储器(如闪速存储器)、或其他存储器系统 中。用于实现本发明各实施例功能的程序可以记录在计算机可读记录介质上。可以通过使计算机系统读取记录在所述记录介质上的程序并执行这些程序来实现相应的功能。此处的所谓“计算机系统”可以是嵌入在该设备中的计算机系统，可以包括操作系统或硬件(如外围设备)。
如上，已经参考附图对本发明的实施例进行了详细描述。但是，具体的结构并不局限于上述实施例，本发明也包括不偏离本发明主旨的任何设计改动。另外，可以在权利要求的范围内对本发明进行多种改动，通过适当地组合不同实施例所公开的技术手段所得到的实施例也包含在本发明的技术范围内。此外，上述实施例中所描述的具有相同效果的组件可以相互替代。

Claims (15)

  1. 一种用于处理视频数据的方法，包括：
    对图像序列进行时间下采样，形成多个子序列；
    对多个子序列分别进行编码，形成编码后的多个子序列；以及
    根据编码后的多个子序列中的各个帧的大小以及信道带宽，选择要传输的帧。
  2. 根据权利要求1所述的方法，其中，选择要传输的帧包括：
    从每一个编码后的子序列中找出时间上最早的帧，并将这些帧进行组合以形成一个组；
    依次重复该操作以形成多个组；以及
    根据每个组中各个帧的大小以及信道带宽，选择每个组中要传输的帧。
  3. 根据权利要求2所述的方法，其中，根据以下条件来选择组中要传输的帧：
    S≤T
    其中，S表示组中所选择的帧的总码流大小，T表示信道带宽。
  4. 根据权利要求2所述的方法，其中，根据以下条件来选择组中要传输的帧：
    T-D≤S≤T+D
    其中，S表示组中所选择的帧的总码流大小，T表示信道带宽，D表示容忍门限。
  5. 根据权利要求3或4所述的方法，其中，选择每个组中尽可能多的数量的帧进行传输。
  6. 一种无人机，包括：
    成像装置，被配置为捕获图像序列；
    处理器，被配置为：
    对图像序列进行时间下采样，形成多个子序列；
    对多个子序列分别进行编码，形成编码后的多个子序列；以及
    根据编码后的多个子序列中的各个帧的大小以及信道带宽，选择要传输的帧；以及
    传输单元，被配置为传输所选择的帧。
  7. 根据权利要求6所述的无人机，其中，所述处理器被配置为：
    从每一个编码后的子序列中找出时间上最早的帧，并将这些帧进行组合以形成一个组；
    依次重复该操作以形成多个组；以及
    根据每个组中各个帧的大小以及信道带宽，选择每个组中要传输的帧。
  8. 根据权利要求7所述的无人机，其中，所述处理器被配置为根据以下条件来选择组中要传输的帧：
    S≤T
    其中，S表示组中所选择的帧的总码流大小，T表示信道带宽。
  9. 根据权利要求7所述的无人机，其中，所述处理器被配置为根据以下条件来选择组中要传输的帧：
    T-D≤S≤T+D
    其中，S表示组中所选择的帧的总码流大小，T表示信道带宽，D表示容忍门限。
  10. 根据权利要求8或9所述的无人机，其中，所述处理器被配置为：选择每个组中尽可能多的数量的帧进行传输。
  11. 一种计算机可读存储介质，存储有计算机程序，当所述计算机程序由至少一个处理器运行时，使所述至少一个处理器执行以下操作：
    对图像序列进行时间下采样，形成多个子序列；
    对多个子序列分别进行编码，形成编码后的多个子序列；以及
    根据编码后的多个子序列中的各个帧的大小以及信道带宽，选择要传输的帧。
  12. 根据权利要求11所述的计算机可读存储介质，其中，所述计算机程序使所述至少一个处理器通过以下操作来选择要传输的帧：
    从每一个编码后的子序列中找出时间上最早的帧，并将这些帧进行组合以形成一个组；
    依次重复该操作以形成多个组；以及
    根据每个组中各个帧的大小以及信道带宽，选择每个组中要传输的帧。
  13. 根据权利要求12所述的计算机可读存储介质，其中，所述计算机程序使所述至少一个处理器根据以下条件来选择组中要传输的帧：
    S≤T
    其中，S表示组中所选择的帧的总码流大小，T表示信道带宽。
  14. 根据权利要求12所述的计算机可读存储介质，其中，所述计算机程序使所述至少一个处理器根据以下条件来选择组中要传输的帧：
    T-D≤S≤T+D
    其中，S表示组中所选择的帧的总码流大小，T表示信道带宽，D表示容忍门限。
  15. 根据权利要求13或14所述的计算机可读存储介质，其中，所述计算机程序使所述至少一个处理器选择每个组中尽可能多的数量的帧进行传输。
PCT/CN2017/100701 2017-09-06 2017-09-06 无线数据传输方法和设备 WO2019047059A1 (zh)

Priority Applications (5)

Application Number Priority Date Filing Date Title
EP17924248.2A EP3681156B1 (en) 2017-09-06 2017-09-06 Method and device for transmitting wireless data
CN201780006922.5A CN108521870B (zh) 2017-09-06 2017-09-06 无线数据传输方法和设备
PCT/CN2017/100701 WO2019047059A1 (zh) 2017-09-06 2017-09-06 无线数据传输方法和设备
CN202011468184.6A CN112672152A (zh) 2017-09-06 2017-09-06 无线数据传输方法和设备
US16/727,428 US20200177913A1 (en) 2017-09-06 2019-12-26 Method and device for transmitting wireless data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/100701 WO2019047059A1 (zh) 2017-09-06 2017-09-06 无线数据传输方法和设备

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/727,428 Continuation US20200177913A1 (en) 2017-09-06 2019-12-26 Method and device for transmitting wireless data

Publications (1)

Publication Number Publication Date
WO2019047059A1 true WO2019047059A1 (zh) 2019-03-14

Family

ID=63433139

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/100701 WO2019047059A1 (zh) 2017-09-06 2017-09-06 无线数据传输方法和设备

Country Status (4)

Country Link
US (1) US20200177913A1 (zh)
EP (1) EP3681156B1 (zh)
CN (2) CN112672152A (zh)
WO (1) WO2019047059A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102019002789B4 (de) * 2019-04-16 2022-01-05 Daimler Ag Verfahren und Vorrichtung zur Steuerung eines automatisiert fahrenden Fahrzeuges
CN113949825A (zh) * 2021-11-03 2022-01-18 江苏金视传奇科技有限公司 一种高信息量图像高效传输方法及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101252686A (zh) * 2008-03-20 2008-08-27 上海交通大学 基于交织预测的视频帧内无损编解码方法及系统
CN101895776A (zh) * 2009-04-13 2010-11-24 三星电子株式会社 传输立体图像数据的装置和方法
CN102204257A (zh) * 2008-09-18 2011-09-28 玛格通讯有限公司 低延迟视频编码器
US20160198166A1 (en) * 2015-01-07 2016-07-07 Texas Instruments Incorporated Multi-pass video encoding

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7450646B2 (en) * 2002-06-04 2008-11-11 Panasonic Corporation Image data transmitting apparatus and method and image data reproducing apparatus and method
US7860005B2 (en) * 2004-01-30 2010-12-28 Hewlett-Packard Development Company, L.P. Methods and systems that use information about a frame of video data to make a decision about sending the frame
GB0428155D0 (en) * 2004-12-22 2005-01-26 British Telecomm Buffer underflow prevention
US8296813B2 (en) * 2006-06-22 2012-10-23 Sony Computer Entertainment Inc. Predictive frame dropping to enhance quality of service in streaming data
US7898950B2 (en) * 2006-08-18 2011-03-01 Microsoft Corporation Techniques to perform rate matching for multimedia conference calls
US8897322B1 (en) * 2007-09-20 2014-11-25 Sprint Communications Company L.P. Enhancing video quality for broadcast video services
CN101626512A (zh) * 2009-08-11 2010-01-13 北京交通大学 一种基于相关性优化准则的多描述视频编码方法及装置
US20110268045A1 (en) * 2010-04-30 2011-11-03 Youn Hyoung Heo System and method for uplink control information transmission in carrier aggregation
US9049464B2 (en) * 2011-06-07 2015-06-02 Qualcomm Incorporated Multiple description coding with plural combined diversity
US9001679B2 (en) * 2011-11-07 2015-04-07 Qualcomm Incorporated Supporting voice for flexible bandwidth systems
CN103297202B (zh) * 2012-02-29 2016-10-05 华为技术有限公司 信道状态信息的处理方法及装置
US9313495B2 (en) * 2012-05-14 2016-04-12 Luca Rossato Encoding and decoding based on blending of sequences of samples along time
CN103533330B (zh) * 2013-10-15 2016-01-06 华侨大学 一种基于数据重用的多视点多描述视频编码方法
KR102349450B1 (ko) * 2014-12-08 2022-01-10 삼성전자주식회사 무결성 검사 데이터 제공 방법 및 장치
CN105120230B (zh) * 2015-09-15 2018-08-24 成都时代星光科技有限公司 无人机图像监控和传输系统
WO2017128314A1 (zh) * 2016-01-29 2017-08-03 深圳市大疆创新科技有限公司 视频数据传输方法、系统、设备和拍摄装置
CN106411838A (zh) * 2016-06-14 2017-02-15 青岛乾元通数码科技有限公司 一种多信道负载均衡音视频传输方法及系统
WO2018132964A1 (zh) * 2017-01-18 2018-07-26 深圳市大疆创新科技有限公司 传输编码数据的方法、装置、计算机系统和移动设备
US10379545B2 (en) * 2017-07-03 2019-08-13 Skydio, Inc. Detecting optical discrepancies in captured images

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101252686A (zh) * 2008-03-20 2008-08-27 上海交通大学 基于交织预测的视频帧内无损编解码方法及系统
CN102204257A (zh) * 2008-09-18 2011-09-28 玛格通讯有限公司 低延迟视频编码器
CN101895776A (zh) * 2009-04-13 2010-11-24 三星电子株式会社 传输立体图像数据的装置和方法
US20160198166A1 (en) * 2015-01-07 2016-07-07 Texas Instruments Incorporated Multi-pass video encoding

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3681156A4 *

Also Published As

Publication number Publication date
CN112672152A (zh) 2021-04-16
EP3681156A4 (en) 2020-07-22
EP3681156A1 (en) 2020-07-15
EP3681156B1 (en) 2021-07-14
US20200177913A1 (en) 2020-06-04
CN108521870B (zh) 2021-01-01
CN108521870A (zh) 2018-09-11

Similar Documents

Publication Publication Date Title
US10705709B2 (en) Playing media
JP5860067B2 (ja) 符号化方法及び装置並びに復号化方法及び装置
CN108141581B (zh) 视频编码
US10051280B2 (en) Video encoding/decoding system and diagnosis method thereof
US9241172B2 (en) Moving image encoding and decoding device
US20200099968A1 (en) Method and device for transmitting wireless data
US20230042575A1 (en) Methods and systems for estimating motion in multimedia pictures
WO2019047059A1 (zh) 无线数据传输方法和设备
Bao et al. MDVSC--Wireless Model Division Video Semantic Communication
US9648336B2 (en) Encoding apparatus and method
US10666970B2 (en) Encoding apparatus, encoding method, and storage medium
CN116916066A (zh) 一种视频转码方法及装置、电子设备及存储介质
CN105474644A (zh) 帧处理与重现方法
US10992946B2 (en) Coding of video and audio with initialization fragments
JP2009111625A (ja) 映像符号化装置及び映像符号化方法
US20130339482A1 (en) Data transmitting system, and transmitting apparatus and receiving apparatus and program in data transmitting system
CN106937168B (zh) 一种利用长期参考帧的视频编码方法、电子设备及系统
US10721473B2 (en) Systems and methods for adjusting the frame rate of transmitted video based on the level of motion in the video
CN111988621A (zh) 视频处理器训练方法、装置、视频处理装置及视频处理方法
US9451288B2 (en) Inferred key frames for fast initiation of video coding sessions
WO2021093059A1 (zh) 一种感兴趣区域的识别方法、系统及设备
CN114257818A (zh) 视频的编、解码方法、装置、设备和存储介质
WO2023185305A1 (zh) 编码方法、装置、存储介质及计算机程序产品
WO2023053166A1 (ja) 映像処理システム、情報処理装置、映像処理方法、及び記録媒体
US20130287100A1 (en) Mechanism for facilitating cost-efficient and low-latency encoding of video streams

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17924248

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017924248

Country of ref document: EP

Effective date: 20200406