CN107948546B

CN107948546B - Low-delay video mixing device

Info

Publication number: CN107948546B
Application number: CN201711096984.8A
Authority: CN
Inventors: 郭凡; 李少光; 康健斌; 廖科; 周坚锋
Original assignee: China Aeronautical Radio Electronics Research Institute
Current assignee: China Aeronautical Radio Electronics Research Institute
Priority date: 2017-11-09
Filing date: 2017-11-09
Publication date: 2020-07-31
Anticipated expiration: 2037-11-09
Also published as: CN107948546A

Abstract

The invention discloses a low-delay video mixing device, which comprises a video input interface, a plurality of low-delay video processing synchronization units, a video cache module and a video mixing processing unit, wherein the video input interface selects one path of input video as a main video, other videos are used as slave videos, and received synchronous time sequence signals of the main video are respectively input into each low-delay video processing synchronization unit; each low-delay video processing synchronization unit realizes the synchronization of the slave video and the master video through the reading and writing of the slave video in the video cache module according to the synchronization timing signal; the video mixing processing unit fuses and outputs the main video and the synchronized slave video; the video buffer module allocates no more than 2 frame memories to each low-delay video processing synchronization unit. The invention can reduce the video delay to one half of the traditional method at least by a video mixing scheme of 2-frame buffer and video rate prediction on the video.

Description

Low-delay video mixing device

Technical Field

The invention relates to a processing circuit in the field of video processing and microelectronics, in particular to a device for mixing and processing multiple videos.

Background

In the video processing, multiple paths of videos are often involved to perform mixing processing, such as mixing and presenting an operation interface and a camera video to a user for convenient operation and observation. When a programmable device is used for carrying out multi-channel video mixing processing, the traditional method is to output input video acquisition integrated image data after three-frame caching method buffering, scaling and mixing processing. The method prohibits any one frame in the three-frame buffer from being stored and read simultaneously, can effectively ensure the integrity of the video and is well suitable for the asynchronous mixing of multiple input videos, but the method has obvious defects, large delay of video mixing and more consumed storage resources. The invention provides a video processing method with lower time delay and less consumed storage resources, which is particularly suitable for application occasions where multi-channel videos cannot be synchronized and are sensitive to time delay.

Disclosure of Invention

The invention aims to provide a low-delay video mixing device, which can reduce the video delay to one half of the traditional method at least by a video mixing scheme of carrying out 2-frame buffering and video rate prediction on video.

The invention aims to be realized by the following technical scheme:

a low-delay video mixing device comprises a video input interface, a plurality of low-delay video processing synchronization units, a video cache module and a video mixing processing unit;

the video input interface selects one path of input video as a main video and other videos as slave videos, and respectively inputs the received synchronous time sequence signals of the main video into each low-delay video processing synchronization unit;

each low-delay video processing synchronization unit realizes the synchronization of the slave video and the master video through the reading and writing of the slave video in the video cache module according to the synchronization timing signal;

the video mixing processing unit fuses and outputs the main video and the synchronized slave video;

the video buffer module allocates no more than 2 frame memories to each low-delay video processing synchronization unit.

Preferably, the low-delay video processing synchronization unit comprises a video phase tracking prediction module, an access conflict prediction and cache selection module, a video acquisition module, a video data preprocessing module, a frame memory access controller module, a video data processing module and a video synchronization output module;

the video phase tracking prediction module is used for detecting the speed difference and the phase difference of the slave video and the synchronous time sequence signal;

the access conflict prediction and cache selection module is used for controlling a read pointer and a write pointer of the frame memory access controller module to move on the same frame memory when the rates of the slave video and the synchronous time sequence signal are consistent; when the rate of the slave video is faster than the rate of the synchronous time sequence signal, controlling a write-read pointer and a read pointer of a frame memory access controller module to alternately move on two frame memories, judging whether the write pointer has access conflict or not when the write pointer and the read pointer are on the same frame memory, and if so, not moving the write pointer; when the rate of the slave video is slower than the rate of the synchronous time sequence signal, controlling a write-read pointer and a read pointer of a frame memory access controller module to alternately move on two frame memories, judging whether the read pointer has access or not when the write pointer and the read pointer are on the same frame memory, and if so, not moving the read pointer;

the video acquisition module writes the video into the frame memory according to the position of a writing pointer on the frame memory access controller module;

the video synchronous output module reads out the video from the frame memory by taking the phase difference as the delay from the position of the reading pointer on the frame memory access controller module.

Drawings

FIG. 1 is a schematic diagram of a low-delay video mixing apparatus;

FIG. 2 is a schematic diagram of a low-latency video processing synchronization unit;

FIG. 3 is a diagram illustrating the video transmission rate and the phase difference between the input video and the primary synchronization video;

FIG. 4 is a flowchart illustrating a process of the access conflict prediction and cache selection module.

Detailed Description

The invention is described in further detail below with reference to the figures and examples.

The present embodiment will be described by taking a low-delay video mixing apparatus composed of an FPGA chip and a memory as an example. A video input interface, a plurality of low-delay video processing synchronization units and a video mixing processing unit are arranged in an FPGA chip, and a memory is used as a video cache module.

The video input interface selects one path of video with the most strict delay requirement from the input multiple paths of videos as a main video, and the rest paths of videos are used as slave videos. The video input interface sends the speed of the main video as a synchronous time sequence signal to the low-delay video processing synchronization unit, and the low-delay video processing synchronization unit synchronizes the auxiliary video with the main video through reading and writing in the video cache module according to the synchronous time sequence signal and then completes mixing of all videos in the video mixing processing unit and outputs the mixed videos. The low-delay video processing synchronization unit has the characteristic that data writing and reading are sequentially carried out during video caching processing, buffer management control is carried out through video phase tracking prediction, and when the fact that data writing and reading of the same frame cache are not conflicted is predicted, the unit controls the frame cache being written to be directly read out for processing.

As shown in fig. 2, the low-latency video processing synchronization unit includes a video phase tracking prediction module, an access conflict prediction and cache selection module, a video acquisition module, a video data preprocessing module, a frame access controller module, a video data processing module, and a video synchronization output module.

The video phase tracking prediction module is used for continuously detecting the speed difference and the phase difference of the slave video signal and the synchronous video signal. The input signals of the video phase tracking prediction module comprise slave video signals (din [23:0], hs _ in, vs _ in, pclk _ in and de _ in), synchronous timing signals (sync _ hs, sync _ vs, sync _ de and sync _ pclk), and a video phase tracking component clock (mclk). The video transmission rate and the phase difference between the input video and the master synchronization video are shown in fig. 3.

Where Vnum _ i is the period of vs _ in, Vnum _ sync is the period of sync _ vs, and the unit is the video phase prediction tracking component clock (mclk), the larger the values of Vnum _ i and Vnum _ o, the slower the rate, and the smaller the values, the faster the rate. D _ io _ num is the time difference of the field period between videos, and D _ io _ num is that the main video is fast and the auxiliary video is slow; d _ io _ num is negative and indicates that the master video is slow and the slave video is fast. A larger absolute value of D _ io _ num indicates a larger rate difference between the slave video and the master video. Dnum represents the phase difference between the input video and the main synchronization video, Dnum is positive and represents that the phase of the input video lags behind the main synchronization video, and Dnum is negative and represents that the phase of the input video leads the main synchronization video, and the larger the absolute value is, the larger the phase difference is.

And the access conflict prediction and cache selection module continuously predicts whether the collected video writing operation and the output video reading operation are conflicted on the same frame buffer area, and selects the frame buffer area which can control the total delay to be minimum and does not generate the conflict to operate according to the prediction result when one frame collection or one frame output is finished. And comprehensively judging the access conflict according to the speed, the phase and the processing delay of the video signal.

The access conflict prediction and cache selection processing flow is shown in fig. 4:

when the transmission rate of the slave video is the same as that of the master video, the phase difference between the slave video and the output video is a fixed value, the frame memory access controller module only uses one frame buffer memory, the delayed output of the input video by the output video is realized, and the delay time is the phase difference between the slave video and the master video.

When the transmission rate of the slave video is high, the frame memory access controller module uses double-frame cache, and writes the slave video data and reads the output video data respectively in a mode of alternately accessing the frame cache. Wherein the reading of the output video data alternately reads the two frame buffers by adopting a fixed ping-pong operation. At the beginning of an input video frame, judging whether a frame buffer to be used by ping-pong operation is a frame buffer currently used by an output video or not, calculating a phase difference between a clock cycle number C _ in for completing writing in the current frame and a clock cycle number C _ out for completing outputting the current frame according to the speed, the phase difference and the current processing delay of a slave video and a main video, when the C _ in is less than the C _ out, indicating that an access conflict can occur as a prediction result, not switching, directly selecting a writing frame buffer used in the previous field period, wherein the total delay of the video content being input at the moment reaches the maximum and is about the transmission time of one frame of video, and starting from the next frame, the delay is adjusted to the lowest delay state.

When the transmission rate of the main video is high, the frame memory access controller module uses double-frame cache, and respectively writes input video data and reads output video data in a mode of alternately accessing the frame cache. Wherein writing input video data alternately reads two frame buffers using a fixed ping-pong operation. When the output video data is read, at the beginning of the output video frame, judging whether a frame buffer used for ping-pong operation is a frame buffer currently used for video input or not, if so, calculating the phase difference between the number of clock cycles C _ in for completing writing in the current frame and the number of clock cycles C _ out for completing outputting the current frame according to the speed, the phase difference and the current processing delay of the input video and the synchronous video, and when C _ in is larger than C _ out, indicating that an access conflict can occur in a prediction result, repeatedly reading the original frame buffer by the output video to ensure that the output video data is the same frame video data. At this time, the maximum delay between the output video and the input video is about one frame of video transmission time, and from the next frame, the delay is adjusted to the lowest delay state.

Video capture is used to convert a video stream into sequential memory access operations. The video data preprocessing is used in occasions with reduction application, and reduction preprocessing is carried out, so that the risk of reducing potential bandwidth shortage during output is avoided. The video data processing is used for occasions with application such as enhancement, amplification and the like, and data are read from the frame buffer, video data processing is carried out on the data, and then synchronous videos are output.

Claims

1. A low-delay video mixing device comprises a video input interface, a plurality of low-delay video processing synchronization units, a video buffer module and a video mixing processing unit, and is characterized in that:

the video cache module allocates no more than 2 frame memories to each low-delay video processing synchronization unit;

the low-delay video processing synchronization unit comprises a video phase tracking prediction module, an access conflict prediction and cache selection module, a video acquisition module, a video data preprocessing module, a frame memory access controller module, a video data processing module and a video synchronization output module;

the access conflict prediction and cache selection module is used for controlling a read pointer and a write pointer of the frame memory access controller module to move on the same frame memory when the rates of the slave video and the synchronous time sequence signal are consistent; when the rate of the slave video is faster than the rate of the synchronous time sequence signal, controlling a write-read pointer and a read pointer of a frame memory access controller module to alternately move on two frame memories, judging whether the write pointer has access conflict or not when the write pointer and the read pointer are on the same frame memory, and if so, not moving the write pointer; when the rate of the slave video is slower than the rate of the synchronous time sequence signal, controlling a write-read pointer and a read pointer of a frame memory access controller module to alternately move on two frame memories, judging whether the read pointer has access conflict or not when the write pointer and the read pointer are on the same frame memory, and if so, not moving the read pointer;

and the video synchronous output module reads the video from the frame memory by taking the phase difference as the delay from the position of the reading pointer on the frame memory access controller module.