JP7521604B2 - Apparatus, method and program for synthesizing video signals - Google Patents

Description

This relates to a video synthesis system that synthesizes multiple video signals into a single screen and outputs it.

In recent years, many video devices have come into use. The images produced by these many video devices use a wide variety of pixel counts (resolutions) and frame rates. Although the video signals of these video devices differ in terms of physical signals, control signals, etc. depending on the standard, one screen is transmitted in a time period equal to one half of the frame rate.

One way to use these images is to display multiple cameras on fewer monitors than the number of cameras, such as in video conferencing. In such cases, multiple images are composited, for example by splitting them onto one screen, or by fitting one image onto another image in a reduced size.

Normally, the timing of a video signal is not synchronized, and since the timing of other video signals to be mixed is different, the signals are temporarily buffered in memory before being mixed. As a result, there is a delay in the output of the mixed screen.

If we imagine a video conference in which an ensemble playing music in a remote location is held using this type of screen composition, the delay involved in this composition would greatly impair its feasibility. For example, in a song with 120 beats per second (hereafter referred to as 120 BPM (beats per minute)), the duration of one beat is 60/120 seconds = 500 milliseconds. If we need to match this with an accuracy of 5%, then the delay between capturing the image with a camera and displaying it needs to be reduced to 500 x 0.05 = 25 milliseconds or less.

In reality, the process from capturing an image with a camera to displaying it requires other delays besides the process related to composition, such as the time it takes to process the image on the camera, the time it takes to display it on the monitor, and the time it takes to transmit it. As a result, with conventional technology, it has been difficult to collaborate in applications where timing is important, such as playing an ensemble while watching each other's videos from remote locations.

Therefore, for collaborative work that requires strict low latency, it is necessary to provide a system that synthesizes multiple video signals from multiple locations, etc., and that reduces the latency between the input of asynchronous video signals and the output of the synthesized video signal.

VESA and Industry Standards and Guidelines for Computer Display Monitor Timing (DMT), Version 1.0, Rev. 13, February 8, 2013

The purpose of this disclosure is to reduce the delay time until the composite image is output.

The device disclosed herein is a device that synthesizes and displays multiple asynchronous video signals, and selects and synthesizes a combination of video signals from each of the multiple input video signals that minimizes the delay time of the output video.

The image synthesizing device and image synthesizing method disclosed herein include
Detects the delay between input frames that compose multiple asynchronously input video signals,
selecting input frames of the plurality of video signals so as to minimize a delay time of an output frame obtained by combining the plurality of video signals;
A selected input frame is used to generate an output frame that combines the multiple video signals.

The image compositing method according to the present disclosure includes an image compositing device,
Detects the delay between input frames that compose multiple asynchronously input video signals,
selecting input frames of the plurality of video signals so as to minimize a delay time of an output frame obtained by combining the plurality of video signals;
A selected input frame is used to generate an output frame that combines the multiple video signals.

The program of the present disclosure is a program for causing a computer to realize each functional unit of the device of the present disclosure, and is a program for causing a computer to execute each step of the method performed by the device of the present disclosure.

This disclosure can reduce the delay time until the composite image is output.

3 shows an example of screen information included in a video signal. An example of screen composition is shown below. 1 illustrates an example of a video synthesis method related to the present disclosure. 1 illustrates an example of a video synthesis method according to the present disclosure. 1 illustrates an example of a video synthesis method according to the present disclosure. 1 shows an example of the configuration of an image synthesizing device according to an embodiment of the present invention. 1 illustrates an example of a video synthesis method according to the present disclosure. 1 illustrates an example of a video synthesis method according to the present disclosure.

Below, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings are considered to be identical to each other.

Figure 1 shows an example of screen information contained in a video signal. The screen information is transmitted by scanning the screen horizontally, one scan line 21 at a time, and then scanning the scan lines 21 below in sequence. This scanning includes scanning of the display screen 24, as well as blanking portion 22 and overhead information/signals such as border portion 23. The blanking portion 22 may contain information other than video information, such as control information and audio information. (For example, see Non-Patent Document 1, Chapter 3.)

Figure 2 shows an example of video signal synthesis. In this disclosure, as an example, four video signals, inputs 1 to 4, are input to a video synthesis device, which synthesizes them into one video signal and outputs it. A video signal transmits one screen over a period of time equal to one frame rate. For example, if the video signal has 60 frames per second, one screen of the video signal is transmitted over 1/60th of a second, or approximately 16.7 milliseconds (hereinafter, 60 fps (Frames per Second)). The information of one screen at each time point contained in the video signal is referred to as a "frame," the information of one screen of each video signal input to the video synthesis device is referred to as an "input frame," and the information of one synthesized screen output from the video synthesis device is referred to as an "output frame."

For example, consider a case where a video synthesizer reads all input frames, synthesizes them into one output frame, and outputs it, as shown in Figure 3. In this case, if the frame time of each input frame is T_f and the synthesis processing time is T_p, the output of the output frame will be delayed by a maximum of 2T_f + T_p from the time of input of the first input frame, Input 1.

The present disclosure relates to a system that inputs multiple asynchronous video images and synthesizes those images, characterized by selecting the input frames to be synthesized so as to minimize the delay after synthesis.

If the kth output frame is {O, k frame}, the transformation function from the input is expressed as f(input1, input2, ...). Also, inputs 1, 2, 3, and 4 are inputs in order of earliest input timing.

(First Synthesis Example)
4 shows a first synthesis example of the present disclosure. When {O,k}=f({1,k}, {2,k}, {3,k}, {4,k}), as shown in the figure, inputs {1,k}, {2,k}, {3,k} are input without delay time, and {4,k} has an input delay time D_in4 compared to the others. In this case, the delay time of the O,k frame is T_f+T_p for input 4, and T_f+T_p+D_in4 for the other inputs 1, 2, and 3. In this case, the average delay value of the four inputs is
(Equation 1)
T_f+T_p+3D_in4/4 (1)
It is.

(Second Synthesis Example)
FIG. 5 shows a second synthesis example of the present disclosure. In the case where {O,k}=f({1,k+1}, {2,k+1}, {3,k+1}, {4,k}), inputs {1,k+1}, {2,k+1}, {3,k+1} are input after (T_f-D_in4) from input {4,k}, and synthesis processing is performed at T_p immediately after inputs {1,k+1}, {2,k+1}, {3,k+1}. In this case, the delay time of O,k frames is T_f+T_p for inputs 1, 2, and 3, and 2T_f+T_p-D_in4 for input 4. In this case, the average delay value of the four inputs is
(Equation 2)
5T_f/4+T_p-D_in4/4 (2)
It is.

Here, if T_f<4D_in4, the average delay time is shorter in the synthesis example of formula (2) than in the synthesis example of formula (1). In this way, by changing the combination of positions (times) of the input frames that are the source of the output according to the amount of delay time of the input frames, a combination exists that minimizes the average value, and by constructing the output frame with this minimum combination, it is possible to minimize the synthesis delay.

That is, for (O, k), f({1, k}, {2, k}, {3, k}, {4, k})
f({1, k+1}, {2, k}, {3, k}, {4, k})
f ({1, k+1}, {2, k+1}, {3, k}, {4, k})
f ({1, k+1}, {2, k+1}, {3, k+1}, {4, k})
The delay time for each combination is calculated, and the combination of input frames that has the smallest average delay time is selected as the output combination.

The combination of input frames is not limited to the combination that minimizes the average delay time, but may be the combination that minimizes the maximum delay time. In addition, in cases where low latency is required only for some input frames, the combination may be the combination that minimizes the average delay time of some input frames among all input frames, or the combination that minimizes the maximum delay time of some input frames. In this embodiment, an example of synthesizing input frames of four video signals is shown, but this can be applied to input frames of any N video signals. In addition, in Figures 4 and 5, the kth and k+1th frame numbers are used for ease of understanding, but the video signals assumed in this disclosure are asynchronous, and the frame numbers and input timing of each frame are different.

Figure 6 shows an example of the configuration of an image synthesis device according to this embodiment. The image synthesis device 10 according to this embodiment comprises a detection unit 101, a crossbar switch 102, an up/down converter 103, a buffer 104, and a pixel synthesis unit 105. The figure shows four inputs and one output, but any number N of inputs and outputs may be used.

Reference numeral 101 denotes a functional unit that detects the input order and input delay time within a frame time for N input frames. For example, the input delay time D_in4 of the input 4 shown in FIG.
A crossbar switch 102 has a function of sorting and outputting the frames in the order of the detection result of the order of the input frames from 101. For example, the inputs 1, 2, 3, and 4 shown in Figs. 4 and 5 are sorted and output in the order of the kth and k+1th.
An up-down converter 103 enlarges or reduces the number of pixels to an arbitrary size. For example, the number of pixels of the input 1 is enlarged or reduced so as to match the size of the screen shown in FIG.
The inputs 102 and 103 may be connected inversely to the inputs (a, b, c, d, ...). In other words, the inputs a, b, c, d may be enlarged or reduced by 103, and then the inputs 1, 2, 3, 4 may be rearranged and output by 102 in that order.
A buffer 104 stores each input frame. The input of 103 or 102 can be buffered and output in any order.
Reference numeral 105 denotes a pixel synthesis unit. The pixel synthesis unit 105 selects a frame number for an input combination that minimizes delay based on the delay time from 101 from the entire output screen, reads data from 104, synthesizes the data to generate an output frame, and outputs the output frame. As a result, an image in which four video signals are synthesized as shown in Fig. 2 is displayed on the screen. 105 may add an arbitrary control signal to the blanking portion 22 of the screen.

The device disclosed herein can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided over a network.

In the above embodiment, an example of four inputs and one screen divided into four has been shown, but the present disclosure is not limited to this and can be applied to any input. In addition, in the above embodiment, an example has been shown in which inputs 1 to 4 have the same frame rate, i.e., frame time T_f, but the present disclosure can also be applied to inputs 1 to 4 with different frame times T_f.

For example, for input frames of input 1 having a higher frame rate than the output frame, unnecessary input frames may be thinned out as shown in Fig. 7. The unnecessary input frames are, for example, input frames that increase the delay time of the output frame when input completion times _T11 and _T12 are used as references, such as the k-1th frame. The input completion time may be a timing predicted using the frame length written at the beginning of the input frame, for example.

On the other hand, as shown in FIG. 8, for the input frame of input 1 having a lower frame rate than the output frame, the input frame ({1, k}, { ₂ , k}, { ₃ , k}, {4, k}) that shortens the delay time of the output frame based on the input completion time points T11 and T12 can be selected for the output frame O, k. Missing frames can be complemented by using past frames. For example, the input frame ({1, k}, {2, k+1}, {3, k+1}, {4, k+1}) can be selected for the output frame O, k+1. In this way, the present disclosure may use the kth input frame for multiple consecutive output frames such as the kth and k+1th, or may use multiple consecutive input frames such as the kth and k+1th for one kth output frame.

In addition, when minimizing the delay time, the input combination may be optimized for the multiple output frames. In other words, in the above example, the input combination is optimized only for output frame O,k, but it cannot necessarily be said to be optimized for output frame O,k+1. Therefore, for multiple output frames such as output frame O,k and output frame O,k+1, optimization can be performed to minimize delay values such as average or maximum.

(Effects of the present disclosure)
The present disclosure can reduce the delay time until the output of asynchronous video input signals after synthesis by selecting a combination of input frames that minimizes the output delay time. This enables collaborative work with strict low-latency requirements in a system that synthesizes multiple screens from multiple locations, etc.

As an example, the effect of the present disclosure will be shown in the case of the input frame timing shown in Figures 4 and 5. For example, when 60 fps (T_f = about 16.7 ms), T_p = 0, and D_in4 = 0.7T_f, 0.8T_f, and 0.9T_f, the values of formula (1) before the application of the present disclosure are 25.4 ms, 25.7 ms, and 27.9 ms, respectively, which exceed 25 ms, while the values of formula (2) after the application of the present disclosure are 17.9 ms, 17.5 ms, and 17.1 ms, respectively, which are below 25 ms. Therefore, the present disclosure can provide a system that generates an output frame by combining appropriate input frames from among input frames that are consecutive in time, and can synthesize and display images from multiple locations even in collaborative work that requires low latency, such as an ensemble.

This disclosure can be applied to the information and communications industry, which distributes video and game content, as well as the film, advertising, and game industries involved in video production.

10: Video synthesizer 21: Scanning line 22: Blanking portion 23: Border portion 24: Display screen 101: Detector 102: Crossbar switch 103: Up/down converter 104: Buffer 105: Pixel synthesizer

Claims (8)

Detects the delay between input frames that compose multiple asynchronously input video signals,
comparing delay times of an output frame when the multiple video signals are synthesized using adjacent input frames at the same time among the input frames of different video signals that are consecutive in time with each other, and selecting input frames of the multiple video signals so that the delay time of the output frame obtained by synthesizing the multiple video signals is minimized;
generating an output frame by combining the plurality of video signals using the selected input frame;
Image synthesis device. Detects the delay between input frames that compose multiple asynchronously input video signals,
calculating a delay time of an output frame when the multiple video signals are synthesized using input frames adjacent to each other at the same time among the multiple video signal input frames that are consecutive in time, for multiple output frames that are different in time, and selecting input frames of the multiple video signals such that an average value or a maximum value of the delay times of the multiple output frames is minimized;
generating an output frame by combining the plurality of video signals using the selected input frame;
Image synthesis device. Calculating an average delay time of an output frame obtained by combining the plurality of video signals;
selecting input frames of the plurality of video signals so that the average value is minimized;
2. The image synthesizing apparatus according to claim 1. Calculating a maximum delay time of an output frame obtained by combining the plurality of video signals;
selecting input frames of the plurality of video signals such that the maximum value is minimized;
2. The image synthesizing apparatus according to claim 1. Detect the delay between input frames based on the input completion time of the input frames.
3. The image synthesizing apparatus according to claim 1 or 2. The image synthesizer
Detects the delay between input frames that compose multiple asynchronously input video signals,
comparing delay times of an output frame when the multiple video signals are synthesized using adjacent input frames at the same time among the input frames of different video signals that are consecutive in time with each other, and selecting input frames of the multiple video signals so that the delay time of the output frame obtained by synthesizing the multiple video signals is minimized;
generating an output frame by combining the plurality of video signals using the selected input frame;
Video compositing method. The image synthesizer
Detects the delay between input frames that compose multiple asynchronously input video signals,
calculating a delay time of an output frame when the multiple video signals are synthesized using input frames adjacent to each other at the same time among the multiple video signal input frames that are consecutive in time, for multiple output frames that are different in time, and selecting input frames of the multiple video signals such that an average value or a maximum value of the delay times of the multiple output frames is minimized;
generating an output frame by combining the plurality of video signals using the selected input frame;
Video compositing method. A program for causing a computer to realize each functional unit of the image synthesizing device according to claim 1 or 2.

Images