JP7346124B2 - Video processing device and video processing program - Google Patents

Description

The present invention relates to a video processing device and a video processing program.

In recent years, high frame rate video with a frame frequency of 120 Hz (120 frames per second) or 240 Hz (240 frames per second) has become popular. High frame rate video is characterized by its ability to display fast-moving subjects clearly and smoothly, making it suitable for content such as sports. Note that in the following, "video" means a moving image, and "image" means a still image.

For high frame rate video, the uncompressed bit rate is high due to the high frame frequency, so selecting an interface to use for transmission becomes an issue. For example, U-SDI defined in ARIB STD-B58 exists as an interface capable of transmitting 8K 120Hz uncompressed video (maximum 144 Gbps), and is used for input/output of cameras, displays, uncompressed recorders, etc.

On the other hand, as described in Non-Patent Document 1, in high frame rate video encoding/decoding devices, etc., the input video is divided in the spatiotemporal direction in consideration of cost and implementation, and the input video is divided into 4K 60Hz video. Transmission may be performed using multiple 12G-SDI, Display Ports, etc. that are currently supported.

“NHK Giken Research Annual Report 2017”, P. 9, [online], published May 2018, Japan Broadcasting Corporation, [searched March 19, 2019], Internet <URL: https://www.nhk.or.jp/strl/publica/nenpou-h29/ 2017-chap01.pdf＞

As described in Non-Patent Document 1, when handling 120Hz video using an interface for 60Hz video, the 60Hz video obtained by thinning out the 120Hz input video one frame at a time is divided into even-numbered frames and odd-numbered frames. They are generated in groups, and when outputting a 120Hz video, the two sets of 60Hz video are combined.

However, due to malfunctions during these generation and composition processes, the order of even-numbered frames and odd-numbered frames may be reversed in the 120 Hz output video. When such an anomaly occurs, the frame number repeatedly changes back and forth, and although it is possible to visually confirm the unnatural movement of objects back and forth in the output video, The problem is that it is difficult to detect the moment when a condition occurs.

Furthermore, if the interface has frame number information, it is possible to detect and correct an inverted frame and output it, but if this is not the case, it is difficult to detect and correct the inverted frame.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a video processing device and a video processing program that can detect frame inversion even when using an interface that does not have frame number information.

The video processing device according to the first aspect is a device that processes high frame rate video consisting of even numbered frames and odd numbered frames. The video processing device includes a frame reversal detection unit that detects frame reversal in which the order of the even-numbered frames and the odd-numbered frames is reversed. The frame reversal detection unit is configured to detect, for each target frame in the high frame rate video, each of a plurality of reference frames including at least one of a frame immediately before the target frame and a frame immediately after the target frame. a calculation unit that calculates difference information representing the magnitude of the difference between the image of the target frame and the image of the target frame; and a determination unit that determines whether or not the frame reversal has occurred in the target frame based on the difference information. has.

The video processing program according to the second aspect is a program that processes high frame rate video consisting of even numbered frames and odd numbered frames. The video processing program includes a frame reversal detection step of detecting a frame reversal in which the order of the even-numbered frames and the odd-numbered frames is reversed. The frame reversal detection step includes, for each target frame in the high frame rate video, each of a plurality of reference frames including at least one of a frame immediately before the target frame and a frame immediately after the target frame. a calculating step of calculating difference information representing a difference between the image of the target frame and the image of the target frame; and a determining step of determining whether or not the frame reversal has occurred in the target frame based on the difference information. has.

According to the present invention, it is possible to provide a video processing device and a video processing program that can detect frame inversion even when using an interface that does not have frame number information.

1 is a diagram showing the configuration of a video transmission system according to a first embodiment and a second embodiment. FIG. 1 is a diagram showing the configuration of a video processing device according to a first embodiment. FIG. 3 is a diagram showing the operation of the video processing device according to the first embodiment. FIG. 3 is a diagram showing the operation of the video processing device according to the first embodiment. FIG. 3 is a diagram showing the operation of the video processing device according to the first embodiment. FIG. 3 is a diagram illustrating an example of an operation flow of the video processing device according to the first embodiment. FIG. 2 is a diagram showing the configuration of a video processing device according to a second embodiment. FIG. 7 is a diagram showing the operation of the video processing device according to the second embodiment. FIG. 7 is a diagram showing the operation of the video processing device according to the second embodiment. FIG. 7 is a diagram showing the operation of the video processing device according to the second embodiment. FIG. 7 is a diagram illustrating an example of the operation flow of the video processing device according to the second embodiment.

A video processing device according to an embodiment will be described with reference to the drawings. In the description of the drawings below, the same or similar parts are designated by the same or similar symbols.

<First embodiment>
(Video transmission system configuration)
First, a video transmission system according to this embodiment will be explained. FIG. 1 is a diagram showing the configuration of a video transmission system according to this embodiment.

As shown in FIG. 1A, the video transmission system according to this embodiment includes a transmitting device 1 and a receiving device 2. As shown in FIG.

A high frame rate video having a frame frequency of 120 Hz (120 frames per second) or 240 Hz (240 frames per second) is input to the transmitting side device 1 . An example in which the frame frequency of high frame rate video is 120 Hz will be mainly described below.

The transmitting device 1 transmits the 120 Hz video to the receiving device 2 using two interfaces for 60 Hz video. Specifically, the transmitting device 1 includes a dividing section 11 .

As shown in FIG. 1B, the dividing unit 11 generates a 60Hz video by thinning out the 120Hz input video one frame at a time, dividing it into two sets of even frames and odd frames. Then, the two sets of 60Hz videos are transmitted to the receiving device 2 via two 60Hz video interfaces. In this embodiment, it is assumed that the interface for 60 Hz video is an interface that does not have frame number information.

The receiving device 2 receives even-numbered frames and odd-numbered frames from the transmitting device 1. The receiving device 2 includes a combining section 21 and a video processing device 22.

As shown in FIG. 1(b), the combining unit 21 combines the received even-numbered frames and odd-numbered frames and outputs a 120 Hz composite video, which is a high frame rate video, to the video processing device 22. Here, due to a malfunction of the dividing section 11 or the combining section 21, the order of even-numbered frames and odd-numbered frames may be reversed in the 120 Hz output video output by the combining section 21.

The video processing device 22 processes the 120 Hz composite video, which is a high frame rate video output by the combining unit 21. In this embodiment, the video processing device 22 detects frame reversal in which the order of even-numbered frames and odd-numbered frames is reversed.

(Configuration of video processing device)
Next, the configuration of the video processing device 22 according to this embodiment will be explained. FIG. 2 is a diagram showing the configuration of the video processing device 22 according to this embodiment.

As shown in FIG. 2, the video processing device 22 according to this embodiment includes an input section 221, a buffer section 222, a frame inversion detection section 223, and an output section 224.

A 120 Hz composite video, which is a high frame rate video, is input to the input unit 221 . The input unit 221 outputs the input 120Hz composite video to the buffer unit 222.

The buffer section 222 temporarily stores each frame constituting the 120 Hz video output from the input section 221 in the order of input.

The frame reversal detection unit 223 sequentially reads frames from the frames stored in the buffer unit 222 and detects frame reversal in which the order of even-numbered frames and odd-numbered frames is reversed. If frame inversion is not detected, the frame inversion detection unit 223 sequentially outputs the frames read from the buffer unit 222 to the output unit 224. On the other hand, when frame inversion is detected, frame inversion detection section 223 outputs information indicating that frame inversion has been detected to output section 224. The frame reversal detection section 223 includes a calculation section 223a and a determination section 223b.

The calculation unit 223a calculates, for each target frame in the high frame rate video (120 Hz video), difference information representing the magnitude of the difference between the image of the target frame and the image of each of the plurality of reference frames. In the present embodiment, the plurality of reference frames include a frame immediately before the target frame and a frame immediately after the target frame.

The determination unit 223b determines whether frame inversion has occurred in the target frame based on the difference information calculated by the calculation unit 223a. In the present embodiment, the determination unit 223b determines that the difference information calculated by the calculation unit 223a for the first target frame, which is one of the target frames, is the difference information calculated by the calculation unit 223a for the second target frame immediately before the first target frame. If the difference information has increased compared to the calculated difference information, it is determined that frame inversion has occurred for the first target frame.

The output unit 224 sequentially outputs the frames output from the frame inversion detection unit 223 to the outside (for example, an external display device). The output unit 224 may be a display device that sequentially outputs the frame images output from the frame reversal detection unit 223. When the frame inversion detection unit 223 outputs information indicating that frame inversion has been detected, the output unit 224 outputs the information.

(Operation of video processing device)
Next, the operation of the video processing device 22 according to this embodiment will be explained. 3 and 4 are diagrams showing the operation of the video processing device 22 according to this embodiment. In FIGS. 3 and 4, the frame number on the horizontal axis indicates the frame number in the correct display order.

FIG. 3A shows the relationship between the screen position (vertical axis) of an object moving at a constant velocity in a certain video and the frame display order (horizontal axis). The screen position shown in FIG. 3A indicates, for example, the horizontal (x) or vertical (y) position of an object included in one frame of image.

As shown in FIG. 3(a), the position of the object included in the image changes at equal intervals as the frame number increases. Not all objects move at a uniform velocity, but in high frame rate video the time interval between successive frames is short and changes are minute, so here we will focus on objects that are moving approximately at a uniform velocity. It is treated as

FIGS. 3(b) and 3(c) show a case where frame inversion occurs in the video of FIG. 3(a).

Specifically, FIG. 3(b) shows a case where even-numbered frames precede odd-numbered frames. Even-numbered frames after frame number "4" precede odd-numbered frames, and the frame numbers are 0, 1, 2, 4, 3, 6, 5, . . . .

FIG. 3(c) shows a case where odd-numbered frames precede even-numbered frames. Odd-numbered frames after frame number "3" precede even-numbered frames, and the frame numbers are 0, 1, 3, 2, 5, 4, 7, 6, . . . .

Here, the screen position P of the object in the nth frame (target frame) is P _n , the screen position P of the object in the n-1st frame (previous frame) is P _n-1 , and the screen position P of the object in the n+1th frame ( When the screen position P of the object in the immediately following frame is Pn ₊₁ , the absolute value of the second-order differential _ΔPn of _Pn in the temporal direction is calculated by the following equation (1).

|ΔP _n |=|2P _n -P _n-1 -P _n+1 | ...(1)
The calculation results of |ΔP _n | for each of FIGS. 3(a) to (c) are shown in FIGS. 4(a) to (c).

As shown in FIG. 4(a), the object in the image in FIG. 3 is moving at a constant velocity, so |ΔP _n | becomes 0. On the other hand, in Figures 3(b) and (c) where frame inversion occurred, the position of the object repeatedly increases and decreases, so ΔP _n alternately takes positive and negative values, and its absolute value |ΔP _n | gradually increases. After that, it takes a constant value. Based on this principle, the frame inversion detection section 223 can detect frame inversion.

Figure 5 shows the results of applying a similar method to 8K 120Hz video.

FIG. 5 shows a case where the luminance component Y of the nth frame is _Yn , and the vertical axis is the variance of the second-order differential _ΔYn = _2Yn -Yn _{-1 -Yn+1} in the time direction. There is.

For example, the calculation unit 223a of the frame reversal detection unit 223 calculates ΔY _n for each image position (x, y) for each target frame, and also calculates the variance for each frame. In this embodiment, the variance of ΔY _n corresponds to difference information indicating the magnitude of the difference between the image of the target frame and the images of each of the plurality of reference frames.

Although an example in which the variance of ΔY _n is used as the difference information will be mainly described below, other statistical quantities of ΔY _n , such as the average of ΔY _n , may be used as the difference information.

Further, in this embodiment, the calculation unit 223a of the frame inversion detection unit 223 uses the entire 8K frame for calculation, but in order to reduce the amount of calculation, calculation may be performed after reducing the frame to 4K or the like. However, the calculation may be performed on a portion of the image, such as the central 2K area.

FIG. 5A shows a case where the frames are displayed correctly in the order of frame numbers, and FIGS. 5B and 5C show a case where frame inversion occurs in which an even-numbered frame and an odd-numbered frame precede each other, respectively.

The dispersion of ΔY _n in FIGS. 5(b) and 5(c) is larger than that in FIG. 5(a). Therefore, the determination unit 223b of the frame inversion detection unit 223 can determine whether frame inversion has occurred in the target frame based on the variance of ΔY _n . For example, the determination unit 223b of the frame reversal detection unit 223 determines that the variance of ΔY _n calculated for one frame (first target frame) is equal to the variance of ΔY _n− calculated for the frame immediately before this frame (second target frame). If the variance has increased compared to the variance of ₁ , it is determined that frame inversion has occurred for the first target frame.

(Example of operation flow of video processing device)
Next, an example of the operation flow of the video processing device 22 according to this embodiment will be described. FIG. 6 is a diagram showing an example of the operation flow of the video processing device 22 according to this embodiment.

As shown in FIG. 6, in step S11, the calculation unit 223a of the frame inversion detection unit 223 calculates the variance (denoted as V _n ) of ΔY _n of the n-th frame as difference information.

In step S12, the determination unit 223b of the frame reversal detection unit 223 compares the variance of ΔY _n-1 (denoted as V _n-1 ) of the previous (n-1st) frame with V _n , and It is determined whether _n has increased from V _n-1 .

The determination unit 223b of the frame reversal detection unit 223 is not limited to directly comparing V _n-1 and V _n , but may also compare V _n-1 with a coefficient added to V _n . For example, the determination unit 223b of the frame reversal detection unit 223 compares V _n and 1.2×V _n-1 , and determines that V _n has increased from 1.2×V _n-1 (V _n >1.2 ×V _n-1 ).

Further, the determination unit 223b of the frame reversal detection unit 223 may determine whether V _n has increased continuously for a plurality of frames. For example, the determination unit 223b of the frame reversal detection unit 223 may determine whether the condition V _n >V _n-1 >V _n-2 is satisfied. Such a determination is effective in preventing false detection due to a scene change (change in picture pattern). Further, the determination unit 223b of the frame reversal detection unit 223 introduces the coefficients as described above, and determines that the conditions of V _n >V _n-1 >V _n-2 and V _n >1.1×V _n-2 are satisfied. It may be determined whether or not it has been done. Such a determination is particularly effective when the change in dispersion due to frame inversion is small, such as in fixedly photographed video.

Note that the value of the coefficient is not limited to "1.2" or "1.1", and other values may be selected.

If "Yes" in step S12, the determination unit 223b of the frame inversion detection unit 223 determines that frame inversion has occurred for the n-th frame in step S13, and ends the process.

On the other hand, if "No" is determined in step S12, the determination unit 223b of the frame reversal detection unit 223 determines whether the next (n+1th) frame is the last frame or not in step S14. If "Yes" in step S12, the process ends.

If "No" in step S12, in step S15, the calculation unit 223a of the frame reversal detection unit 223 substitutes n+1 for n, and restarts the process from step S11.

Although the process ends when frame inversion is detected in step S13 in FIG. 6, the determination unit 223b of the frame inversion detection unit 223 subsequently advances the process to step S14 and determines whether the frame inversion has returned to its original state. It may be determined whether or not. In that case, V _n before and after detecting frame reversal may be stored, and it may be determined whether V _n approaches before detecting frame reversal (the reverse of step S12).

In this way, according to the present embodiment, even when using an interface that does not have frame number information, it is possible to detect the inversion of even-numbered and odd-numbered frames of a high frame rate video based on the image of each frame.

<Second embodiment>
Next, the second embodiment will be mainly described with respect to differences from the first embodiment.

In the first embodiment, frame inversion can be detected, but frame inversion cannot be corrected. In this embodiment, when a frame inversion is detected, the frame inversion can be corrected.

(Configuration of video processing device)
FIG. 7 is a diagram showing the configuration of the video processing device 22 according to this embodiment.

As shown in FIG. 7, the video processing device 22 according to this embodiment differs from the first embodiment in that it further includes a frame inversion correction section 225. When the frame inversion detection unit 223 detects a frame inversion, the frame inversion correction unit 225 corrects the detected frame inversion and outputs the corrected frame to the output unit 224.

In the present embodiment, the calculation unit 223a of the frame inversion detection unit 223 uses a frame immediately before the target frame (previous frame) as a plurality of reference frames to be referred to when calculating difference information for the target frame. The frame three frames before the target frame is used.

Alternatively, the calculating unit 223a of the frame reversal detecting unit 223 may calculate the frame after the target frame (immediate frame) and the three target frames as a plurality of reference frames to be referred to when calculating difference information for the target frame. A later frame may also be used.

The frame inversion correction unit 225 calculates that the difference information calculated by the calculation unit 223a for the first target frame, which is one of the target frames, is the difference information calculated by the calculation unit 223a for the second target frame immediately before the first target frame. If the number of frames increases compared to the information, it is determined that the first target frame is a frame that preceded by frame inversion, and the order of the first target frame is corrected so that it is placed later.

On the other hand, the frame inversion correction unit 225 calculates that the difference information calculated by the calculation unit 223a for the first target frame, which is one of the target frames, is the difference information calculated by the calculation unit 223a for the second target frame immediately before the first target frame. If the difference information is decreased compared to the difference information, it is determined that the first target frame is a subsequent frame due to frame inversion, and the order of the first target frame is corrected so that it comes first.

(Operation of video processing device)
Next, the operation of the video processing device 22 according to this embodiment will be explained. 8 to 10 are diagrams showing the operation of the video processing device 22 according to this embodiment. In FIGS. 8 to 10, the frame number on the horizontal axis indicates the frame number in the correct display order.

8 and 9 show the calculation results of |2P _n −P _n−1 −P _n−3 | and |2P _n −P _n+1 −P _n+3 | in the same example as FIG. 3, respectively. Specifically, FIGS. 8(a) and 9(a) show the case where the display order is correct, and FIGS. 8(b) and 9(b) show the case where the even numbered frame precedes the odd numbered frame. 8(c) and FIG. 9(c) show the case where the odd-numbered frame precedes the even-numbered frame.

As shown in FIGS. 8(a) and 9(a), when the display order is correct, calculation of |2P _n -P _n-1 -P _n-3 | and |2P _n -P _n+1 -P _n+3 | The result will be the same value and will take a constant value.

On the other hand, as shown in FIGS. 8(b) and (c), the calculation results of |2P _n -P _n-1 -P _n-3 | 8(b) and 8(c)), the values are larger than in the case where the display order is correct; , even numbered frames), it becomes 0. When it becomes 0, it corresponds to calculating the second-order differential ΔP _n shown in the first embodiment in the correct display order.

Furthermore, in FIGS. 9B and 9C, in which |2P _n −P _n+1 −P _n+3 | is calculated, the relationship between the preceding and subsequent frames is opposite to that in FIGS. 8B and 8C.

Therefore, by calculating at least one of |2P _n -P _n-1 -P _n-3 | and |2P _n -P _n+1 -P _n+3 The frame flip can be corrected.

FIG. 10 shows an example in which a similar method is applied to 8K 120Hz video. Specifically, the calculation unit 223a of the frame inversion detection unit 223 sets the luminance component Y of the n-th frame (horizontal axis) to Y _n , and the forward difference Difprev _n =2Y _n -Y _n-1 -Y _n-3 An example of calculating the variance of is shown. Similarly, the calculation corresponding to FIG. 9 is obtained by dispersing the backward difference DifBack _n =2Y _n -Y _n+1 -Y _n+3 .

In this embodiment, the variance of the forward difference Difprev _n corresponds to difference information representing the magnitude of the difference between the image of the target frame and the images of each of the plurality of reference frames. By calculating the forward difference Difprev _n , it is possible to determine whether the n-th frame is an inverted preceding or subsequent frame, and to correct the frame inversion.

In this embodiment, an example will be described in which only the forward difference Difprev _n is used to correct frame inversion, but frame inversion may be corrected using only the backward difference DifBack _n , or the forward difference Difprev _n and the backward difference DifBack _n You may use both. Furthermore, after using the method of the first embodiment to detect frame inversion, the correction processing according to the present embodiment may be performed.

(Example of operation flow of video processing device)
Next, an example of the operation flow of the video processing device 22 according to this embodiment will be described. FIG. 11 is a diagram showing an example of the operation flow of the video processing device 22 according to this embodiment.

As shown in FIG. 11, in step S21, the calculation unit 223a of the frame inversion detection unit 223 calculates the variance of the forward difference Difprev _n of the n-th frame as difference information.

In step S22, the determination unit 223b of the frame reversal detection unit 223 determines whether Difprev _n is approximately the same as the variance of the forward difference Difprev _n-1 of the n-1th frame (for example, ±5%). judge.

If "Yes" in step S22, the determination unit 223b of the frame inversion detection unit 223 determines that frame inversion has not occurred, and advances the process to step S26.

On the other hand, if "No" in step S22, the determination unit 223b of the frame inversion detection unit 223 determines that frame inversion has occurred, and advances the process to step S23.

In step S23, the frame inversion correction unit 225 determines whether Difprev _n has increased with respect to Difprev _n-1 .

If “Yes” in step S23, in step S24, the frame inversion correction unit 225 determines that the n-th frame is the frame that preceded it by frame inversion, and corrects the display order of the n-th frame to one place later. Then, the process advances to step S26.

On the other hand, if "No" in step S23, in step S25, the frame inversion correction unit 225 determines that the nth frame is a subsequent frame by frame inversion, and changes the display order of the nth frame by one. The error is corrected beforehand, and the process proceeds to step S26.

On the other hand, in step S26, the determination unit 223b of the frame inversion detection unit 223 or the frame inversion correction unit 225 determines whether the n-th frame is the last frame. If "Yes" in step S26, the process ends. If "No" in step S26, the calculation unit 223a of the frame reversal detection unit 223 substitutes n+1 for n in step S27, and restarts the process from step S21. Note that when the display order is corrected in step S24 or S25, Difprev _n and Difprev _n-1 in steps S21 to S23 are calculated in the state after the correction is applied.

In this way, according to the present embodiment, even when using an interface that does not have frame number information, it is possible to correct the inversion of even and odd frames of a high frame rate video based on the image of each frame.

<Other embodiments>
A program that causes a computer to execute each process performed by the video processing device 22 may be provided. The program may be recorded on a computer readable medium. Computer-readable media allow programs to be installed on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Further, the circuits that execute each process performed by the video processing device 22 may be integrated, and the video processing device 22 may be configured by a semiconductor integrated circuit (chip set, SoC).

Although the embodiments have been described above in detail with reference to the drawings, the specific configuration is not limited to that described above, and various design changes can be made without departing from the scope of the invention.

1: Sending device 2: Receiving device 11: Dividing section 21: Combining section 22: Video processing device 221: Input section 222: Buffer section 223: Frame inversion detection section 223a: Calculation section 223b: Determination section 224: Output section 225 :Frame inversion correction section

Claims (8)

A video processing device that processes high frame rate video consisting of even numbered frames and odd numbered frames,
comprising a frame reversal detection unit that detects a frame reversal in which the order of the even numbered frame and the odd numbered frame is reversed;
The frame reversal detection section includes:
For each target frame in the high frame rate video, the target frame for each image of a plurality of reference frames including at least one of the immediately preceding frame of the target frame and the immediately following frame of the target frame. a calculation unit that calculates difference information representing the size of the difference between the images;
A video processing device comprising: a determination unit that determines whether or not the frame inversion has occurred in the target frame based on the difference information. The video processing apparatus according to claim 1, wherein the plurality of reference frames include both the immediately preceding frame and the immediately following frame. The determination unit is configured to determine whether the difference information calculated by the calculation unit for a first target frame, which is one of the target frames, is the difference information calculated by the calculation unit for a second target frame immediately before the first target frame. 3. The video processing apparatus according to claim 2, wherein when the difference information increases, it is determined that the frame inversion has occurred in the first target frame. The video processing device according to claim 1, further comprising a frame inversion correction unit that corrects the detected frame inversion when the frame inversion detection unit detects the frame inversion. The video processing apparatus according to claim 4, wherein the plurality of reference frames include the immediately preceding frame and a frame three frames before the target frame. The video processing apparatus according to claim 4, wherein the plurality of reference frames include the immediately following frame and a frame three frames after the target frame. The frame inversion correction unit is configured such that the difference information calculated by the calculation unit for a first target frame, which is one of the target frames, is calculated by the calculation unit for a second target frame immediately before the first target frame. Determining whether the first target frame is a preceding frame due to the frame inversion or a subsequent frame due to the frame inversion, depending on whether the first target frame is increased or decreased compared to the calculated difference information. The video processing device according to any one of claims 4 to 6, characterized in that: A computer is made to function as the video processing device according to claim 1.
A video processing program characterized by:

Images