JP7273504B2 - VIDEO ENCODED STREAM EDITING DEVICE AND PROGRAM - Google Patents

Description

The present invention relates to a video encoded stream editing apparatus and program for editing a video encoded stream.

A broadcasting station converts a complete program that can be broadcast as a program as it is into AVC (Advanced Video Coding)/H.264. Files are often managed in the form of video encoded streams encoded using a video encoding method such as H.264 (see, for example, Non-Patent Document 1). The completed program file is recorded in the delivery server and waits for its delivery time (broadcast time).

When a portion requiring correction is found in the completed program recorded in the transmission server, partial replacement editing is performed to replace only the portion requiring correction with correct content. Partial replacement editing is often performed when there is a problem with a specific cut and only this cut is replaced, or when there is an error in characters in a telop and only the frame to which this telop is added is corrected.

In the system disclosed in Non-Patent Document 1, as shown in Table 1, the completed program is converted to AVC/H. In the H.264 video coding system, a video coded stream consists of only I pictures coded using only intra-frame prediction. Since this coded video stream is composed only of I pictures that can be decoded individually, it is not necessary to once decode and re-encode the coded video stream when performing partial replacement editing. Therefore, it is possible to easily perform partial replacement editing of a section with an arbitrary frame as a start point and an end point.

In this way, the system disclosed in Non-Patent Document 1 adopts a structure of only I pictures (Intra Only) for ease of editing. However, although Intra Only is easy to edit, the coding efficiency is low. On the other hand, a structure (Long GOP (Group of Picture)) using P pictures coded using inter-frame prediction by unidirectional prediction and B pictures coded using inter-frame prediction by bi-directional prediction is , there is a drawback in terms of easiness of editing, but there is an advantage that the encoding efficiency is high because the encoding is performed by utilizing the correlation between frames.

The system disclosed in Non-Patent Document 1 is intended for HD (High Definition) video, and uses AVC/H. 264, Intra Only, 50 Mbps, but when targeting 8K Super Hi-Vision video, which is expected to spread in the future, as the amount of information in the video signal increases, a coding method with higher coding efficiency , GOP structure is required.

FIG. 6 shows 8K Super Hi-Vision video encoded by HEVC (High Efficiency Video Coding)/H.264. 1 shows graphs of bit rate and PSNR (Peak Signal Noise Ratio) when H.265 is encoded with Intra Only and Long GOP. Note that the higher the PSNR value, the less image quality deterioration. The images to be coded used are maple (Fig. 6(a)) and railcar ( FIG. 6(b)). The GOP structure of Long GOP was M=4, N=32, and Temporal_ID=2. From this figure, it can be seen that the 8K Super Hi-Vision video is converted to HEVC/H. It can be seen that Long GOP is advantageous over Intra Only when encoding with H.265.

Further, in Patent Document 1, a first partial stream from MPEG video stream 1 to immediately before an I or P picture is cut out, a second partial stream after the I or P picture is cut out from MPEG video stream 2, and the first partial stream is cut out. and a second partial stream to generate an edited stream 3 is disclosed. More specifically, if the picture displayed immediately before the second partial stream is an I picture, that picture is an intra-frame coded picture. Further, when the picture displayed immediately before the second partial stream is not an I-picture, by sequentially decoding the unidirectional inter-frame predictive coded pictures from the I-picture immediately before this picture to this picture, A picture obtained by obtaining a decoded image of the picture and re-encoded by intra-frame coding is defined as an intra-frame coded picture. Finally, an intra-frame coded picture is inserted between the first partial stream and the second partial stream, and concatenation processing is performed.

JP-A-2002-300528

"Introduction of file-based production facilities", Niimi et al., Journal of the Institute of Image Media, Vol.67, No. 5, pp.374-378, 2013

In the system of Non-Patent Document 1, which targets HD video, the completed program is encoded in Intra Only. GOP encoding is required.

However, since P-pictures and B-pictures cannot be decoded individually, when editing a section that is not a GOP unit, the coded stream after editing cannot be correctly decoded. The reason for this will be explained with reference to FIG. I, B, and P in the drawing mean I picture, B picture, and P picture, respectively. If the GOP structure of the pre-replacement stream, which is the video encoded stream before partial replacement editing, is as shown in FIG. 4A, the second to fourth frames are B pictures. , refer to the 1st frame, which is the I picture in the front, and the 5th frame, which is the P picture in the rear. Since the 10th to 12th frames are B pictures, the 9th frame, which is the forward P picture, and the 13th frame, which is the backward P picture, are referred to when decoding. Since the 13th frame is a P-picture, the 9th frame, which is the preceding P-picture, is referred to during decoding. Since the 14th to 16th frames are B-pictures, the 13th frame, which is the forward P-picture, and the backward P-picture or I-picture are referred to during decoding.

FIG. 4(b) shows frames from the fifth frame to the ninth frame (P#5, B#6, B #7, B#8, P#9) are partially replaced with replacement streams (I#1′, B#2′, B#3′, B#4′, P#5′). ing. At this time, the 2nd to 4th frames (B#2, B#3, B#4) of the post-replacement stream, which is the coded video stream after the partial replacement editing, are frames to be referred to P Since #5 does not exist, it cannot be decoded correctly. Similarly, the 10th to 13th frames (B#10, B#11, B#12, P#13) of the post-replacement stream do not have P#9, which is the frame to be referred to, so cannot be decrypted. Also, since the 13th frame (P#13) cannot be decoded correctly, the 14th to 16th frames (B#14, B#15, B#16) with the 13th frame as the reference frame cannot be decoded correctly.

Therefore, a technique for making the coded stream after editing correctly decodable will be described with reference to FIG. FIG. 5 shows frames from the fifth frame to the ninth frame (P#5, B#6, B#7, B#8, P#9) are partially replaced with replacement streams (I#1′, B#2′, B#3′, B#4′, P#5′). First, a GOP including a partial replacement section is decoded in GOP units to generate a video before replacement. After that, the pre-replacement video is edited, and the 5th to 9th frames (#5, #6, #7, #8, #9) are replaced with the replacement video (#1', #2', #3', #4', #5') to generate a post-replacement video. When data to be replaced is given as a video encoded stream (replacement stream), it is necessary to decode the replacement stream to generate a replacement video as shown in FIG. According to this method, the post-substitution video can be encoded correctly. However, since the pre-replacement stream, in which coding deterioration has already occurred, is once decoded and re-encoded, the quality of the post-replacement stream further deteriorates.

Further, in the technique disclosed in Patent Document 1, by generating the intra-frame coded picture described above, the video coded stream can be cut out and connected. I had a problem that I couldn't.

An object of the present invention, which has been made in view of such circumstances, is to provide a coded video stream editing apparatus and a program capable of partially editing an arbitrary section of a coded video stream and reducing deterioration in quality due to partial editing. to provide.

In order to solve the above-mentioned problems, a video encoded stream editing apparatus according to the present invention performs partial replacement editing on a video encoded stream encoded using inter-frame prediction, and performs video editing before partial replacement editing. An coded video stream editing device for generating a post-replacement stream, which is a video coded stream after partial substitution editing, from a pre-replacement stream, which is an encoded stream, wherein the pre-replacement stream includes a replacement section. a decoder that decodes a GOP containing the A clipped video is generated by excluding a frame that is located before the start frame of the replacement section in the display order from the video after replacement and that does not refer to frames subsequent to the start frame during encoding. a video clipping unit, an encoder that encodes the clipped video to generate a replacement stream, and a segment corresponding to the clipped video of the pre-replacement stream that is replaced with the replacement stream to generate the replacement stream. and a partial replacement unit that generates a subsequent stream, wherein the encoder sets the first frame in the decoding order of the replacement stream as an I picture, and the last frame in the display order of the replacement stream as a P picture or an I picture. It is characterized by being a picture .

Further, in the coded video stream editing apparatus according to the present invention, the video clipping unit identifies a picture type of the start frame of the replacement section in a GOP including the replacement section, and the picture type is I. In the case of a picture, the clipped video is selected from the start frame to the final frame in the post-replacement video, and if the picture type is not an I-picture, the display order of the post-replacement video. , the frame from the frame one after the P-picture or I-picture located immediately ahead of the starting point frame to the last frame is used as the clipped video.

Further, in order to solve the above problems, a program according to the present invention is characterized by causing a computer to function as the video encoded stream editing device.

According to the present invention, it is possible to partially edit a section whose start point and end point are arbitrary frames of an encoded video stream. Also, it is possible to reduce quality deterioration when partially editing a video encoded stream.

1 is a block diagram showing a configuration example of a video encoded stream editing device according to an embodiment of the present invention; FIG. 4 is a flowchart showing an operation example of the video encoded stream editing device according to one embodiment of the present invention; FIG. 4 is a diagram illustrating a specific example of processing of the video encoded stream editing device according to one embodiment of the present invention; FIG. 10 is a diagram illustrating an example in which an edited encoded stream cannot be decoded correctly; FIG. 10 is a diagram illustrating a technique for making an edited encoded stream correctly decodable; FIG. 10 is a graph showing PSNR versus bit rate when encoding 8K Super Hi-Vision video; FIG.

An embodiment of the present invention will be described in detail below with reference to the drawings.

FIG. 1 shows a configuration example of a video encoded stream editing apparatus 1 according to one embodiment of the present invention. The video encoded stream editing device 1 shown in FIG. Prepare.

The video encoded stream editing device 1 is a device that performs partial replacement editing in which a part of a video encoded stream encoded using inter-frame prediction is replaced with another stream. In the following description, the coded video stream before partial replacement editing is referred to as "pre-replacement stream", and the video coded stream after partial replacement editing is referred to as "post-replacement stream". Since the video encoded stream is encoded using inter-frame prediction, the GOP structure is Long GOP. However, the present invention targets Closed GOP and excludes Open GOP.

The transmission server 20 transmits the pre-replacement stream to the decoder 12 , the partial replacement unit 16 and the video clipping unit 14 of the coded video stream editing device 1 . The transmission server 20 also receives the post-replacement stream from the partial replacement unit 16 .

When the operator of the video coded stream editing apparatus 1 designates the start point and end point of the partial replacement editing, the replacement position specifying unit 11 generates start point/end point information indicating the start point and end point of the partial replacement edit. . The replacement position specifying unit 11 then outputs the start point/end point information to the decoder 12 , the video replacement unit 13 , the video clipping unit 14 , and the partial replacement unit 16 . The start point and end point can be specified using, for example, the time code in the stream or the frame number in display order. Also, any two frames of the encoded video stream can be specified as the start point and the end point, respectively, and the replacement sections set at the start point and the end point need not be in units of GOPs.

Based on the start point/end point information input from the replacement position specifying unit 11, the decoder 12 selects a GOP including the start point and the end point of the pre-replacement stream input from the transmission server 20 (that is, a GOP or a plurality of GOPs) is decoded to generate a video before replacement. The decoder 12 then outputs the generated video before replacement to the video replacement unit 13 .

The video replacement unit 13 acquires a replacement video from a peripheral device of the video encoded stream editing apparatus 1 such as an editing machine, a recording/reproducing device, or the like. Also, the video replacement unit 13 acquires the video before replacement from the decoder 12 . Then, based on the start point/end point information input from the replacement position specifying unit 11, the image replacement unit 13 replaces the image from the start point to the end point (replacement section) of the image before replacement with the replacement image. Generate a post-switch video. The video replacement unit 13 outputs the generated video after replacement to the video clipping unit 14 .

The video clipping unit 14 extracts a frame that is located before the start frame of the replacement section in the display order in the video after replacement input from the video replacement unit 13, and that is after the start frame at the time of encoding. A clipped video is generated by excluding frames that do not refer to frames. Then, the video clipping unit 14 outputs the generated clipped video to the encoder 15 .

The encoder 15 encodes the clipped video input from the video clipping unit 14 to generate a replacement stream. The encoder 15 then outputs the generated replacement stream to the partial replacement unit 16 . The first frame in decoding order of the replacement stream is an I picture.

The partial replacement unit 16 replaces the section corresponding to the clipped video of the pre-replacement stream received from the transmission server 20 with the replacement stream input from the encoder 15 to generate the post-replacement stream. The partial replacement unit 16 then transmits the generated post-replacement stream to the transmission server 20 .

FIG. 2 is a flow chart showing an operation example of the video encoded stream editing apparatus 1. As shown in FIG. FIG. 3 is a diagram for explaining a specific example of processing of the video encoded stream editing apparatus 1. As shown in FIG. The operation of the video encoded stream editing apparatus 1 will be described with reference to FIGS. 2 and 3. FIG. In FIG. 3, frames are counted in display order. That is, the frame number in FIG. 3 means a POC (Picture Order Count), which is a display order number.

The video encoded stream editing device 1 designates a replacement position using the replacement position designating unit 11 (step S101). In the example shown in FIG. 3, partial replacement editing of the 7th to 10th frames of the n-th GOP is performed on the pre-replacement stream received from the transmission server 20 . In this case, the replacement position specifying unit 11 generates information indicating the 7th and 10th frames of the n-th GOP as the start point/end point information.

Next, the video encoded stream editing apparatus 1 decodes the GOP including the replacement section in the pre-replacement stream using the decoder 12 to generate the pre-replacement video (step S102). In the example shown in FIG. 3, the decoder 12 decodes the nth GOP to generate pre-substitution video.

Next, the video encoded stream editing device 1 acquires the replacement video by the video replacement unit 13, replaces the video in the replacement section of the video before replacement with the replacement video, and generates the video after replacement. (step S103). In the example shown in FIG. 3, the image replacement unit 13 acquires #1', #2', #3', and #4' as replacement images.

Next, the video coded stream editing apparatus 1 confirms the GOP structure of the GOP including the replacement section by the video clipping unit 14, and identifies the picture type of the starting frame of the replacement section. If the picture type of the start frame of the replacement section is an I picture (step S104-Yes), the video clipping unit 14 cuts out the video after replacement from the start frame to the final frame. (Step S105).

On the other hand, if the picture type of the start frame of the replacement section is a P picture or a B picture (step S104-No), the video clipping unit 14 determines the display order of the pre-replacement stream in the post-replacement video. A frame position (POC) after one P-picture or I-picture located immediately ahead of the starting frame (POC order) is specified. Then, of the post-replacement video, the frame corresponding to the frame position (POC) specified earlier to the final frame are taken as the clipped video (step S106).

In the example shown in FIG. 3, since the picture type of the start frame (B#7) of the replacement section is a B picture, the video clipping unit 14 selects the P picture located immediately before the start frame (B#7). Identify the frame position (sixth frame) one after the picture (P#5). Then, the sixth frame (#6) to the final frame (#12) of the post-replacement video are cut out.

Next, the video encoded stream editing apparatus 1 encodes the clipped video using the encoder 15 to generate a replacement stream (step S107). At this time, as shown in FIG. 3, the first frame in the decoding order of the replacement stream is the I picture. Also, the last frame in the display order of the replacement stream is the P-picture or the I-picture. Other picture types in the replacement stream are arbitrary, but B-pictures or P-pictures are preferred because the coding efficiency decreases as the number of I-pictures increases.

Next, the video encoded stream editing apparatus 1 causes the partial replacement unit 16 to replace the section corresponding to the clipped video of the pre-replacement stream with the replacement stream to generate the post-replacement stream (step S108). In the example shown in FIG. 3, the first 5 frames (I#1, B#2, B#3, B#4, P#5) and the last 4 frames (I# 13, B#14, B#15, B#16) are the same as the pre-replacement stream. Therefore, quality degradation due to re-encoding does not occur for these frames.

Although the coded video stream editing device 1 has been described above, it is also possible to use a computer to function as the coded video stream editing device 1 . Such a computer stores a program describing the processing details for realizing each function of the video coded stream editing apparatus 1 in the memory of the computer, and the CPU of the computer reads and executes the program. can be realized by

Also, this program may be recorded on a computer-readable medium. It can be installed on a computer using a computer readable medium. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, but may be, for example, a recording medium such as CD-ROM or DVD-ROM.

As described above, in the present invention, when performing partial replacement editing on an encoded stream whose GOP structure is Long GOP, among the frames positioned before the starting frame in the display order, For frames that do not refer to frames, the original pre-edited video encoded stream is used as it is. Therefore, according to the present invention, it is possible to reduce deterioration in image quality due to re-encoding even if partial editing of a video encoded stream is performed.

Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many modifications and substitutions may be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as limited by the embodiments described above, and various modifications and changes are possible without departing from the scope of the appended claims. For example, it is possible to combine a plurality of configuration blocks described in the configuration diagrams of the embodiments into one, or divide one configuration block.

1 Video Encoded Stream Editing Apparatus 11 Substitution Position Designating Section 12 Decoder 13 Video Substitution Section 14 Video Clipping Section 15 Encoder 16 Partial Substitution Section 20 Sending Server

Claims (3)

Partial replacement editing is performed on a video coded stream encoded using inter-frame prediction, and a video code after partial replacement editing is obtained from a pre-replacement stream, which is a video coded stream before partial replacement editing. A coded video stream editing device that generates a post-replacement stream that is a coded stream,
a decoder that decodes a GOP including a replacement section in the pre-replacement stream to generate a pre-replacement video;
a video replacement unit that replaces the video in the replacement section of the pre-replacement video with a replacement video to generate a post-replacement video;
Generating a clipped video by excluding a frame, from the post-replacement video, which is located before the start frame of the replacement section in display order and which does not refer to frames subsequent to the start frame during encoding. a video clipping unit to
an encoder that encodes the clipped video to generate a replacement stream;
a partial replacement unit that replaces a section corresponding to the clipped video of the pre-replacement stream with the replacement stream to generate the post-replacement stream;
with
The encoded video stream, wherein the encoder sets the first frame in the decoding order of the replacement stream as an I picture, and the last frame in the display order of the replacement stream as a P picture or an I picture. editing device. The video clipping unit identifies a picture type of the starting frame of the replacement section in a GOP including the replacement section,
if the picture type is an I-picture, of the post-replacement video, from the start point frame to the final frame are taken as the clipped video;
If the picture type is not an I-picture, the switching from the frame one after the P-picture or I-picture positioned immediately before the start-point frame in display order to the final frame in the post-replacement video. 2. The coded video stream editing apparatus according to claim 1, wherein the coded video stream is an output video. A program for causing a computer to function as the video encoded stream editing device according to claim 1 or 2.

Images