TWI224928B - Real-time MPEG video encoding method to keep output synchronization of video and audio - Google Patents

Abstract

The invention relates to a real-time MPEG video encoding method to keep output synchronization of video and audio. When real-time MPEG video encoding, if any lost frames are detected, the system then uses some null frames to make up the deficiency of the lost frames. Or, when the system performance of real-time MPEG video encoding is insufficient, then the system encodes according to a predetermined frame-type sequence. And the null frames are distributed averagely in the predetermined frame-type sequence. The encoding of null frame is to transform a null predictive frame into a discrete cosine transform coefficient by the discrete cosine transform algorithm, of which the pixel value of null predictive frame is 0. The discrete cosine transform coefficient is then transformed into a quantized discrete cosine transform coefficient by quantization. And the quantized discrete cosine transform coefficient is then encoded with a null motion vector by variable length coding to output null frames, and the null motion vector value is (0, 0).

Description

1224928

1. [Technical Field to which the Invention belongs] The present invention relates to a video coding method. In particular, the present invention relates to a method for real-time video encoding that maintains MPEG (Motion Picture Experts Group, animation compression standard) video and audio synchronization. : "[Previous Technology] Video Compact Disc (VCD) and Digital Versatile Disc (DVD) are small in size and good in storage. At the same time, users can easily create and edit personal The VCD or DVD can be saved or distributed to friends and relatives, and even the DVD's high quality and large storage capacity can be used as real-time recording of TV programs. Due to the advantages of these digital audio and video technologies, VCD ADVD has gradually replaced the market of traditional video tapes and is welcomed by consumers. '’And in the VCD and DVD specifications, ffiMPEG 丨 and liver“ 2 video encoding methods are used as the main formats for storing video. The National Televisi () n System is used in North America and Japan.

Committee, NTSC) regulations, the video resolution of VCD is 352 x 24 pixels (Pixel), 29.97 frames per second (frame), the uncompressed data reaches 58M bps ° DVD video resolution is 72 〇χ 48〇 pixels, 2917½ frames per second (frame), uncompressed data reached 237M, bps. In addition, the Phase Alternate Lines (PAL) specifications adopted by most European countries, Australia and other countries, the VCD image is 35 2 χ 2 88 pixels, 25 frames per second, uncompressed data The resolution of the image reaching 58M bps 0DVD is 72 °} (576 pixels, every 1224928 V. Description of the invention (2)

Play 25 screens (f r ame), the research topics to be studied under such a high amount of data. Uncompressed data also reached 237M bps. How to achieve real-time encoding and compression becomes the next. Then briefly explain the principles of MPEG 1 and MPEG 2 video coding. Stand up

If a sequence of image carousel is divided into two parts: the first part is called intra-coding, which can generate j picture (1111; 1 ^ picture), and the basin is a method using discrete cosine transform (Discrete c 〇sine TfansfQi ^ DCT) algorithm is a compression coding method for a single image. Therefore, the picture does not need to refer to the forward or backward frames to generate a static frame, which occupies the largest amount of data. The second part is called external coding (Inter_coding), which uses the left and right adjacent frames to predict the frame. A method of predicting and reconstructing the frame. This method can encode and generate a p-picture (predicted (Bi- The difference between directional predicted picture, p-picture and B-picture is that p-picture uses only forward prediction (i.e., p-picture), which means that the reconstruction of p-picture only refers to the forward grid. And B-picture can use forward and backward. 'Prediction or average reconstruction of the two, it has the highest coding efficiency, but the B picture itself is no longer used for other predictive coding.

I, for example, record a fen film falling from a building in a sphere in MPEG format. First of all, the first I picture is generated by directly coding with DC.T: technology. If the P picture is generated by predicting the distance from the previous fence to 1, the next P picture is referenced to the previous 1 picture and reconstructed with motion estimation and compensation (motion estimation and compensation) technology. For example, the static frame of the first {body at the height of the building, because the building is a stationary object that changes ^, so the second day of the building can refer to the previous ^

1224928 V. Description of the invention (3) Building in Figure I. The falling part of the sphere can be used for motion estimation to calculate the motion vector and motion compensation technology. Refer to the previous I picture to reconstruct the second p picture. Similarly, the third p-map is reconstructed with reference to the second p-map. Therefore, the P picture only needs to record the difference from the previous frame to be reconstructed, so the amount of data is greatly reduced and the purpose of compression is achieved.

In addition, as shown in Figure 1, after the first I-picture is spaced a fixed number of p-pictures, an I-picture must be created to maintain the image quality. The user can also select a certain clip in the video to start playing without having to play sequentially from the first I picture. If two I-pictures are far apart, that is, a large number of P-pictures are included, the image quality is poor. But if two! The pictures are too close. Because the picture is a complete static frame, it will occupy a considerable amount of data. Therefore, the effect of using compression technology to reduce the amount of data will not appear. The previous day grid from one I picture to the next I picture is called a picture group (21: 011). How many day grids are in a picture group, you can set encoding rules according to actual needs. However, when the above-mentioned MPEG video encoding method is used for real-time encoding and compression, video and audio may be out of sync. The main reason is that the MPEG video encoder cannot achieve the speed required for real-time encoding and compression, which makes the total number of diurnal cells insufficient. For example, according to the NTSC standard, when a 1-second movie is input, 30 frames will be played every time. When the speed of the MPEG video encoder is only 15 days per second, the encoded file will only have 15 days. When playing, the MPEG file with 15 days will be within 0.5 seconds. The playback is complete, but the sound is still playing for 丨 seconds, making the video end earlier than the audio by 0.5 seconds. As a result, the video source is input every second

Page 12 1224928

V. Description of the invention (4) = When the number of grids is not enough, even the encoding speed of the MpEG video encoder is inconsistent. It is required that the encoded MPEG file will produce different results from video and audio. In summary, when the real-time encoding speed of the video encoder is insufficient, or when the number of frames per second input by Mai Yuan's video is insufficient, it can still be played without video and audio synchronization after encoding. Solution 3. [Content of the invention] To solve the above problems, the purpose of the present invention is to provide a method for maintaining MPEG video time video encoding, which can be used in other video encoding systems and audio systems. When the following MPEG slot case keeps the video full, it can still ensure that after encoding: 7 The case is said to keep the video and audio synchronized to achieve the above-mentioned purpose. 'Maintaining MPEG video and audio synchronization § fl encoding method of the present invention' includes a detection method, a detection method, a detection method, and a detection method. Private sequence, a complementing painting program, and private sequence. The detection process is missing. Make up for Jin Baozpi + 丄 Very good 1 疋 Is there a frame day frame to make up for the missing frame 7 佥 袼] When the 4 frames are lost, the zen video sequence of MPEG video coding is used as a virtual frame. . And the encoding of the virtual frame d: Edit: go and output the encoded picture for encoding, where ==;

1224928 V. Description of the invention (5)

Ceding, VLC)), the input motion vector is (0, 〇). Since the step of motion estimation in video coding is skipped and the calculations are quick, so check the conversion and quantity of cosine.

The picture is fast, and the lost time can be instantly replenished. [1], [P], and [B] The other mpeg image of the present invention; the real-time synchronization: sequence = 1 ::: unified performance program,-select frame ii two code program. Detecting the system performance program, system performance: and predicting the diurnal rating that the system can encode per second. The pattern output sequence of the frame is the same thing. The sequence of the day cells in the group, the states ΐ 袼 孓 L include I-picture, ρ-picture and virtual painting. Book-turns into &: = 碥 code sequence according to the selected output diurnal pattern sequence, according to code. I. 'Sequence input sequence video sequence in sequence, the instant video to maintain synchronization of liver EG video and audio in the present invention Editing: Edit: r degree is faster, so it can be real-time: Foot = ΐ, the case has enough frames Count 'to ensure vision; and sound two or four, [implementation]

, To explain the method for encoding video information according to a preferred embodiment of the present invention, in which the same components are referred to the related drawings below.

1224928 V. Description of Invention (6) The same symbol will be used for explanation. The master of the 22nd example on the main f supports the real-time video encoding method of MPEG video and audio synchronization. Ί g = P-test method to generate P maps. The predicted distance is! When the source of the image has a book w-a day grid. Please refer to FIG. 2a, (S22). From the time (), Zhe Lin and Zhe 4 judged whether there was a missing day frame. The reason may be that the system encodes MPEG video: the number of frames input by the standard and image sources is not

For example, if it is in accordance with the NTSC standard, 30 day frames will be played per second, and 30 frames will be input for video encoding by the system. Assume that the image with a length of 2 seconds has a daylight grid number provided in the input daylight grid. Starting from the number one by one, the daylight grid number will be changed to 30. The system will also encode the inputted 30 frames one by one. Under normal circumstances, when the serial number of the day frame is input ^ 25, the system = 24 frames have been encoded, and video encoding will be performed for the 25th frame. "Because of the system's insufficient performance and poor transmission quality, when the serial number of the input grid is 25, the number of day grids encoded by the system is only 2 q. . The missing number of diurnal periods F_t can be calculated from the following equations: Check: 疋 丨 Poor preparation of the opera mouth and other factors. As for how many missing numbers can be obtained by comparing the number of time divisions that have been coded by the current system with the timestamp s Danger serial number provided by the input screen.

lost i nput-Fc〇ded 1 * where Flnput is the serial number of the day grid currently entered, which is the number of system coded spoons. Therefore, the number of day cells lost in the previous example is 4 paintings. Similarly, the relative diurnal serial number can also be calculated with the time stamp. In the above example, the phase

1224928

V. Description of the invention (7) The time difference between the time marks of the two adjacent day marks is about 1/30 second. If the relative frame number of the time mark of 2 is 1, then the following picture is == / Time stamp, calculate the relative frame number with the time difference of 1/30 second, and then calculate the number of missing frames. Relative paintings When there is a missing frame ($ 22), the process proceeds to step S23 to output a rum frame to fill the missing frame until it completely fills up the missing frame (please describe the coding method of the virtual frame later). Make up for the lost daytime horror. The daylight grid of 5 people sentenced to video coding. As for the book that should be coded

= Type = It is determined in step S24 whether the picture group has accumulated enough daylight time. Since a picture group is composed of at least one I picture and a plurality of ρ pictures, where the P picture can be replaced by a virtual grid, when the picture group has accumulated enough number of day grids, the input grid code is output as I picture (S2 5), starting to accumulate a picture to generate „new picture groups, if no money, output ρ picture (S26) 'to make the current picture group accumulate a sufficient number of frames. I picture The video encoding methods of P and P are conventional techniques and will not be described here again. Afterwards, 'return to step S21 to determine whether there is a day grid input, and if there is no frame input', the entire video encoding process is ended.

In order to make the spirit of the embodiment of the present invention easier to understand, please refer to FIG. 2b for description. When the image source has a diurnal input, the diurnal program 27 is detected to check whether the number of diurnal inputs is consistent with the number of diurnal codes encoded by the system, that is, whether any diurnal loss has occurred. When the diurnal grid is missing, make up the diurnal program 28, using the diminished diurnal grid to make up for the missing diurnal grid. After compensating the missing frames, a day-space coding program 29 is performed, and the input frames are encoded using MpEG video coding standard procedures and the encoded frames are output. If there is no frame

Page 16 1224928

Missing code output. Then, directly perform the frame coding program 29 to edit the input frame, and then refer to FIG. 3 to describe the frarae of the virtual frame 31 (S31). The pixel value of the predicted day frame is 0W = 1Ct1Ve discrete cosine transform algorithm. Processing, generating scattered cosine = predicted diurnal meridian 2 (S32), and then quantizing the discrete cosine conversion coefficient 32, and / or the scattered cosine conversion coefficient 3 3 (S3 3). Let ’s introduce the brother from the inside, let ’s say that A 1 r 1 moves the converted discrete cosine conversion coefficient 33 to Dong (nu “otlon vector) for variable length

(vanable length coding, VLC) (S34) to output a CS35), in which the virtual motion vector is rfct (from 0 to τ μ, which is U), 0). Because the virtual frame omits the operation evaluation, and the operation of discrete cosine transformation and quantization is faster, the coding speed of the virtual day lattice is much faster than that of I, P, and B pictures.

Please refer to FIG. 4, illustrating a real-time video encoding method for maintaining swollen video and audio synchronization according to another embodiment of the present invention. First, the performance of the MPEG video encoding system 41 for processing video encoding is detected (S41), and the prediction is performed according to the performance evaluation of the system 41 The number of diurnal output (§ 4 2) of the video coding output of the system 41 per second, hereinafter referred to as the number of codeable frames per second Fp. It is then the number of diurgable coded frames per second. Sequence table, select the appropriate output diurnal sequence P (S 4 3). The grid type output sequence list is a comparison table defined in advance based on the experimental results, and its content is to define each constituent diurnal in each picture group and group The frame pattern sequence. The frame pattern includes I, P, and virtual frames. For example, as shown in Table 1, when the system can encode the number of frames per frame ^ is 20 to At 21 day divisions, each of the 15 day divisions in terms of NTSC specifications

Page 17 1224928 Schematic illustration, GOP picture group I I NTSC National Television System Committee (Nati〇na i

Television System Committee) specification PP if S2; l ~ S26 flow of real-time video coding method 53 BU S35 54 BU S45 PAL Phase Alternate Lines Specification The preferred embodiment of the present invention maintains MPEG video and audio synchronization ^ The flow of the virtual day encoding in the preferred embodiment of the present invention is another preferred embodiment of the present invention. The p-step real-time video I flat stone horse method is presented, and supports the MPEG shirt.

T β timeline virtual frame

Page 21

Claims (1)

1224928 6. Scope of patent application 1 ~ 1__ ---- 1 · A real-time video encoding method to maintain MPEG, (Motion Picture Experts Group, animation compression standard) video and audio synchronization, including: ^ 炊 一 ^^ 格格 程序 ' It is used to judge whether the input number of day frames is consistent with a coded frame number. A complementing painting program is used to make up a missing frame with a virtual frame when the number of coded day frames is counted. & / input-frame coding program, used to encode an input dice with the standard procedure of MpEG video and output a coded dice. … 2. The real-time video encoding method for maintaining MPEG video and audio as described in item 1 of the scope of the patent application, wherein the frame format of the encoded frame includes human I and P images, where the day grid type is The coded diurnal system of p-picture is generated by 3 predictions, and the prediction distance is 1. 3. The real-time video encoding method for maintaining MPEG video and audio as described in item 1 of the scope of patent application 5 wherein the number of input frames is the number of the current input frame provided by a 9-round search ^ 查. 'Dange 4. The real-time video encoding method for maintaining MpEG video and audio wells as described in item 丨 of the patent application scope, wherein the input frame number is a time stamp (ti me stamp) provided by a round of gold deposit. Calculate one of the currently input ° frame numbers. Λ Cut-in 5. The real-time video coding method for maintaining MPEG video and audio as described in item 3 of the scope of the patent application, where the number of frames of the residual beam is calculated by the following 5 丨 '4 1 M program. Lost -Finput-FCOfied -1 No. 22 I 1224928 6. Scope of patent application Where I ^ st is the number of missing day frames, Fuput is the current input picture number ′ FC〇ded is the number of coded picture frames. 6. The real-time video encoding method for maintaining MPEG video and audio as described in item 1 of the scope of the patent application, wherein the encoding step of the virtual frame includes: (1) inputting a virtual prediction golden grid; performing discrete cosine conversion to generate a discrete cosine Transform coefficients; quantize the discrete cosine transform coefficient (quantizati〇 quantized discrete cosine transform coefficient; and generate-quantize the quantized discrete cosine transform coefficient with an imaginary motion direction ^ J ^ I (VariaMeLength c〇ding,% · '7 2. The method of maintaining MpEG video and audio video coding as described in item 6 of the patent application range, wherein the pixel value of the virtual pre-engraved diurnal grid is 02: The video encoding of maintaining MPEG video 17 as described in item 6 of the patent application range Method, where the virtual motion vector is (〇, 〇). Including ··· A real-time video encoding method to maintain MPEG video and audio synchronization, including this system: measuring system performance procedures, used to evaluate-system performance, and predict death One mother second can encode the number of diurnal frames; exhaustive drawing: "Prologue" is used to select a group of pictures according to the editable list per second (gr0 ", the output sequence list of cancer type ^^, ιΡ; ΓΓδ, G0P) 41 ^ ^ ^ ^ ^ The frame coding program is to output the input frame according to the frame type Page 23 Corresponding to the sequence — j q ★ 'Lattice type, for MPEG video coding. Real-time vision σ I asks for the control 2 method of maintaining the mpeg video and audio synchronization as described in item 9 of the patent scope first, where the frame type output sequence list is one thing and the day grid 3 [/ 能 、 二 ， 其The content contains at least the number of codeable frames per second and this column is output as ~ soil w. Immediate and Loss: Maintain the MPEG video and audio synchronization of virtual books as described in item 10 of the patent scope. Λ editing method, where the frame shape includes 1 picture, p picture, and 12. Rushen ’s real-time video editor prediction, prediction distance 13. Rushen ’s real-time video editor inputs a dummy to discrete the discrete quantized discrete residual Coding the measured variable length, 1 4. Rushen's real-time video editing 15. Rushen's real-time video editing 16. Please refer to the method for maintaining MPEG video and audio synchronization code described in item U of the patent scope, wherein the frame type of the P picture is generated by using the forward separation method. The method for maintaining an MPEG video synchronization code as described in item 11 of the patent scope, wherein the virtual frame encoding step includes: predicting the frame; cosine transform to generate a discrete cosine transform coefficient; quantization of the cosine transform coefficient to generate a Chord conversion factor; The discrete cosine transform coefficient and an imaginary motion vector are combined to output the virtual frame. The method for maintaining the MPEG video and audio synchronization code described in item 13 of the patent scope, wherein the pixel value of the virtual prediction daytime is zero. The method for maintaining the MPEG video and audio synchronization code described in the patent scope item 13, wherein the virtual motion vector is (0, 0). Real-time frame video encoding method for MPEG pro Page 24 1224928 6. Encoding of patent application range to complement the insufficient grid, the method includes: inputting an imaginary prediction day lattice; performing discrete cosine conversion to generate a discrete cosine conversion coefficient; quantizing the discrete cosine conversion coefficient to produce A quantized discrete cosine transform coefficient; and / or variable-length encoding the quantized discrete cosine transform coefficient and an imaginary motion vector to output an imaginary day lattice. 17. The real-time video coding method of the virtual day frame as described in item 16 of the scope of patent application, wherein the pixel value of the virtual prediction frame is 0. 18. The real-time video coding method of the virtual diurnal as described in item 16 of the scope of patent application, wherein the virtual motion vector is (0,0). 19. The real-time video coding method for a virtual frame as described in item 16 of the scope of patent application, wherein the pixel value of the virtual prediction diurnal is 0; and the virtual motion vector is (0, 0). Page 24 1224928 _ Case No. 9113 V. Faming Ming (10) In the picture group, the sequence of the diurnal pattern is I P // P P // P P // P P # p P V. As far as the PAL specification is concerned, in each picture group with a length of 13 diurnal patterns, the sequence of diurnal patterns is IPP // PP // PP " PP //, where I is an I picture and P is a p picture , # Is a virtual day grid. Table 1 Encoding frames per second (Fp) Frame type output sequence 'NTSC Picture Group (GOP) length is 15 PAL Picture Group (GOP) length is 13 jppppppppppppp ^ IPPPPPPPPPPPP I PPPPPP ^ PPPPPP / ^ IPPPPPPPPPPPP 2 ? 25 ^ _ IPPPAPPPPAPPPP # IPPPPPPPPPPPP IPPVPPPVPPPAPP # I p P ppp // ppppp / z I? Β ?? β ?? β ?? β ?? β I ppp / zppp / zppp / z Ts3L— I ?? β? Β? β? β? β? β? I ?? β ?? β ?? β ?? β 1— Iβ? β? β? β? β? β? β? I? β ?? β ?? β? β? β ferz 1β? β? β? β? β? β? ββ I? β? β? β? β? β? β Iββ ^ ββ ^ β ^ β ^ β ^ ββ 1β? β? β? β? β? ββ- -i〇-l2 ___ Iββν ββ? ββ? ββ? ββ Iββ ^ ββ ^ β ^ β ^ ββ ββββγ > βββγ > βββγββ I ββ ^ ρ βμγ > ββ ^ > βββ ____ Iββββγβββββββββββββ Ιβββββββββββββββ I βββββββ β βββββ _ — — 9 ^ 3-Ιβββββββββββββββ 1ββββββββββββ does not support the selection of an appropriate output grid sequence. When the image source has an input grid (S41), you can follow the output day Column grid corresponding day type sequence P.18 1224928 Amendment Fifth, description of the invention (11) Video coding (S45). Continuing the above example, as far as the NTSC specification is concerned, when the video coding is performed, the Chapsu with PP #PP # according to the IPAPP: ί The video coding, that is, the composition of a picture group: the first Is the I graph, the second diurnal grid is the ρ graph, the third ^ grid, and the fourth book is the ρ graph. It is virtual den * 柊, below; i dihedral is ρ, and the sixth frame is virtual ΐ t & The video encoding process is repeatedly performed until μ ΐ. Figure 1 and Figure ρ are the conventional MP_message codes:: The coding method of the earth and virtual densities is as described above, and will not be repeated here; the output diurnal sequences of the two columns are just examples, any · Γ; ΓΙ: Set an appropriate V-out lattice sequence according to the experimental results or their special requirements. In addition, a picture of a plate can be modified equivalently to other factors. According to the embodiment of the present invention, since the operations of discrete cosine transform and quantization are 7 sub-estimation graphs, P graphs, and B graphs are fast. Therefore, the speed of Shixu day frame coding is far faster. If a frame loss is detected, then the frame will be lost ^^. In addition, if the virtual grid of the system is used to complement the previously defined output frame pattern order%, it will be ': when this is insufficient, the virtual day grid can also be used to fill it'. The format of the MPEG slot after the block code keeps the same number of video and audio, so as to ensure that the above description is only for example, and those skilled in the art can use the above-mentioned inventions for each: Anyone who is familiar with this without leaving its spirit and fan mouth. Any ^ example is modified equivalently, and the equivalent modification or change made to it: away from the spirit and scope of the invention, please be in the scope of patents. The text should be included in the attached application page 19 1224928 Case No. 911_ Brief description of the diagram 5. [Simplified description of the diagram] Figure 1 is a schematic diagram of a group of pictures composed of the conventional real-time encoding sequence of MPEG video. Figure 2a is a flowchart showing the preferred embodiment of the present invention. ^ ^ Real-time video encoding method of video synchronization. Maintenance of MpEG in a consistent embodiment. Figure 2b is a flowchart showing the best of 112a. a Simplified flow of the audio synchronization real-time video encoding method ^ & Example maintains MPEG image 3 as a flowchart showing the virtual book process in Fig. 2a. —Flat code method flow FIG. 4 is a flowchart showing the flow of another instant video encoding method of MPEG video and audio synchronization according to the present invention. ^ Symbol description of the maintenance elements of the embodiment 27 Detecting the diurnal program 28 Complementing the diurnal program 29 Trellis encoding program 31 Imaginary prediction diurnal 32 Discrete cosine conversion coefficient 33 Quantized discrete cosine transform 35 Virtual motion vector 41 MPEG video coding system