JP3556381B2 - Information multiplexing device - Google Patents

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an information multiplexing apparatus that packet-multiplexes an encoded video bit sequence, an encoded audio bit sequence, and another encoded bit sequence and transmits the packet.
[0002]
[Prior art]
MPEG-1 (ISO / IEC11172), which targets storage media with a relatively low bit rate, next-generation digital broadcasting, VOD (Video On Demand) or DVD (Digital) as international standards for compression encoding of moving images. MPEG-2 (ISO / IEC13818) and the like, which are expected to be applied to Video Disks, are known.
[0003]
In the MPEG encoding method, a method of independently encoding video, audio, and other data streams (data strings), packetizing the encoded data, and performing packet multiplexing is adopted. Further, between the transmitting side and the receiving side, synchronization is established by transmitting each time stamp of the packet arrival time, decoding time, and display time of the packet at a certain frequency or more.
[0004]
The time stamps of the decoding time and the display time usually differ depending on the encoding parameters of each image and audio. In the case where encoding is performed by being divided into encoded frames at regular intervals in the input order, these time stamps can be updated by always adding a constant difference value. However, in the MPEG video encoding, the display order and the encoding order are different, and the relationship may change dynamically. In such a case, it is impossible to update the time stamp by adding a certain value, and it is necessary to appropriately update the time stamp based on the header information of the input encoded data.
[0005]
Further, the transmitting side must select a packet multiplexing order and multiplex a timestamp so that synchronization is correctly established without causing a failure such as an input buffer overflow or an interflow on the receiving side. .
[0006]
Usually, in a multiplexing device constituted by hardware, these time stamps are calculated based on the time base of the multiplexing processing unit. The time base of the multiplexing processing unit may be synchronized with the clock of the transmission path or the input source, or may operate asynchronously with them. Usually, the configuration of a multiplexing system differs greatly depending on the synchronization / asynchronization of these clocks.
[0007]
[Problems to be solved by the invention]
In the MPEG coding method or the like, in order to calculate each time stamp of the decode time and the display time from only the header information of the input coded data, the delay of the maximum value of the shift between the display order and the coding order is required. , The analysis of the header information of the encoded data must be sequentially performed. In a transmission system that performs encoding, transmission, and decoding in real time, this delay amount cannot be ignored.
[0008]
A single time is used in all processes from encoding of information from each information source such as video and audio to obtain an encoded data stream, multiplexing them and obtaining a multiplexed bit stream. In the information multiplexing apparatus assuming synchronization with the base, the time, the number of coded bits, and the coding rate can be completely one-to-one. Therefore, the selection of the packet multiplexing order and the calculation of the time stamp and the like in the multiplexing device can be logically performed based on the virtual time base obtained from the encoded data stream.
[0009]
In such an information multiplexing apparatus, multiplexing processing can be realized only by software processing, and a low-cost and versatile system can be constructed with respect to dedicated hardware for real-time management.
[0010]
On the other hand, in multiplexing processing by software processing without real-time management, the throughput of the multiplexing processing is required to be at least sufficiently higher than the actual coding rate. Accordingly, the multiplexing process is performed in a burst manner, and a large jitter accompanies between the virtual time base and the real time base on software.
[0011]
Normally, this jitter needs to be absorbed by the transmission buffer. However, when encoding and decoding are performed using real-time transmission, it is required to minimize the buffer delay amount. Also, it is not preferable on the receiving side to make the input buffer delay for absorbing jitter unnecessarily large. Therefore, assuming that the input buffer delay is minimized on the receiving side and a normal decoding operation is started from the input time of the encoded data stream, because of a shift between the virtual time base of the multiplexing unit and the real time, If the transmission start time of the multiplexed bit stream is advanced, there is a possibility that an underflow occurs on the transmission path or the receiving side.
[0012]
Therefore, the present invention has been made in view of the above problems, Can reduce the multiplexing delay between video encoded data and audio encoded data An object of the present invention is to provide an information multiplexing device.
[0014]
[Means for Solving the Problems]
An information multiplexing apparatus according to the present invention includes a multiplexing unit that packetizes and multiplexes at least encoded data of a moving image and encoded data of audio, and multiplexed data obtained by the multiplexing unit. Output means for outputting, and input means for inputting an encoding parameter obtained when encoding moving image data, the multiplexing means, based on the encoding parameter input by the input means Calculating the time management information for decoding / reproducing the encoded data of the moving image, adding it to the packet of the encoded data of the moving image, and multiplexing the packet, thereby obtaining the time for decoding / reproducing the time stamp or the like. The multiplexing delay caused by analyzing the header information for calculating the management information can be reduced.
[0015]
Further, the information multiplexing apparatus of the present invention includes a plurality of buffers for temporarily storing at least the input coded data of the moving image and the coded data of the audio, and the coded data stored in each of the plurality of buffers. Multiplexing means for packetizing and multiplexing the data, and output means for outputting multiplexed data obtained by multiplexing by the multiplexing means, wherein the multiplexing means is temporarily stored in the buffer. Real-time management by software by determining the timing to start multiplexing processing based on the occupancy of the buffer of the coded data in which the transition of the occupancy of the buffer among the coded data can be a time reference. Can be easily performed, preventing underflow in the receiving buffer on the decoding side, and not requiring complicated hardware.
[0016]
Further, the information multiplexing apparatus of the present invention includes a plurality of buffers for temporarily storing at least the input coded data of the moving image and the coded data of the audio, and the coded data stored in each of the plurality of buffers. Multiplexing means for packetizing and multiplexing the data, and output means for outputting multiplexed data obtained by multiplexing by the multiplexing means, wherein the multiplexing means analyzes the encoded data. By counting the number of coded frames and determining the timing of starting the multiplexing process based on the counted value, real-time management by software can be easily performed and underflow in the receiving buffer on the decoding side is prevented. In addition, no complicated hardware is required.
Therefore, it is possible to provide an information multiplexing apparatus capable of easily real-time management of information encoding, transmission, and decoding and synchronous multiplexing of individually encoded information.
[0017]
BEST MODE FOR CARRYING OUT THE INVENTION
An embodiment of the present invention will be described with reference to the drawings.
(1st Embodiment)
FIG. 1 is a block diagram schematically showing a configuration of an information multiplexing device according to the first embodiment of the present invention.
[0018]
FIG. 1 shows a case where a moving image signal 10 of one channel, an audio signal of n channels, and a data signal of one channel are multiplexed.
The moving image (video) signal 10 and the plurality of audio signals 11-1 to 11-n are input to the moving image encoding device 15 and the audio encoding devices 16-1 to 16-n, respectively, and are encoded. Are input to the input buffers 23, 24-1 to 24-n, respectively.
[0019]
A data stream 14 that does not require encoding or has already been encoded is also input to the input buffer 25.
The encoded data temporarily stored in the input buffers 23, 24-1 to 24-n and the data temporarily stored in the input buffer 25 are respectively processed by the packetization processing units 33, 34-1 to 34-n, 35. The packets are individually packetized, and these packets are multiplexed by the multiplexing processing unit 43 to obtain a multiplexed bit stream. The multiplexed bit stream 44 is temporarily stored in an output buffer 45 and then transmitted at a predetermined timing, for example, via a network.
[0020]
The moving image (video) and audio signal packetization processing units 33, 34-1 to 34-n have respective encoding devices 15, 16-1 to 16-n when encoding moving images and audio. The obtained encoding parameters are input as side information 50 to 53.
[0021]
The side information can include an encoding bit rate, a frame encoding data length, a display frame rate, a display period of each frame, parameters related to a prediction structure of video encoding, and the like.
[0022]
The packetization processing units 33, 34-1 to 34-n perform packetization, calculation of a time stamp, determination of a multiplexing order, and the like using these side information.
[0023]
Further, a part of the side information can be extracted by performing grammatical analysis on the input encoded data.
FIG. 2 is for explaining the processing order of frames in MPEG video coding and their arrangement.
[0024]
In MPEG video coding, coding in which an intra-frame coded image (I picture), a forward prediction coded image (P picture), and a bidirectional prediction coded image (B picture) are combined is possible. However, when a B picture is included, the encoding order and the display order are different.
[0025]
FIG. 2 shows an example in which a prediction structure in which two B-pictures always enter between an I-picture and a P-picture is used.
In this case, on the decoding side, the I-picture and the P-picture are displayed after a lapse of three frame periods from the decoding time, during which the I-picture and the P-picture are stored in the frame memory of the decoding device and decoded as reference images of the predicted image Used for
[0026]
In the MPEG system multiplexing standard, when video and audio are multiplexed and transmitted, a time to be decoded (DTS: Decoding Time Stamp) and a time to be displayed (PTS: Presentation Time Stamp) are defined by the MPEG system standard. Multiplexed with the header information to be transmitted.
[0027]
In FIG. 2, the PTS of the I picture and the P picture are calculated by adding the DTS of the picture for a period of three frame periods. Further, the obtained PTS can be used as the DTS of the next encoded I picture or P picture.
[0028]
However, as shown in FIG. 3, the number of B-pictures sandwiched between I-pictures or P-pictures can be arbitrarily changed, and it is generally impossible to calculate PTS from DTS as described above. Normally, for a given I picture or P picture, when the next coded I picture or P picture is input, the number of B pictures included therebetween is determined. Therefore, in the moving picture packetization processor 33 shown in FIG. 1, the time stamp cannot be calculated at the same time as the input of the I or P picture, and the encoded data of the next encoded I or P picture Multiplexing delay until is obtained.
[0029]
In the information multiplexing device according to the first embodiment, the frame interval of an I picture or a P picture is input from the coding device 15 to the packetization processing unit 33 as side information, so that the coded data of each I picture or a P picture is input. Is obtained, the time stamp required for packetization can be easily calculated, and the packetization and multiplexing processing can be performed without causing the multiplexing delay described above.
[0030]
Further, when encoding is performed with the interval between I-pictures or P-pictures always constant, it is also possible to input the constant value to the packetization processing unit 33 before the start of encoding, for example.
[0031]
Further, in the MPEG2 video encoding, the period for displaying one encoded frame as shown in FIG. 4 can be arbitrarily set to two fields or three fields. This is used when a movie source having a frame cycle of 24 Hz is converted into a frame rate by a receiver having a frame cycle of 30 Hz and displayed. In this case, even when coding is performed with the interval between I pictures or P pictures always constant, the interval between DTS and PTS of each I picture or P picture does not become a fixed value. Occurs.
[0032]
In the information multiplexing apparatus according to the first embodiment, the frame interval of the I picture or the P picture is set as the number of fields in consideration of the display period of the encoded frame, and the side information is transmitted from the video encoding apparatus 15 to the packetization processing unit 33. As a result, packetization and multiplexing can be performed without causing the above-described multiplexing delay.
[0033]
FIG. 7 shows an example of the data structure of MPEG video encoded data. A sequence header relating to the entire video sequence, a picture group (GOP: Group Of Pictures) grouped into units of about 15 pictures starting with an I picture. , And a header of each picture and encoded data.
[0034]
The encoded data having the configuration shown in FIG. 7 is output from the moving image encoding device 15 and input to the packetization processing unit 33 via the input buffer 23.
7 is input from the input buffer 23 to the packetization processing unit 33, and the video encoding device 15 outputs the I-picture or P-picture frame interval or the frame A value in which the interval is represented by the number of fields in consideration of the display period of the encoded frame is input as side information, and a time stamp (DTS, PTS) required for packetization is calculated based on the side information. When the data sequence having the configuration shown in FIG. 7 is packetized, the calculated time stamp is added to the data sequence and output to the multiplex processing unit 43.
(Second embodiment)
FIG. 5 is a block diagram schematically showing a configuration of an information multiplexing device according to the second embodiment of the present invention.
[0035]
5, the same parts as those in FIG. 1 are denoted by the same reference numerals, and different parts will be described. That is, the information multiplexing device shown in FIG. 5 includes a control unit 60 for controlling the moving image and audio coding devices 15, 16-1 to 16-n and the packetization processing units 33, 34-1 to 34-n. Is provided. The side information obtained at the time of encoding in the moving picture and audio encoding devices 15 and 16-1 to 16-n shown in FIG. 1 to 34-n.
(Third embodiment)
FIG. 6 is a block diagram schematically showing a configuration of an information multiplexing device according to the third embodiment of the present invention.
[0036]
6, the same parts as those in FIG. 1 are denoted by the same reference numerals, and different parts will be described. That is, the information multiplexing apparatus shown in FIG. 6 multiplexes, among the side information shown in the first embodiment, data and the like particularly necessary for calculating the time stamp into each coded data. 15, 16-1 to 16-n.
[0037]
In the data structure of the MPEG video coded data shown in FIG. 7, for example, user data can be multiplexed following each header. An example of multiplexing of user data in that case will be described with reference to FIGS.
[0038]
In FIG. 8, user data is multiplexed following the sequence header. The user data includes a parameter related to the interval between the I-picture or the P-picture, and the packetization processing unit calculates a time stamp according to the parameter. If the interval between I-pictures or P-pictures is constant throughout the sequence of one unit and the above-described frame rate conversion is not used, it is possible to reduce the multiplexing delay using this method.
[0039]
In FIG. 9, user data is multiplexed following the GOP header. The user data includes a parameter related to the interval between the I-picture or the P-picture, and the packetization processing unit calculates a time stamp according to the parameter. When the interval between I-pictures or P-pictures is variable in GOP units and the above-described frame rate conversion is not used, multiplexing delay can be reduced using this method.
[0040]
In FIG. 10, user data is multiplexed following the picture header. The user data includes a parameter relating to the above-mentioned interval between the I picture and the P picture, and the packetization processing unit 43 calculates a time stamp according to the parameter. In this case, it is possible to always calculate a time stamp for any prediction structure without multiplexing delay.
[0041]
As described above, according to the first to third embodiments, in a multiplexing method such as an MPEG system that requires multiplexing of time stamps, information such as a moving image and audio is encoded by the encoding device 15. 16-1 to 16-n, a parameter indicating the relationship between the input order and the encoding order, a parameter indicating the display duration of the frame, and the like, obtained as side information, are used as the side information. 34-1 to 34-n, and the packetization processing sections 33 and 34-1 to 34-n analyze the header of the encoded data by sequentially calculating time stamps according to the input parameters. As compared with the conventional method of calculating a time stamp based on the information obtained as a result, multiplexing processing with low delay can be realized. Normally, when calculating the time stamp by analyzing the encoded data, a delay of several frames occurs in principle, but in the present invention, the principle delay for calculating the time stamp does not occur.
[0042]
The side information including the encoding parameter may be input from a path independent of the encoded data, but the hardware may be simplified by multiplexing the encoded information in advance and inputting it.
[0043]
If the coding parameters are dynamically changed in frame units, they are sequentially input to the packetization processing unit. If the coding parameters are constant in sequence units, the packetization processing is performed only once for one sequence. This can be realized by inputting the information to the section.
[0044]
(Fourth embodiment)
FIG. 11 is a block diagram schematically showing a configuration of an information multiplexing device according to the fourth embodiment of the present invention. 11, the same parts as those in FIG. 1 are denoted by the same reference numerals, and different parts will be described.
[0045]
In FIG. 11, the packetization processing units 33, 34-1 to 34-n, 35 and the multiplexing processing unit 43 sequentially perform multiplexing of time stamps and control of the multiplexing order according to a virtual time base.
[0046]
The control unit 48 monitors the buffer occupancy of the coded data input to the input buffers 23, 24-1 to 24-n, 25, and sends the multiplexed data to the multiplexing processing unit 43. It controls the timing of the start of transmission of the multiplexed bit stream for notifying the start time. For example, multiplex the timing at which the occupancy of each input buffer becomes the buffer occupancy corresponding to the data transfer start timing logically derived from the input buffer occupancy of the encoded audio data input at a fixed rate. The conversion processing unit 43 is notified.
[0047]
FIG. 12 more specifically shows the configuration of a main part of the information multiplexing apparatus of FIG. 11 (a part surrounded by a dotted line in FIG. 11 and hereinafter referred to as a multiplexing unit 47).
The moving image and audio coded data 19, 20-1 to 20-n and other data 14 coded and output by the coding devices 15, 16-1 to 16-n are coded data input devices, respectively. 100, 101-1 to 101-n, and 102, and further input to the main memory 125 via the data bus 120 by the DMA controllers 110, 111-1 to 111-n, and 112, respectively. The data is transferred to the buffers 130, 131-1 to 131-n, 132.
[0048]
The transferred data is multiplexed into a multiplexed bit stream by the CPU 140 and software 141 which control the processing operations corresponding to the functions of the packetization processing units 33, 34-1 to 34-n and 35 and the multiplexing processing unit 43 in FIG. And written to the output buffer 150 in the main memory 125.
[0049]
The multiplexed bit stream written in the output buffer 150 is output from the output device 162 to the transmission path via the data bus 120 by the DMA controller 161.
[0050]
Here, the CPU 140 and the multiplexing software 141 update the virtual time base by counting the number of output bits of the multiplexed bit stream, and in a fifth embodiment described later, The processing of counting the number of encoded frames and updating the time stamp is performed. The CPU 140 and the multiplexing software 141 constitute a control unit 48 in FIG. 11 and a control unit 349 in FIG. 18 described later.
[0051]
13 and 14 are diagrams for explaining a synchronization method using a buffer. 13, 170 and 174 are encoded data, 171 is a transmission (output) buffer, 172 is a transmission path, and 173 is a reception (input) buffer. The transmission buffer and the reception buffer are also called a smoothing buffer.
[0052]
The encoded data 170 is input to the transmission buffer 171, temporarily stored, delayed, and then input to the reception buffer 173 via the transmission path 172. The transmitted encoded data is also output after being similarly delayed in the reception buffer 173.
[0053]
Here, by controlling the sum of the delay amount in the transmission buffer 171 and the delay amount in the reception buffer 173 to be always constant, the encoded data 170 is input to the transmission buffer 171 and then output from the reception buffer 173. The total amount of delay until the data is transmitted is always constant, and synchronous data transfer between transmission and reception is possible.
[0054]
FIG. 14 is a timing chart of transmission / reception when the smoothing buffer of FIG. 13 is used.
In FIG. 14, for example, image data of one frame of a moving image is encoded at a predetermined interval, encoded data having a different code amount is generated for each frame, and sequentially transmitted to the transmission buffer 171.
[0055]
Reference numeral 190 denotes a stay time in the transmission buffer 171, and 192 denotes a stay time in the reception buffer 173. According to the principle of the smoothing buffer, the residence time in the transmission buffer 171 and the residence time in the reception buffer 173 may vary with time, respectively, but the total delay between transmission and reception, which is the sum thereof, is always constant. Is controlled as follows.
[0056]
In an encoding method in which the code amount changes in frame units such as MPEG video coding, the transmission code amount is smoothed using the transmission / reception buffer 171.
As described above, on the transmission side, after the encoded data is input, it is output after being delayed by the transmission buffer 171. Since the sizes of the transmission buffer 171 and the reception buffer 173 are both defined finite values, the amount of delay in each buffer must be strictly controlled so as not to cause underflow and overflow in each buffer. .
[0057]
FIG. 15 shows the input buffer occupancy in the multiplexing processing unit 43 of each encoded data on the transmission side at the time of startup. Each of the encoded data of the image and the audio is input to the input buffers 130, 131-1 to 131-n with an encoding delay specific to the encoding method and the encoding parameter, and further, a time corresponding to the transmission buffer delay described above. And then multiplexed and transmitted.
[0058]
In FIG. 15, reference numeral 201 denotes encoded image data, and reference numerals 202 and 203 denote encoded audio data, respectively, showing an example in which these three data are multiplexed. FIG. 15 shows a transition from the initial state of the input buffer occupancy when all the encoded data are input to the multiplexing unit 47 at a constant rate. Here, the encoded image data has a transmission-side delay, so transmission starts at the time 204, and the audio encoded data 202 and 203 are multiplexed and transmitted from the timings 206 and 207, respectively. You have to start.
[0059]
When the code amount per frame fluctuates like coded data of a moving image, the input interval of a frame to be coded, the code amount obtained by coding the frame, the coding rate, and the like are completely one-to-one. When the function of the multiplexing unit 47 is realized by software, it is usually difficult to manage the real time (real time) only by performing the smoothing using the buffer described above.
[0060]
In other words, if the transmission side starts transmission ahead of time (without performing the timing control of the transmission start of the multiplexed bit stream of the present invention), the reception of the multiplexed bit stream is delayed due to the difference between the virtual time base of the multiplexing unit 47 and the real time. The possibility of underflow on the side.
[0061]
As described above, real-time management means that in the case of encoded data of a moving image, the actual decoding of the actual picture by the decoder on the receiving side is also performed by considering the code amount of one frame (screen) which is one encoding unit. It can also be said to be a management control of the transmission timing of the encoded data on the transmitting side such that the encoded data can be decoded while maintaining the interval.
[0062]
Therefore, the input buffer occupancy of encoded data of a specific moving image or audio, for example, audio encoded data 202 which is always input at a fixed bit rate is monitored, and logically calculated from the fixed bit rate. Multiplexing that starts multiplexing processing and transmission from the time when the buffer occupancy (for example, input buffer occupancy 205 in FIG. 15) corresponding to the time of the derived data transfer start timing (for example, time 204 in FIG. 15). Of the transmission start of the coded bit stream. This makes it possible to prevent the underflow in the reception buffer as described above.
[0063]
FIG. 16 shows the transition from the initial state of the input buffer occupancy when the coded data 210 of the moving image is input to the multiplexing unit 47 in a burst manner, and FIG. 17 shows the case where the coded data 220 of the moving image is multiplexed. 9 shows a transition from the initial state of the input buffer occupancy when the input rate is input to the conversion unit 47 while changing over time.
[0064]
If the input is not at a fixed rate, it is difficult to estimate the actual time from the buffer occupancy, but if there is accompanying audio encoded data (212 in FIG. 16 and 222 in FIG. 17) input at a fixed rate, Similarly to the description of FIG. 15, the data transfer start timing logically derived from the input buffer occupancy of the multiplexing unit 47 for the encoded data (for example, time 214 in FIG. 16 and time 224 in FIG. 17). ) (For example, the input buffer occupancy 215 in FIG. 16 and the input buffer occupancy 225 in FIG. 17), the transmission of the multiplexed bit stream for starting the multiplexing process and transmission starts. Is performed.
[0065]
(Fifth embodiment)
In the configuration of the multiplexing unit 47 shown in FIG. 12, since the input buffers 130, 131-1 to 131-n, and 132 of the respective encoded data are configured on the main memory 125, the data is transparent from software. It will be. Therefore, instead of measuring the occupation amount of the encoded data in the input buffer, the number of encoded frames may be counted, and the current time for controlling the multiplexing start timing may be derived.
[0066]
FIG. 18 is a block diagram schematically showing the configuration of the information multiplexing device according to the fifth embodiment.
FIG. 18 shows a case where a plurality of programs as a set of a moving image and a sound are simultaneously encoded and multiplexed. Here, the programs Ch1 and Ch2, which are composed of moving images and sounds encoded in real time, and the program Ch3, which is encoded in advance and recorded on the storage medium, are simultaneously multiplexed with a total of 3 Ch programs. is there. The detailed configuration of the main part of the information multiplexing device of FIG. 18 (a portion surrounded by a dotted line in FIG. 18 and hereinafter referred to as a multiplexing unit 347) is the same as that of FIG.
[0067]
The moving image (video) signal 300 and the audio signal 301 and the moving image signal 302 and the audio image 303 are input to the moving image encoding device 315, the audio encoding device 316, the moving image encoding device 317, and the audio encoding device 318, respectively. The encoded data is input to the input buffers 323, 324, 325, and 326 as individual encoded data.
[0068]
The program data 304 read from the storage medium is similarly input to the input buffer 327.
Further, a data stream 305 that does not require encoding or has already been encoded may be similarly input to the input buffer 351.
[0069]
Each of the encoded data temporarily stored in each input buffer is individually packetized by the packetizing processing units 333 to 337 and 353, and these packets are multiplexed by the multiplexing processing unit 343 to be multiplexed. A bit stream is obtained. The multiplexed bit stream 344 is temporarily stored in the output buffer 345 and then transmitted at a predetermined timing, for example, via a network.
[0070]
Each of the packetization processing units 333 to 337 and 353 includes a data analysis unit 333a, 334a, 335a, 336a, 337a, and 353a that extracts and analyzes a header portion such as encoded data input from each input buffer. ing.
[0071]
The control unit 349 counts the number of encoded frames of the moving image based on the analysis result of the data analysis unit, and controls the multiplexing start timing, that is, calculates the real time, and determines the multiplexing / transmission timing. The multiplex processing unit 343 is notified. If the time bases of the programs Ch1 to Ch3 are independent, the virtual clock in the multiplexing software is calculated independently, and the multiplexing start timing is controlled for each program. Further, in a program having a common time base, the virtual clock in the multiplexing software is also made common, and the multiplexing start timing is controlled.
[0072]
As described above, according to the fourth embodiment, when synchronously multiplexing individually encoded encoded data such as moving image and audio and other data, the control unit 48 controls each input signal. The occupancy of the data in the buffer is monitored, and the time of the data transfer start timing logically derived from the fixed bit rate for the specific image or audio encoded data input at the fixed bit rate (for example, in FIG. 15, The multiplex processing unit 43 notifies the multiplex processing unit 43 of the point in time when the buffer occupancy amount (for example, the input buffer occupation amount 205 in FIG. 15) corresponding to the time 204) is reached. By starting the multiplexing processing and transmission according to the above, it is possible to realize multiplexing processing that can withstand real-time transmission using only software. By realizing the multiplexing unit 47 only by software, it is not necessary to perform timing control by complicated hardware, and the system configuration can be simplified. Also, customization by software becomes easy, and it is possible to flexibly cope with various applications.
[0073]
Further, according to the fifth embodiment of the present invention, when synchronously multiplexing individually encoded encoded data such as moving image and audio and other data, the control unit 349 controls each packetization processing unit. , The actual time is calculated by counting the number of encoded frames of the moving image based on the analysis result of the data analysis unit, and the actual time is notified to the multiplex processing unit 343. By starting the multiplexing process and the transmission in accordance with the timing notified from, the multiplexing process that can withstand real-time transmission can be realized only by software. By realizing the multiplexing unit 347 only by software, there is no need to perform timing control using complicated hardware, and the system configuration can be simplified. Also, customization by software becomes easy, and it is possible to flexibly cope with various applications.
Furthermore, it is also effective to use the first to fifth embodiments in appropriate combination.
[0074]
【The invention's effect】
As described above, according to the present invention, by inputting the encoding parameters related to the decoding time and display time of the moving image, audio, and other encoded data to the multiplexing unit independently of the header information. In addition, it is possible to provide an information multiplexing apparatus capable of reducing a multiplexing delay generated for calculating a time stamp by analyzing header information.
[0075]
Further, according to the present invention, a value obtained by converting the coded data of a specific image or audio input at a fixed bit rate from the input buffer occupation amount or the number of coded frames in the input buffer into real time Control of the start of multiplexed bit stream transmission by using multiplexing, real-time management can be easily performed, underflow of the transmission path or the receiving buffer on the decoding side can be prevented, and information multiplexing that does not require complicated hardware is performed. Equipment can be provided.
As described above, according to the information multiplexing apparatus of the present invention, real-time management of information encoding, transmission, and decoding, and synchronous multiplexing of individually encoded information can be easily performed.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of an information multiplexing device according to a first embodiment of the present invention.
FIG. 2 is a diagram for explaining a frame processing order and its arrangement in MPEG video encoding, in which the number of B pictures is fixed.
FIG. 3 is a diagram for explaining the processing order of frames involved in MPEG video encoding and their arrangement, showing a case where the number of B pictures is variable.
FIG. 4 is a diagram for explaining a display period of each encoded frame in MPRG video encoding.
FIG. 5 is a block diagram showing a configuration of an information multiplexing device according to a second embodiment of the present invention.
FIG. 6 is a block diagram showing a configuration of an information multiplexing device according to a third embodiment of the present invention.
FIG. 7 is a diagram showing an example of a data structure of MPEG video encoded data in the information multiplexing device according to the first embodiment and the second embodiment.
FIG. 8 is a diagram showing an example of a data structure of MPEG video encoded data in the information multiplexing device according to the third embodiment.
FIG. 9 is a view showing another example of the data structure of MPEG video encoded data in the information multiplexing device according to the third embodiment.
FIG. 10 is a view showing still another example of the data structure of MPEG video encoded data in the information multiplexing device according to the third embodiment.
FIG. 11 is a block diagram showing a configuration of an information multiplexing device according to a fourth embodiment of the present invention.
FIG. 12 is a block diagram showing a configuration of a main part of the information multiplexing device in more detail.
FIG. 13 is a view for explaining a synchronization method using a transmission / reception buffer (smoothing buffer).
FIG. 14 is a diagram for explaining the operation of the smoothing buffer.
FIG. 15 is a diagram for explaining multiplexing processing and transmission start timing control according to the present invention, and shows an initial state of an input buffer occupancy when all encoded data are input to a multiplexing unit at a constant rate. Shows the transition from.
FIG. 16 is a diagram for describing multiplexing processing and transmission start timing control according to the present invention, and illustrates an initial state of an input buffer occupancy when coded data of a moving image is input to a multiplexing unit in a burst manner. The transition from the state is shown.
FIG. 17 is a diagram for describing multiplexing processing and transmission start timing control according to the present invention; FIG. 17 is an input buffer in the case where encoded data of a moving image is input to the multiplexing unit while the input rate varies with time; The transition of the occupancy from the initial state is shown.
FIG. 18 is a block diagram showing a configuration of an information multiplexing device according to a fifth embodiment of the present invention.
[Explanation of symbols]
15: Moving picture coding device, 16-1 to 16-n: Voice coding device, 23, 241-1 to 24-n, 25: Input buffer, 33, 34-1 to 34-n, 35: Packetization Processing unit, 43: multiplex processing unit, 44: multiplexed bit stream, 45: output buffer, 47: multiplex unit, 48, 60: control unit, 347: multiplex unit.

Claims (6)

First input means for inputting first encoded data which is MPEG encoded data of a moving image and second encoded data which is encoded data of audio;
For each I picture / P picture in the first encoded data, input the number of fields representing the display period of the B picture existing between the I picture / P picture and the next I picture / P picture. Second input means for performing
The I picture / P picture based on the number of fields input by the second input means corresponding to each I picture / P picture in the first encoded data input by the first input means Is calculated, and the first encoded data and the second encoded data input by the first input unit are packetized, respectively, and the time stamp is added to the packet of the first encoded data. Packetizing means for adding a stamp;
Multiplexing means for multiplexing the packet of the second encoded data and the packet of the first encoded data to which the time stamp is added;
Output means for outputting multiplexed data obtained by multiplexing in the multiplexing means,
An information multiplexing device comprising: Display of MPEG encoded data of a moving image and a B picture corresponding to each I picture / P picture in the MPEG encoded data and existing between the I picture / P picture and the next I picture / P picture First input means for inputting first encoded data including the number of fields indicating a period, and second encoded data which is audio encoded data, respectively;
Calculating a time stamp of the I picture / P picture based on the number of fields corresponding to each I picture / P picture in the first coded data input by the first input means; Packetizing means for packetizing the first coded data and the second coded data input by the first input means, and adding the time stamp to a packet of the first coded data;
Multiplexing means for multiplexing the packet of the second encoded data and the packet of the first encoded data to which the time stamp is added;
Output means for outputting multiplexed data obtained by multiplexing in the multiplexing means,
An information multiplexing device comprising: For each GOP (group of pictures) in the MPEG encoded data of the moving picture and the MPEG encoded data, there exists between one I picture / P picture and the next I picture / P picture included in the GOP. The first coded data including the number of fields common to each I picture / P picture in the GOP and the second coded data that is audio coded data are input. First input means for performing
The time stamp of each I picture / P picture in the GOP is calculated based on the number of fields corresponding to each GOP in the first coded data input by the first input means, and Packetizing means for packetizing the first coded data and the second coded data input by the first input means, and adding the time stamp to a packet of the first coded data;
Multiplexing means for multiplexing the packet of the second encoded data and the packet of the first encoded data to which the time stamp is added;
Output means for outputting multiplexed data obtained by multiplexing in the multiplexing means,
An information multiplexing device comprising: One I picture / P picture and the next I picture included in the MPEG encoded data of a moving image and one unit sequence including a plurality of GOPs (groups of pictures) in the MPEG encoded data are included in the sequence. / P picture, the first encoded data including the number of fields common to each I picture / P picture in the sequence, and the first encoded data that is audio encoded data. First input means for respectively inputting the two encoded data;
Calculating a time stamp of each I picture / P picture in the sequence based on the number of fields corresponding to each sequence in the first coded data input by the first input means; Means for packetizing each of the first encoded data and the second encoded data input by the first input means, and adding the time stamp to a packet of the first encoded data;
Multiplexing means for multiplexing the packet of the second encoded data and the packet of the first encoded data to which the time stamp is added;
Output means for outputting multiplexed data obtained by multiplexing in the multiplexing means,
An information multiplexing device comprising: A buffer that temporarily stores the second encoded data input from the first input unit at a constant bit rate;
The multiplexing means starts the multiplexing process every time the elapsed time determined from the data amount of the second encoded data stored in the buffer and the bit rate reaches a predetermined time. The information multiplexing device according to claim 1, wherein The apparatus further comprises means for counting the number of encoded frames included in the first encoded data input from the first input means,
2. The information multiplexing apparatus according to claim 1, wherein the multiplexing unit starts multiplexing processing every time the number of encoded frames reaches a number corresponding to a predetermined elapsed time.

Images