JPH10271463A - Data multiplexer and method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To multiplex audio and video data by a simple algorithm without using a floating decimal point arithmetic operation so as not to cause a reception buffer to be overflowed without taking a residual storage capacity into account. SOLUTION: A CPU 424 controls a data quantity generated by a video encoder 20 so as to distribute video data to video frames without causing a fraction. The CPU 424 calculates a data quantity A of audio data to be multiplexed for each frame and a data quantity B of audio data to be multiplexed for each of a prescribed number of frames such as 30 frames based on a data rate of audio data generated by the video encoder 24, adds '1' to the data quantity A depending whether or not a data quantity C of the audio data that are multiplexed for 30 frames is more than the data quantity B to calculate a data quantity D. Moreover, the CPU 424 controls each component of a data multiplexer 2 to multiplex the video data by a prescribed quantity with the data quantity D calculated by each frame.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data multiplexing apparatus for multiplexing video data and audio data (audio / video data) and generating a transport stream for transmission, and a method therefor.

[0002]

2. Description of the Related Art Digital television broadcasting which compresses and encodes video data and audio data (audio / video data) according to an MPEG system or the like, multiplexes them into predetermined transmission packets, generates a transport stream, and distributes the transport stream to viewers. Has been put to practical use. When multiplexing audio / video data, it is necessary to consider the remaining buffer capacity of the decompression decoding device. That is, on the transmitting side, the receiving buffer (4 in case of audio,
Both overflow and underflow in KB)
It is necessary to multiplex audio / video data in a way that does not occur.

However, even if the reception buffer does not fail, if the data transmission interval is too wide, for example, if the time interval of one second is large in the audio data, the audio data cannot be normally reproduced on the receiving side. Possibilities arise. In addition, an algorithm for planning compression / encoding and multiplexing of audio / video data so that the reception buffer does not fail is complicated, and requires a long floating point operation because of the need for floating-point arithmetic.

[0004]

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems of the prior art. In digital television broadcasting equipment and the like, the remaining storage capacity of a reception buffer of a decompression decoding device on the receiving side is reduced. Without consideration, and
With a simple algorithm of only integer operation without using floating point operation, so as not to cause failure in the reception buffer,
It is an object of the present invention to provide a data multiplexing apparatus and a method thereof capable of multiplexing audio / video data into transmission packets and generating a transport stream. Also, the present invention provides a data multiplexing apparatus capable of always multiplexing audio / video data into transmission packets and generating a transport stream so that audio / video data can be normally reproduced on a receiving side. And a method thereof.

[0005]

In order to achieve the above object, a data multiplexing device according to the present invention provides a data multiplexing apparatus for generating video data generated in each video frame period of video data by an equal data amount so as not to generate a fraction. A video target data amount calculation unit that calculates a target data amount that satisfies a condition that can be distributed at a time; a video data generation unit that generates video data having a data amount equal to the calculated target data amount; For each different audio frame period, audio data generation means for generating audio data of a predetermined data amount, audio data distribution means for distributing the generated audio data by a data amount substantially equal to each of the video frame periods, For each video frame period,
Multiplexing means for multiplexing the video data having the same data amount and the allocated audio data into a predetermined transport stream.

[0006] Preferably, the video data generating means compresses the uncompressed video data to generate the video data having a data rate suitable for the target data amount.

Preferably, the audio data distribution unit includes an audio target data amount calculation unit that calculates a target data amount of the audio data to be multiplexed during a predetermined number of the video frame periods, In the past, audio data counting means for counting the data amount of the audio data multiplexed during the predetermined number of video frame periods, the calculated target data amount of the audio data, and the counted data of the audio data Audio data distribution amount calculating means for calculating a data amount of the audio data to be allocated to the video frame cycle based on the amount.

Preferably, the apparatus further comprises additional data generating means for generating additional data to be added to the audio data and the video data, and the multiplexing means further converts the generated additional data into the predetermined transport stream. Multiplex.

The video target data amount calculating means generates the video data in such a manner that the video data having the same data amount with no fraction in byte units can be allocated to each frame period (video frame period) of the video data. The means calculates a target data amount indicating a data amount of the video data to be generated for each predetermined unit time (for example, a time (1 second) for 30 video frame periods).

The video data generating means compresses and encodes, for example, the uncompressed video data in accordance with the MPEG method, and compresses the compressed video data having a data amount equal to the target data amount calculated by the video target data amount calculating means every unit time. Generate.

The audio data generating means compresses and encodes, for example, uncompressed audio data in accordance with the MPEG system, and generates a fixed amount of compressed audio data for each sampling cycle of the audio data. The additional data generation means generates, for example, additional data (private data) used when multiplexing subtitles in a plurality of languages.

The audio data distribution means calculates, for example, a target data amount of audio data indicating the data amount of audio data to be multiplexed during the unit time (audio target data amount calculation means). Next, the audio data distribution unit counts, for each video frame period, the amount of compressed audio data multiplexed by the multiplexing unit during the unit time in the past of the video frame period (audio data counting unit) .

Further, the audio data distribution means includes, for example,
Depending on whether the counted data amount of the audio data is larger than the calculated target data amount of the audio data, the data amount of the audio data to be multiplexed during the unit time is determined by the video frame included in the unit time. Divide by the number of periods, add 1 to the data amount obtained by rounding down the decimal point, and calculate the data amount of audio data to be allocated to the video frame period (audio data allocation amount calculation means).

The multiplexing means converts the generated video data into
In synchronization with the video frame period, for each video frame period, an equal amount of compressed video data, audio data of the data amount allocated to the video frame period, and the generated additional data are multiplexed into transmission packets, Generate a transport stream.

Further, the data multiplexing method according to the present invention comprises:
A target data amount that satisfies the condition that the video data generated in each video frame period of the video data can be distributed by an equal data amount so that no fraction occurs is calculated, and a target data amount equal to the calculated target data amount is calculated. Generate video data, for each audio frame period different from the video frame period, generate audio data of a predetermined data amount, and distribute the generated audio data by a data amount substantially equal to each of the video frame periods, Video data of the same data amount for each video frame period;
Then, the allocated audio data is multiplexed into a predetermined transmission packet to generate a transport stream.

Preferably, the uncompressed video data is compressed,
The video data having a data rate matching the target data amount is generated.

Preferably, a target data amount of the audio data to be multiplexed during a predetermined number of video frame periods is calculated, and the target data amount is multiplexed during the predetermined number of video frame periods in the past of the video frame period. The data amount of the audio data to be distributed to the video frame period based on the target data amount of the audio data calculated by counting the data amount of the converted audio data, and the calculated data amount of the audio data. Is calculated.

Preferably, additional data to be added to the audio data and the video data is generated, and the generated additional data is further multiplexed into the predetermined transport stream.

[0019]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS First Embodiment Hereinafter, a first embodiment of the present invention will be described.

The diagram 1 of a data multiplexing apparatus 1 is a diagram showing a configuration of a data multiplexing apparatus 1 according to the present invention shown as a first embodiment. As shown in FIG.
The data multiplexing device 1 includes a video encoder (video encoder).
er) 20, audio encoder (audio encoder) 24
And a subtitle encoder 28 and a multiplexer 10. The multiplexer 10 includes a buffer memory 100 and a transport stream generator.
ker) 102.

With these components, the data multiplexing apparatus 1 uses the video encoder 20, the audio encoder 24, and the subtitle encoder 28 to encode a movie or music promotion video stream and audio stream, respectively. ) And subtitle stream (subti
tle stream) to generate a transport stream.

Video Encoder 20 The video encoder 20 converts video data input from an external device (not shown) such as an editing device into MPEG2
The data is compressed and encoded by a system or the like, and fixed-rate encoding is performed to generate a video stream having substantially the same data amount at a relatively long fixed time interval, and output to the buffer memory 100 of the multiplexer 10. Therefore, video encoder 2
0, the data rate of each video frame of the video stream generated on average is almost constant.

Audio Encoder 24 The audio encoder 24 compresses and encodes audio data input from an external device by, for example, the MPEG2 method or the like, and generates audio frames [24 ms, 26.1 ms, 36 m
s = sampling period of audio data (1152 / 48k)
Hz, 1152 / 44.1 kHz, 1152/32 kHz
z)], an audio stream having the same data amount is generated and output to the buffer memory 100 of the multiplexing device 10.

Subtitle encoder 28 The subtitle encoder 28 encodes subtitle data input from an external device for generating subtitle data by linear quantization processing and fixed-length encoding processing, and outputs the encoded data as a subtitle stream to the multiplexer 1.
0 is output to the buffer memory 100.

Buffer Memory 100 The buffer memory 100 of the multiplexer 10 buffers the video stream, audio stream, and subtitle stream input from the video encoder 20, the audio encoder 24, and the subtitle encoder 28, respectively. Output to 102. The transport stream generation device 102 is constituted by, for example, a high-speed computer using a microprocessor (CPU) and the like.
The video stream, the audio stream, and the subtitle stream (each also referred to as an elementary stream) buffered in the buffer memory 100 are read out at a predetermined timing to perform scheduling, and a header or the like is added according to the transport stream transmission format. To generate and output a transport stream.

According to the data multiplexing apparatus 1 shown as the first embodiment, audio data such as a movie, video data and subtitle data are compression-encoded by the MPEG system or the like, and these compression-encoded data are multiplexed. To generate and output one transport stream.

Second Embodiment Hereinafter, a second embodiment of the present invention will be described.

In the data multiplexing apparatus 1 shown as the first embodiment, the buffer memory 100 stores all the elementary streams once in the buffer memory 100, and furthermore, the CPU of the transport stream generating apparatus 102
Performs processing such as scheduling and addition of a header to generate a transport stream. As described above, the method of generating a transport stream by directly processing the words (bytes) of the elementary stream via the data bus by the CPU performing the processing is performed by the transport stream. High-speed processing cannot be performed due to a bus neck or the like occurring in the data bus of the generation device 102, and the method cannot be applied to generation of a high-speed transport stream.

A data multiplexing apparatus 2 according to the present invention shown as a second embodiment is an apparatus for solving the problem of the data multiplexing apparatus 1, and is a control system for scheduling elementary stream multiplexing. And a multiplexing system that directly multiplexes elementary streams, thereby enabling high-speed generation of a transport stream.

FIG. 2 shows a data multiplexer 2 according to the present invention.
FIG. 3 is a diagram showing the configuration of FIG. FIG. 3 is a diagram showing a configuration of the video encoder 20 shown in FIG. FIG. 4 is a diagram showing a configuration of the audio encoder 24 shown in FIG.

As shown in diagram 2 of the data multiplexer 2, the data multiplexer 2 is composed of a video encoder 20, audio encoder 24, the subtitle encoder 28, a multiplexing system 12 and control system 42.

The multiplexing system 12 includes FIFO memories 32a, 32b, 32c for input, and a first switch circuit (switch).
34, a second switch circuit 36, a FIFO memory 38 for output, and a SCSI (small computer system interface).
ce) An interface circuit (SCSIIF circuit) 40 is provided. The control system 42 includes a data size counting interface circuit (data size IF) 30a, 30
b, 30c, RAM 430, Ethernet interface circuit (ether-net interface; ENIF circuit) 42
0, serial interface circuit (serial interfac
e; SIF circuit) 422, CPU 424, processing RAM
426 and a control data RAM 428 are connected to each other via a CPU bus.

[0033] Video encoder 20 video encoder 20, as shown in FIG. 3, the addition circuit 202, DCT circuit 204, quantization circuit (The quantize) 20
6. Inverse quantization circuit (I quantize) 208, inverse DCT circuit (I
DCT) 210, addition circuit 212, frame memory circuit 2
16, a variable length coding circuit (VLC circuit) 218, a bit rate control circuit (bit rate control) 220, and a variable length code buffer (VLC buffer) 222.

The CPU 424 controls the bit rate control circuit 2
Target data amount video_rate_ta for 20
Set rget. The bit rate control circuit 220
Based on the data amount of the video stream (MPEG Video stream) actually generated by the variable length coding circuit 218, the data amount after the compression encoding is set to the set target data amount vid.
The quantization circuit 206 is controlled so as to be equal to eo_rate_target.

The target data amount video_ of the CPU 424
By setting the rate_target and controlling the bit rate control circuit 220, each component of the data multiplexing apparatus 2 compresses and encodes the video data VideoIN input from the external device according to the MPEG2 system, and sets the target data amount vi.
A video stream having a data amount substantially equal to “deo_rate_target” is generated and output to the data size IF 30 a and the FIFO memory 32 a of the data multiplexing device 2.

The audio encoder 24 audio encoder 24, as shown in FIG. 4, sub-band analysis filter bank (SAFB) 240, linear quantization circuit 242, a bit compression circuit 244, FFT (fast
fourier transfer) circuit 246, psychological auditory model 24
8, dynamic bit allocation circuit 250, scale factor selection information storage circuit 252, scale factor extraction circuit 2
54, a side information encoding circuit 256 and a bit stream generating circuit 258. The audio data (Audio input) input from an external device is compression-encoded by the MPEG2 system, and the audio stream (MPEG Audi
o stream) is generated and output to the data size IF 30b and the FIFO memory 32b of the data multiplexer 2.

Subtitle encoder 28 The subtitle encoder 28 encodes private data such as subtitle data input from an external device, as in the first embodiment. The subtitle encoder 28 may directly receive the encoded subtitle data from an external device. For example, the ENIF circuit 420
N or the SIF 422 receives the encoded subtitle data via a serial line,
Output to the RAM 430 via the CPU bus. The encoded subtitle data has a problem in that the data amount is extremely small compared to the encoded video data or the encoded audio data, and even if transmitted via the CPU bus, it does not affect the data traffic of the CPU bus. Does not occur.

In this case, the ENIF circuit 420 or the SIF 422 detects the header size written in the header attached to the encoded subtitle data, and outputs the detected header size to the CPU 424 via the CPU bus. In addition, the RAM 430 includes a CPU 424
According to the control of (1), the input encoded subcode data is output to the input terminal b of the switch circuit 34.

In the multiplexing system 12 (FIG. 1) of the FIFO memories 32a, 32b, 32c ,
a, 32b, and 32c are video encoders 2 respectively.
0, video streams input from the audio encoder 24 and the subtitle encoder 28, respectively.
The audio stream and the subtitle stream are buffered, and the input terminals a,
Output for b and c.

Switch circuit 34 The switch circuit 34 selects one of the input terminals a, b, c, and d in accordance with the control of the multiplexing system 12 via a control signal (control), and inputs the input terminal to each of these input terminals. The selected elementary stream is selected and multiplexed, and is output to the input terminal b of the switch circuit 36.
The switch circuit 34 is provided when there is no elementary stream input to any of the input terminals, or
When the stuffing process is performed, the input terminals a, b,
Predetermined blank data (continuous logical value 1 or 0) is output without selecting either c or d.

Switch circuit 36 The switch circuit 36 selects one of the input terminals a and b according to the control of the multiplexing system 12 through a control signal (control), and is input to the input terminal b from the switch circuit 34. Selecting and multiplexing one of the elementary streams or a private data stream (header data) input from the processing RAM 426 to the input terminal a,
Output to the FIFO memory 38 and the SCSIIF circuit 40.

The FIFO memory 38 buffers the data stream multiplexed by the switch circuit 36 and outputs it as a transport stream to an external device (not shown) such as a communication line.

SCSIIF circuit 40 The SCSIIF circuit 40 outputs the data stream multiplexed by the switch circuit 36 to a recording device (not shown) of a hard disk device (HDD) or a magneto-optical disk device (MOD), Record.

In the control system 42, the data size IFs 30a, 30b, 30c
b and 30c are a video encoder 20, an audio encoder 24 and a subtitle encoder 28, respectively.
The data size (size) of each frame of the video stream, the audio stream and the subtitle stream input from the
24.

The counting of the data size is performed by a counter built in the data size IF 30a, 30b, 30c. Further, the data size can be counted by detecting the data size of each frame of each elementary stream output by the video encoder 20, the audio encoder 24, and the subtitle encoder 28 by the data size IFs 30a, 30b, 30c. .

ENIF circuit 420 The ENIF circuit 420 has a private data (priority) input via a LAN (not shown) such as Ethernet.
vate data) and the CPU 42 via the CPU bus.
4 is output.

SIF circuit 422 The SIF circuit 422 receives serial private data input from, for example, a computer, and
Output to U424. Note that examples of the private data include the above-described subtitle data, closed caption data, and user data.

CPU 424 The CPU 424 includes, for example, a microprocessor, a ROM for storing programs, and peripheral circuits thereof, and has a data size IF 30 a, 30 b, 30 c, or E.
The data size input from the NIF circuit 420 and the SIF circuit 422 is stored in the processing RAM 426, and the
Based on the remaining storage capacity (remaining buffer capacity) of the O memories 32a, 32b, 32c and the like and the data size stored in the processing RAM 426, the multiplexing order of the elementary streams, multiplexing timing adjustment and scheduling, etc. The method is planned, and the multiplexing operation of the switch circuits 34 and 36 is controlled via the CPU bus according to the planned multiplexing method.

The control data RAM 428 is
The CPU 424 stores control data related to the processing of U424, and according to the scheduling using the control data stored in the control data RAM 428,
The header data is created on the processing RAM 426 and output to the input terminal a of the switch circuit 36.

A part of the header data is input via ENIF circuit 240 or SIF circuit 422,
It may be created based on private data (user data) stored in the AM 426.

The operation of the data multiplexing device 2 in the second embodiment will be described below. Video encoder 20, audio encoder 24 and subtitle encoder 28 each
The input video data, audio data, and subtitle data are encoded. Data size IF 30a, 3
0b and 30c respectively count the data sizes of the video stream, the audio stream and the subtitle stream, and output them to the CPU 424.

When the ENIF circuit 420 receives the encoded subtitle data via a LAN such as Ethernet or the like, or the SIF circuit 422 receives the encoded subtitle data via a serial line instead of the subtitle encoder 28, the ENIF circuit 420 or the SIF The circuit 422 detects the data size and sends the data to the CPU 42 via the CPU bus.
4 is output. Further, the RAM 430 stores the CPU 4
Under the control of 24, the input encoded subcode data is output to the input terminal d of the switch circuit 34.

FIG. 5 shows the data multiplexer 2 shown in FIG.
13 is a flowchart showing processing of the CPU 424 in the second embodiment. The processing shown in FIG. 5 is executed for each video frame.

As shown in FIG. 5, step 102 (S1
02), the CPU 424 determines that the data size IF3
When the data size of each frame of the video stream is received from 0a, the received data size is stored in the processing RAM 426. Further, the CPU 424 controls the data size IF 30b, 30c, the ENIF circuit 420 or the SI
Upon receiving the data size input from the F circuit 422, the data size is stored in the processing RAM 426.

In step 103 (S103), C
The PU 424 initializes a variable n.

In step 104 (S104), C
The PU 424 plans the multiplexing method described above.

Further, the CPU 424 determines the number N of loops in this processing. The number of loops N indicates the number of packets inserted in one video frame period, and the value of the number of loops N is, for example, a packet size of 188 bytes.
Since it is fixed, it depends on the data rate of the transport stream. That is, the number of loops N is calculated by the following formula [N = {data rate of transport stream (Byte)} 30｝ ÷ 188 (Byte); NTSC
In the case of the system, N = {Transport stream data rate (Byte)} 25} 188 (Byte);
In the case of the PAL system], the CPU 424 performs multiplexing scheduling based on the number of loops N and the received data size.

In step 106 (S106), C
PU 424 determines whether the planned multiplexing method indicates to output a video stream. If the multiplexing method indicates to output a video stream,
The CPU 424 proceeds to the process of S107, and if it does not indicate to output the video stream, proceeds to the process of S110.

In step 107 (S107), C
The PU 424 controls the switch circuit 36 so that the RAM 4
The video header data (a) input from the input terminal 26 is selected and output.

In step 108 (S108), C
The PU 424 controls the switch circuit 34 to select a video stream, controls the switch circuit 36 to select a data stream from the switch circuit 34, and multiplexes the video stream into a transport stream.

In step 110 (S110), C
PU 424 determines whether the planned multiplexing method indicates to output an audio stream. If the multiplexing method indicates to output an audio stream, C
The PU 424 proceeds to the process of S111. If it is not indicated to output the audio stream, the PU 424 proceeds to the process of S114.

In step 111 (S111), C
The PU 424 controls the switch circuit 36 so that the RAM 4
Header data (a) for audio input from
And output.

In step 112 (S112), C
The PU 424 controls the switch circuit 34 to select an audio stream, controls the switch circuit 36 to select a data stream from the switch circuit 34, and multiplexes the audio stream with the transport stream.

In step 114 (S114), C
PU 424 determines whether the planned multiplexing method indicates to output a subtitle stream from the subtitle encoder. If the multiplexing method indicates that the subtitle stream is to be output, the CPU 424 proceeds to the process of S115. If the multiplexing method does not indicate that the subtitle stream is to be output, the process proceeds to S118. In step 115 (S115), C
The PU 424 controls the switch circuit 36 so that the RAM 4
26 to select and output the subtitle header data (a) inputted from the subtitle 26.

In step 116 (S116), C
The PU 424 controls the switch circuit 34 to select a subtitle stream, controls the switch circuit 36 to select a data stream from the switch circuit 34, and multiplexes the subtitle stream into a transport stream. In step 118 (S118),
The CPU 424 determines whether or not the planned multiplexing method indicates that header data is added. CPU 424
If the multiplexing method indicates that the header data is to be output, the process proceeds to S119. If the multiplexing method does not indicate that the header data is to be output, the process proceeds to S122.

In step 119 (S119), C
The PU 424 controls the switch circuit 36 to select a private data stream and multiplex the private data stream (header data) with the transport stream. In step 120 (S120), the CPU 424 sets the switch circuits 34 and 36 and R
By controlling the AM 430, the private data from the RAM 430 is output and multiplexed to the transport stream.

In step 122 (S122), C
The PU 424 determines whether or not the variable n = N. If the variable n = N, the process is terminated; if n ≠ N, S
Proceed to 106. The FIFO memory 38 has a FIFO
The transport stream output from the memory 38 is buffered and transmitted to a communication line or the like. S
The CSIIF circuit 40 sends the transport stream output from the FIFO memory 38 to a hard disk device or the like and records it.

As described above, in the data multiplexing apparatus 2 according to the present invention, a multiplexing method is planned each time a transport stream is generated by software processing of the CPU 424. The feature is that the type and number of private data streams (header data) to be added can be arbitrarily changed, as compared with the case where multiplexing is performed on a transport stream by hardware. Therefore, it is easy to change the type and number of private data streams.

For the data multiplexing device 2, the number of elementary streams to be multiplexed into the transport stream is increased or decreased, or the ENIF circuit 420
It is possible to make a change to increase or decrease the number of SIF circuits 422. Further, in addition to the data shown in the second embodiment, the data multiplexing apparatus 2 may perform various processes such as multiplexing a data stream related to an editing process and a process at the time of broadcasting into a transport stream instead of a subtitle stream. Can be adopted.

Third Embodiment Hereinafter, a third embodiment of the present invention will be described.

In the data multiplexing device 2 (FIG. 2) shown as the second embodiment, since an elementary stream having a very large data amount is not transmitted on the CPU bus, a bus neck on the CPU bus is not transmitted. The data multiplexing apparatus 2 is suitable for generating a high-speed transport stream, since the multiplexing of the high-speed elementary stream to the transport stream is not hindered by the occurrence of.

However, the data multiplexing device 2 in the second embodiment has a data amount IF 30a, 30b, 30
Based on the data amount of each elementary stream counted by c and the like and the remaining storage capacity (remaining buffer capacity) of the FIFO memories 32a, 32b, and 32c, the CPU 424 switches the connection of the switch circuit 34 to perform multiplexing. Therefore, the video encoder 20, the audio encoder 24
And the remaining storage capacity (remaining buffer capacity) of the FIFO memories 32a, 32b, and 32c depending on whether the subtitle encoder 28 writes data intermittently or continuously into the FIFO memories 32a, 32b, and 32c, respectively. Fluctuates.

Therefore, the data amount of the audio stream in one part of the multiplexed transport stream is reduced, and the data amount of the audio stream in the other part is increased. The data amount of the mental stream is sparse and dense, and the interval between the elementary streams is biased. For example, when the interval between the audio streams is long, the expansion decoding device on the receiving side cannot normally reproduce the audio. Such a problem may occur.

For example, the CPU 424 determines that the remaining storage capacity (remaining buffer capacity) of the reception buffer of the decompression decoding device on the receiving side.
By performing a multiplexing plan in consideration of the above, it is also possible to avoid such a problem. However, C
The software algorithm for the PU 424 to perform the multiplexing plan asynchronously with the video frame signal in consideration of the reception buffer is complicated and requires a long processing time.
This method hinders speeding up of the multiplexing process.

The third embodiment of the present invention is intended to solve such a problem, and is simple in synchronization with a frame signal (video frame signal) of video data and without considering the remaining storage capacity (remaining buffer capacity). The data multiplexing device 2 is designed so that multiplexing of each elementary stream can be performed at a high speed without causing a failure in the reception buffer of the decompression decoding device on the receiving side.
This is an improvement of the operation shown in FIG.

Multiplexing Plan of CPU 424 In the third embodiment, the CPU 424 (FIG. 2) synchronizes the video frame signal input from the video encoder 20 with the video frame signal as shown by the dotted line in FIG. Cycle [Video frame cycle; 1/30 second (in case of NTSC system), 1/2]
5 seconds (in the case of the PAL system)], and controls each component of the data multiplexing device 2 based on the planned multiplexing method.

Each component of the data multiplexing device 2 has a CP
Under the control of U424, in each video frame period, a video stream of approximately equal data amount a shown in the following equation is multiplexed into, for example, a transmission packet of the ATM system, and the audio stream generated by the audio encoder 24 and the subtitle encoder 28 The private data stream is sequentially multiplexed into transmission packets to generate a transport data stream. Since the data rate of the video stream becomes substantially constant by the fixed rate coding, the data amount a is
As a result, the average data amount per video frame period of the video stream generated by the video encoder 20 is almost the same.

[0078]

A ≒ V / k (1) where a is the data amount of the video stream multiplexed in one video frame period, and V is the coding rate of the video stream.

Further, the reason why the failure of the receiving buffer can be prevented by the above-described multiplexing plan for each video frame period will be described. In the decompression decoding device (decoder) on the receiving side, the video data is read from the reception buffer every video frame period and removed. Therefore, if almost the same amount of video stream in an appropriate range is written from the transmitting side to the receiving buffer in each video frame period in synchronization with the video frame signal, no failure occurs in the receiving buffer. Conversely, for example, the video stream for the (m + 1) th video frame period is not multiplexed in the mth video frame period, but the video stream for two frames is multiplexed into the transmission packet to generate the transport stream. If the transmission is performed in the receiving buffer, there is a possibility that a failure occurs in the receiving buffer.

Similarly, the audio stream also has a sampling period of audio data [audio frame period; as described above, 24 ms (1152/48 kHz), 26.1 m.
s (1152 / 44.1 kHz), 36 ms (1152
/ 32 kHz) {1/30 seconds, 1/25 seconds (video frame period)], and is generated for each audio frame period,
Also, since the data is read from the reception buffer of the decoder and removed, the transmission packet is multiplexed in equal amounts in each audio frame period, and a transport stream is generated and transmitted. it can.

That is, the audio stream has a sampling frequency of audio data of 48 kHz or 44.1.
If the frequency is 32 kHz, it is generated once or twice every video frame period, and is removed from the receiving buffer of the decoder on the receiving side. If the sampling frequency of the audio data is 32 kHz, the video frame is generated. It is generated 0 or 1 time per cycle and is removed from the receiving buffer of the receiving decoder. Therefore, the CPU 424 multiplexes the audio stream into transmission packets at most twice in each video frame period, and performs a multiplexing plan so as to generate a transport stream. Can be prevented from being unable to be reproduced normally.

Operation of Data Multiplexing Apparatus 2 The operation of the data multiplexing apparatus 2 according to the third embodiment will be described below with reference to specific examples of data rates of a video stream and an audio stream.

The video encoder 20 encodes the input video data under the control of the CPU 424,
For example, a video stream having a substantially constant data rate (4 Mbps) is generated. Audio encoder 24
Encodes the input video data according to the control of the CPU 424 and outputs a fixed data rate (384 kbps).
Generate the audio stream of s). The subtitle encoder 28 appropriately encodes the input subtitle data and generates a private data stream.

The data amounts IF 30a, 30b, and 30c respectively count the data amounts of the video stream, the audio stream, and the private data stream generated by the video encoder 20, the audio encoder 24, and the subtitle encoder 28 per fixed time, respectively. The data is output to the CPU 424 as the data rate of the stream.

Here, when the data rate of the video stream of the NTSC system is, for example, 4 Mbps,
As shown in the following equation, the data amount of the video stream per one video frame period is 16666.7 bytes.
Similarly, if the data rate of the audio stream is 384
In the case of kbps, the data amount of the audio stream per one video frame period is 1600 bytes as shown in the following equation.

[0086]

## EQU00002 ## Data amount of video stream per one video frame period; 4,000,000 bps / (8 bits.times.30 frames) = 1666.67 bytes (2-1) Data amount of audio stream per frame; 38400000 bps / (8 bits × 30 frames) = 1600 bytes (2-2)

Therefore, the CPU 424 multiplexes the 16666-byte or 166667-byte video stream into the transmission packet every video frame period, and multiplexes the 1600-byte audio stream in the order shown in FIG. A multiplexing plan is performed, and each component of the data multiplexing device 2 is controlled so as to generate a transport stream according to the multiplexing plan.

FIG. 6 shows the data multiplexing apparatus 2 shown in FIG.
FIG. 14 is a diagram illustrating the order of each elementary stream included in the transport stream generated in the third embodiment. First, the CPU 424 controls the switch circuit 3
6 to select the input terminal a side, and before the video stream and the audio stream, the CPU 42
4 generated PCR (program clock reference) data, PAT (program association table) and PMT
(program map table) The data is stored in the RAM 426 for processing, for example, for each one video frame period (less than the longest retransmission period specified in MPEG).
It multiplexes it into one transmission packet, generates a transport stream, and outputs it outside.

Next, the CPU 424 operates the switch circuit 36.
To select the input terminal b side, and the switch circuit 34
To sequentially change the selection of the input terminal so that the first video stream, the first audio stream, the first subtitle (private data stream), the first
Null data, a second video stream, a second audio stream, a second subtitle (private data stream) and a second Null data are multiplexed into transmission packets in this order to generate a transport stream. Output to the outside.

The Null data shown in FIG. 6 is composed of a video stream, an audio stream, a private data stream and other data in order to keep the data rate of the transport stream output from the data multiplexing device 2 constant. Is invalid data inserted when the data rate is lower than the data rate of the transmission path.

Hereinafter, the processing of the CPU 424 will be further described with reference to FIG. FIG. 7 is a flowchart showing a multiplexing plan process in the third embodiment of the CPU 424 of the data multiplexing device 2 shown in FIG.

As shown in FIG. 7, step 200 (S2
00), the CPU 424 determines that the data amount IF30
a, 30b, and 30c, based on the data amount of each elementary stream per fixed time, how much data amount of each of these elementary streams at each position of the transport stream shown in FIG. Parameter (SIZE) indicating whether only
_FRAMEV, SIZE_FRAME_V1, SIZ
E_FRAME_V2).

In step 202 (S202), C
The PU 424 receives the video frame signal input from the video encoder 20, and proceeds to the process of S204 if a new video frame period has started, and stays at the process of S202 if it does not start. Step 204 (S20
In 4), the CPU 424 checks the PCR data, PAT
Data and PMT data are multiplexed and output to the outside.

In step 206 (S206), C
The PU 424 converts the video stream into the parameter SIZE_FRAME_V1 set in the processing of S200.
Are multiplexed by the amount of data indicated by, and output to the outside as a first video stream. In step 208 (S208), the CPU 424 converts the audio stream into S
Parameter SIZE_F set in process 200
The data is multiplexed by the amount of data indicated by RAME_A1 and output to the outside as a first audio stream.

In step 210 (S210), C
The PU 424 multiplexes the subtitle data (private data stream) by the data amount indicated by the parameter SIZE_FRAME_S1 set in the processing of S200, and outputs the multiplexed data to the outside as a first private data stream. In step 212 (S212), the CPU 424 stores the Null data in the parameter SIZE_FRAME_
The data is multiplexed by the data amount indicated by N1 and output to the outside as first Null data.

In step 214 (S214), C
The PU 424 converts the video stream into the parameter SIZE_FRAME_V2 set in the processing of S200.
Are multiplexed by the data amount indicated by, and output to the outside as a second video stream. In step 216 (S216), the CPU 424 converts the audio stream into S
Parameter SIZE_F set in process 200
The data is multiplexed by the amount of data indicated by RAME_A2 and output to the outside as a second audio stream.

In step 218 (S218), C
The PU 424 multiplexes the subtitle data (private data stream) by the data amount indicated by the parameter SIZE_FRAME_S2 set in the processing of S200, and outputs the same as a second private data stream to the outside. In step 220 (S220), the CPU 424 stores the Null data in the parameter SIZE_FRAME_
The data is multiplexed by the data amount indicated by N2 and output to the outside as second Null data.

FIGS. 6 and 7 show a case where the number of repetitions of multiplexing of each elementary stream is two. Therefore, the data of each elementary stream that is multiplexed in two parts is multiplexed. Make the volumes approximately equal. That is, the relationship between the parameters is SIZE_FRAME_
V1 ≒ SIZE_FRAME_V2, SIZE_FRA
ME_A1 @ SIZE_FRAME_A2, SIZE_
FRAME_S1 ≒ SIZE_FRAME_S2.

Since the number of repetitions of multiplexing of each elementary stream shown in FIGS. 6 and 7 is not limited to two, when the number of repetitions is an odd number, for example, one of these parameters is set to a large value. It is also possible. For example, when the number of repetitions is three, the relationship between the parameters is expressed as SIZE_FRAME_V1 <SIZE
_FRAME_V2 <SIZE_FRAME_V3, S
SIZE_FRAME_A1 <SIZE_FRAME_A
2 <SIZE_FRAME_A3, SIZE_FRAM
E_S1 <SIZE_FRAME_S2 <SIZE_F
RAME_S3 is also possible.

The order of multiplexing of each elementary stream shown in FIG. 6 as the third embodiment is an exemplification, and the operation of the CPU 424 is performed so that the multiplexing of each elementary stream is planned in another order. May be changed. The multiplexing plan of the data multiplexing apparatus 2 shown as the third embodiment is based on the NTSC 525/60 format audio / video.
The present invention can be applied to both video data and audio / video data in the 625/50 format of the PAL system. Also, PES
The unit for multiplexing into packets may be video data for one picture or video data output during one video frame period. Further, the same modification as that of the second embodiment can be applied to the third embodiment.

Fourth Embodiment According to the operation of the data multiplexing apparatus 2 (FIG. 2) shown in the third embodiment, the decompression decoding apparatus on the receiving side can use a simple algorithm that does not consider the remaining storage capacity of the receiving buffer. It is possible to always multiplex audio / video data into transmission packets and generate a transport stream so that normal reproduction of audio / video can be performed without causing a failure in the receiving buffer.

However, the NTSC system 525/6
The video frame period of video data in the 0 format is exactly 1 /
29.97 Hz, for example, the video encoder 20
The data rate of the video stream generated by is 4Mbp
When s is fixed, as shown in the following formula, a fraction of 0.35 bytes of the video stream is generated for each video frame period, and the CPU 424 needs to perform a multiplexing plan in consideration of the fraction. .

[0103]

## EQU3 ## 4 × 10 ⁶ /29.97=1333466.8 bits = 16683.35 bytes (3)

The same applies to the audio stream generated by the audio encoder 24. For example, the data rate of the audio stream is 384 kb.
When ps is fixed, as shown in the following formula, a fraction of 0.6016 bytes of the audio stream is generated for each video frame period, and the CPU 424 needs to perform a multiplexing plan in consideration of the fraction. .

[0105]

384 × 10 ³ /29.97=12812.813 bits = 1601.6016 bytes (4)

On the other hand, the video frame period of the 625/50 format of the PAL system is 1/25 Hz. For example, as described above, the data rate of the video stream is fixed at 4 Mbps, and the data rate of the audio stream is 38
In the case of fixed 4 kbps, no fraction occurs in the video stream and the audio stream as shown in the following equation. However, if the data rate of the video stream and the audio stream is 200 bps (= 2
If it is not a multiple of 5 × 8), a fraction is generated.
The PU 424 needs to perform a multiplexing plan in consideration of the fraction.

[0107]

Equation 5] video ^{stream; 4 × 10 6/25 =} 160000 bits = 20000 bytes ... (5-1) audio ^{stream; 384 × 10 3/25 =} 15360 bits = 1920 bytes ... (5-2)

The fourth embodiment of the present invention has been made in view of the above-described problem, and is an improvement of the third embodiment of the present invention, which further simplifies the multiplexing planning algorithm. .

A multiplexing plan of the CPU 424 Hereinafter, a multiplexing plan of the CPU 424 in the fourth embodiment will be described by taking an example of multiplexing an NTSC video stream.

Data Rate of Video Stream In the fourth embodiment, the CPU 424
Data rate of video stream of video encoder 20 input from F30a video_rate_or
g is a fixed value (frame_video_r) shown in the following equation.
ate) and perform a multiplexing plan.

[0111]

[Equation 6] frame_video_rate (byte) = video_rate_org (bps) / {(8 × 1001) / (1000 × 30)} (6) where video_rate_org is data of a video stream generated by the video encoder 20 Rate the fram
e_video_rate rounds down decimal places.

Here, the data rate f per video frame period of the video stream calculated by the above equation (6)
Although the value of the frame_video_rate is discrete, there is no particular problem because the data rate of the video stream can be any value unless there is a restriction.

In addition, the above equation (6) is equivalent to the operation [1001 / (10
00 × 30)] so that the number of frames per second in the NTSC system, 29.97, can be handled by integer arithmetic.

Note that the value of the target data amount video_rate_target set by the CPU 424 in the bit rate control circuit 220 (FIG. 3) of the video encoder 20 is calculated by the following equation.

[0115]

[Expression 7] video_rate_target (bps) = frame_video_rate (byte) / {(8 × 30 × 1000) / 1001} (7)

For example, the data rate of a video stream generated by the video encoder 20 is video_rate.
When _org is 4 Mbps, the target data amount vid
The value of eo_rate_target is 3.999920
1 Mbps. Note that in this case, the difference between the two values is very small, and even if the CPU 424 sets the data rate video_rate_org as the target data amount video_rate_target in the bit rate control circuit 220, there is no practical problem. Does not occur.

Hereinafter, with further reference to FIG.
4 is a bit rate control circuit 22 of the video encoder 20
Target data amount video_rat to be set to 0 (FIG. 3)
A method for calculating e_target will be described. FIG.
FIG. 9 is a flowchart illustrating a method of calculating a target data amount video_rate_target set in the bit rate control circuit 220 of the video encoder 20 of the data multiplexing device 2 in the example of FIG.

As shown in FIG. 8, step 240 (S2
40), the CPU 424 sets the data amount IF 30a
, The data rate video_rate_org of the video stream generated by the video encoder 20 is received and set in the RAM 428.

In step 242 (S242), C
The PU 424 uses Equation 6 to calculate the data rate of the video stream per video frame period frame_video.
o_rate is calculated.

In step 244 (S244), C
The PU 424 calculates the target data amount vi to be set in the bit rate control circuit 220 of the video encoder 20 according to Expression 7.
Calculate deo_rate_target.

For example, when the data rate video_rate_org of the video stream is 4 Mbps and the payload portion of the transmission packet is 184 bytes, a fraction of 123 bytes is generated as shown in the following equation. In general, the remainder of the payload portion of the transmission packet containing this fraction is 61 bytes (= 184-12).
3) The process of storing the stuffing data is performed.
Performing this process unnecessarily increases the data rate of the transport stream. Therefore, it is preferable to adopt a method of multiplexing this fractional part in the next video frame period.

[0122]

## EQU8 ## = 16683 (Equation 3) / 184 = 90 packets + 123 remaining bytes (8)

Data Rate of Audio Stream FIG. 9 is a chart showing the data rate of an audio stream standardized in the MPEG system. Unlike the data rate of the video stream, as shown in FIG. 9, the data rate of the audio stream is audio.
_Rate_target is a fixed value, and each value is a multiple of 8 × 1000 bps.

For example, the data rate of the audio stream generated by the audio encoder 24 is audio_
When rate_target is 384 kbps, the data amount au of the audio stream output by the audio encoder 24 during 30 video frame periods.
“dio_frame_30frame” indicates that the control cycle is 3
Assuming that the video frame period is 0 (about 1 second), it is calculated as shown in the following equation.

[0125]

[Equation 9] audio_rate_30frame = 384 × 1000 × 1001/1000 = 384384 bits = 48048 bytes (9)

Accordingly, the data amount of the audio stream to be multiplexed into the transport stream for each video frame period is 1601 bytes or 1602 bytes. Therefore, the CPU 424 determines that the audio frames multiplexed during the past 30 video frame periods have the data amount audio_frame_30f calculated by Expression 9.
160 if it is larger than frame (48048 bytes)
The switch circuit 34 multiplexes the audio stream of 2 bytes into the transport stream of the video frame cycle, and if not, multiplexes the audio stream of 1601 bytes into the transport stream of the video frame cycle. Control connections.

Operation of Data Multiplexing Apparatus 2 The operation of the data multiplexing apparatus 2 according to the fourth embodiment will be described below with further reference to FIG. FIG.
FIG. 13 is a flowchart showing a process in which the data multiplexing apparatus 2 calculates the data amount of an audio stream to be multiplexed to a transport stream for each video frame period and multiplexes the data stream with a video stream or the like in the embodiment of FIG.

As shown in FIG. 10, step 300 (S
In 300), the CPU 424 receives the data rate of the audio stream generated by the audio encoder 24 from the data amount IF 30b and sets the data rate in the RAM 428.

In step 302 (S302), C
The PU 424 calculates the data amount frame_audio_rate_pre of the audio stream to be multiplexed into the transport stream for each video frame period,
It is calculated by the following equation.

[0130]

[Equation 10] frame_audio_rate_pre = audio_rate_target / {(8 × 1001) / (1000 × 30)} (10) However, the fractional part of frame_audio_rate_pre is rounded down.

In step 304 (S304), C
According to Equation 9, the PU 424 calculates the data amount audio_fr of the audio stream per 30 video frame periods.
ame_30frame is calculated.

In step 306 (S306), C
The PU 424 receives the video frame signal input from the video encoder 20, and determines whether the video frame period has started. If the video frame period has started, the CPU 424 proceeds to the process of s308. If not started, the CPU 424 remains at the process of S306.

In step 308 (S308), C
The PU 424 determines that the data amount of the audio stream multiplexed during the past 30 video frame periods is equal to the data amount audio_frame_ calculated in the process of S304.
It is determined whether or not the number is larger than 30 frames. If the number is larger, the process proceeds to S310. If not, the process proceeds to S312.

In step 310 (S310), C
The PU 424 outputs the data amount frame_audio of the audio frame to be multiplexed in the video frame period.
o_rate is obtained, and is set as the data amount frame_audio_rate_pre calculated in the process of S302.

In step 312 (S312), C
The PU 424 outputs the data amount frame_audio of the audio frame to be multiplexed in the video frame period.
o_rate is obtained, and the data amount frame_audio_rate_pre ++ calculated in the process of S302.
Let it be 1.

The CPU 424 controls each component of the data multiplexing device 2 such as the switch circuit 34, and the data amount frame_video calculated by the processing shown in FIG.
o_rate video stream, S300 to S300 in FIG.
Data amount frame_ calculated by the process of S312
The audio stream of audio_rate and the private data stream are multiplexed.

As described above, according to the operation of the data multiplexing apparatus 2 shown as the fourth embodiment, the video stream and the audio stream synchronized with the video frame period are obtained by a simple algorithm without using a floating-point operation. Multiplexing can be performed.

At the start of the generation of the transport stream, if the generated video stream does not reach 30 video frame periods, for example, the data amount frame
_Audio_rate to the data amount frame_au
dio_rate_pre and data amount frame_
audio_rate_pre + 1 may be set.

The data amount video_frame of a video stream to be multiplexed per video frame period
In addition to the method of changing _rate for each video frame period as described above, for example, the data amount frame_audi is obtained every 30 video frame periods (predetermined control periods).
o_rate is replaced by the data amount frame_audio_r
ate_pre and data amount frame_audio
_Rate_pre + 1 may be changed.

The data rates of the video stream and the audio stream (4 Mbps, 384 kbp)
s, etc.) and the unit of data amount (byte) are examples, and by making appropriate changes, the method of multiplexing planning shown as the fourth embodiment can be applied to video streams of other data rates and The present invention can be applied to a multiplexing plan using an audio stream and other data amount units.

[0141]

As described above, the present invention is suitable for multiplexing audio data, video data and private data in digital television broadcasting and the like. Further, according to the present invention, it is suitable for high-speed multiplexing of audio data, video data, and additional data for digital television broadcasting and the like, and can flexibly cope with changes in the type and processing of the additional data.

Further, according to the present invention, in a digital television broadcasting facility or the like, it is not necessary to consider the remaining storage capacity of the reception buffer of the decompression decoding device on the receiving side, and it is also possible to simply use only integer arithmetic without using floating point arithmetic. Voice and audio to prevent the reception buffer from breaking down
Video data can be multiplexed into transmission packets to generate a transport stream. Further, according to the present invention, a transport stream can be generated by multiplexing audio / video data into transmission packets so that audio / video data can always be normally reproduced on the receiving side.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a data multiplexing apparatus according to the present invention shown as a first embodiment.

FIG. 2 is a diagram showing a configuration of a data multiplexing device according to the present invention.

FIG. 3 is a diagram illustrating a configuration of a video encoder illustrated in FIG. 2;

FIG. 4 is a diagram illustrating a configuration of an audio encoder illustrated in FIG. 2;

FIG. 5 is a flowchart showing processing of a CPU in a second embodiment of the data multiplexing device shown in FIG. 2;

FIG. 6 is a diagram exemplifying an order of each elementary stream included in a transport stream generated by the data multiplexing apparatus shown in FIG. 2 in a third embodiment.

FIG. 7 shows a third example of the CPU of the data multiplexer shown in FIG. 2;
It is a flowchart which shows the process of the multiplexing plan in Example of FIG.

FIG. 8 is a diagram showing a target data amount video_rate_target set in a bit rate control circuit of a video encoder of a data multiplexer (FIG. 2) in a fourth embodiment;
It is a flowchart figure which shows the calculation method of t.

FIG. 9 is a chart showing a data rate of an audio stream standardized in the MPEG system.

FIG. 10 is a diagram illustrating a process in which a data multiplexer (FIG. 2) calculates a data amount of an audio stream to be multiplexed into a transport stream for each video frame period and multiplexes the data with a video stream or the like in a fourth embodiment. It is a flowchart figure which shows.

[Explanation of symbols]

1, 2, ... data multiplexing device, 10 ... multiplexing device, 100
... Buffer memory, 102 ... Transport stream generator, 12 ... Multiplexing system, 20 ... Video encoder,
24: audio encoder, 28: subtitle encoder, 32a, 32b, 32c: FIFO memory, 3
4, 36: switch circuit, 38: FIFO memory, 40
... SCSIIF circuit, 42 ... control system, 30a, 30b,
30c: data amount IF, 420: ENIF circuit, 422
... SIF circuit, 424 ... CPU, 426 ... RA for processing
M, 428: RAM for control data, 430: RAM.

Claims (8)

[Claims] 1. A video target data amount calculating means for calculating a target data amount satisfying a condition that video data generated in each video frame period of video data can be distributed by an equal data amount so that a fraction does not occur. A video data generating unit that generates video data having a data amount equal to the calculated target data amount; and an audio data generating unit that generates audio data having a predetermined data amount for each audio frame period different from the video frame period. An audio data distribution unit that distributes the generated audio data by a data amount substantially equal to each of the video frame periods; and for each of the video frame periods, the video data having the same data amount and the allocated audio data. Multiplexing means for multiplexing data into a predetermined transport stream. Place. 2. The data multiplexing apparatus according to claim 1, wherein said video data generating means compresses the uncompressed video data and generates the video data having a data rate suitable for the target data amount. 3. The audio data distribution unit includes: an audio target data amount calculation unit that calculates a target data amount of the audio data to be multiplexed during a predetermined number of the video frame periods; Audio data counting means for counting a data amount of the audio data multiplexed during the predetermined number of video frame periods; a calculated target data amount of the audio data; and a counted data amount of the audio data. 2. The data multiplexing apparatus according to claim 1, further comprising: audio data distribution amount calculation means for calculating a data amount of the audio data to be allocated to the video frame cycle based on the audio data. 4. An apparatus according to claim 1, further comprising additional data generating means for generating additional data to be added to said audio data and said video data, wherein said multiplexing means further multiplexes said generated additional data into said predetermined transport stream. Claim 1
3. The data multiplexing device according to 1. 5. A target data amount that satisfies a condition that video data generated in each video frame period of the video data can be distributed by an equal data amount so that a fraction does not occur, and the calculated target data amount is calculated. And generating audio data of a predetermined data amount for each audio frame period different from the video frame period, and generating the generated audio data with data substantially equal to each of the video frame periods. A data multiplexing method of allocating video data of the same data amount and the allocated audio data to a predetermined transport stream for each video frame period. 6. The data multiplexing method according to claim 5, wherein uncompressed video data is compressed to generate said video data having a data rate adapted to said target data amount. 7. A target data amount of the audio data to be multiplexed during a predetermined number of the video frame periods, and multiplexed during the predetermined number of the video frame periods in the past of the video frame period. The data amount of the audio data is counted, and the data amount of the audio data to be allocated to the video frame cycle is calculated based on the calculated target data amount of the audio data and the counted data amount of the audio data. The data multiplexing method according to claim 5, wherein 8. The data multiplexing method according to claim 5, wherein additional data to be added to said audio data and said video data is generated, and said generated additional data is further multiplexed into said predetermined transport stream.