JP2005101860A - Media information processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a media information processing method by which the transmission delay of a transport stream packet can be shortened. <P>SOLUTION: In an encoder 10, media information is taken in and coded. Media information coded in an ES accumulator 11 is temporarily held. A TS packet size monitor 12 monitors whether coded media information held in the ES accumulator 11 are assembled by a prescribed data size or not within a prescribed time. When it detects that coded media information is assembled by the prescribed data size, a packet division part 13 makes coded media information into TS packets. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a media information processing method and apparatus for efficiently converting media information into fixed-length packets.

  In recent years, the multimedia era has come to handle voice, image, and other pixel values in an integrated manner, and conventional information media, that is, means for transmitting information such as newspapers, magazines, televisions, radios, telephones, etc., to people have become multimedia. It has come to be taken up as a target. In general, multimedia refers to not only characters but also figures, sounds, especially images, etc. that are associated with each other at the same time. It is an essential condition.

  However, when the information amount of each information medium is estimated as a digital information amount, the amount of information per character is 1 to 2 bytes in the case of characters, whereas 64 Kbits (phone quality) per second in the case of speech. In addition, for a moving image, an information amount of 100 Mbits (current television reception quality) or more per second is required, and it is not realistic to handle the enormous amount of information in the digital format as it is with the information medium. For example, a video phone has already been put into practical use by an Integrated Services Digital Network (ISDN) having a transmission rate of 64 Kbit / s to 1.5 Mbits / s. It is impossible to send.

  Therefore, what is required is information compression technology. For example, in the case of a videophone, H.264 has been internationally standardized by ITU-T (International Telecommunication Union Telecommunication Standardization Sector). 261 and H.264. H.263 standard video compression technology is used. In addition, according to the information compression technology of the MPEG-1 standard, it is possible to put image information together with audio information on a normal music CD (compact disc).

  Here, MPEG (Moving Picture Experts Group) is an international standard for moving picture signal compression, and MPEG-1 is up to 1.5 Mbps for moving picture signals, that is, up to about 1 / 100th of information of television signals. It is a standard to compress. In addition, since the transmission speed for the MPEG-1 standard is mainly limited to about 1.5 Mbps, MPEG-2 standardized to meet the demand for higher image quality has 2 to 2 video signals. Compressed to 15 Mbps. Furthermore, at present, the working group (ISO / IEC JTC1 / SC29 / WG11), which has been standardizing with MPEG-1 and MPEG-2, has achieved a compression ratio that exceeds MPEG-1 and MPEG-2, and further, on a per-object basis. MPEG-4 has been standardized that enables encoding, decoding, and operation and realizing new functions necessary in the multimedia era. In MPEG-4, it was originally aimed at standardizing a low bit rate encoding method, but now it has been extended to a higher bit rate and more general purpose encoding including interlaced images.

  By the way, media information such as video and audio (hereinafter, a composite form of these media information is referred to as “multimedia information”) is generated in parallel, and the multimedia information is transmitted and received via a communication channel. In order to reproduce on the side, it is necessary to multiplex it into one stream together with the synchronization information. One of international standards for realizing such multiplexing is Transport Stream (hereinafter abbreviated as MPEG-2 TS) of the MPEG-2 system.

FIG. 14 is a diagram for explaining the outline of MPEG-2 TS.
As shown in FIG. 14, in MPEG-2 TS, encoded media information called ES (Elementary Stream) (reference numerals 1000, 1002, 1004, and 1006) is packetized in an appropriate unit for each medium, By attaching headers (reference numerals 1008 and 1012), variable-length PES (Packetized Elementary Stream) packets (reference numerals 1010 and 1014) are formed. The PES packet is further divided into fixed length (188 bytes) TS (Transport Stream) packets (reference numerals 1020, 1026, 1034 and 1040, hereinafter abbreviated as TSP as appropriate), and the TSP is multiplexed. Make a single stream.

  TSP is a program time for reproducing video and audio synchronously when decoding and reproducing encoded media information obtained by encoding media information such as video and audio in addition to sending the media information. In some cases, PCR (Program Clock Reference), which is reference value information, is transmitted. In order to synchronize the time of the media information, the TSP including the PCR must be transmitted at a constant clock cycle, and the MPEG-2 system standard and operation regulations such as digital television broadcasting (for example, DVB: DVB DOCUMENT A001 rev5 etc. in Digital Video Broadcasting).

  Further, it is stipulated that the media information (frames referred to as video data and audio data) constituting the TSP must be converted to TSP within a predetermined time so as not to cause a delay in decoding on the playback side. It is defined in the system standard (ISO / IEC 13818-1: 2000 (E)). According to section 2.7.4 of the standard, it is specified that all TSPs constituting one frame of media information must be included in an interval within 0.7 seconds.

On the other hand, in order to configure and transmit one TSP, it is necessary to have all the encoded data for configuring one TSP. Therefore, especially in low bit rate encoding, packetization processing delay Is a problem. In particular, when trying to realize TSP with software, a processing delay may occur if the processing time of each time is not constant.
In order to reduce this processing delay, as a conventional multiplexing method, as in TSP1034, redundant data 1032 (stuffing data) of the same amount as the amount of data that is insufficient to constitute one TSP is used in the multiplexing layer. Some of them are inserted and constitute one TSP together with encoded data 1030 (see, for example, Patent Document 1). In this case, since the PES packets 1010 and 1014 have a variable length, the amount of stuffing data is not constant.

However, since the stuffing data 1032 is data irrelevant to the original media information, it is preferable that the stuffing data 1032 is not included in the transmitted TSP as much as possible.
In view of this, a method has been devised in which the PES packet length is set to a fixed length and the fixed length is set to an integral multiple of the size of the payload 1018 in the TSP (the area portion other than the TSP header 1016 in the TSP 1020) (for example, Patent Documents). 2).
JP 9-74546 A (page 4-7, FIG. 1) JP 2003-108194 A

  However, the method disclosed in Patent Document 2 has a problem in that the ES data necessary for configuring the TSP cannot be prepared and the TSP cannot be transmitted by setting the PES packet to a fixed length.

This problem will be described with reference to FIGS.
FIG. 15 shows a conceptual diagram from ES into which media information is encoded into a PES packet to TSP.
In the figure, time axes 2000 and 2002 drawn horizontally indicate the same time. In addition, in actual encoding and packetization, the processing time required for each process is strictly required, but in this specification, it is assumed that the processing time is instantaneously performed.
Assuming applications such as live broadcasting, it is desirable that the TSPs 2016, 2018, 2020 and 2022 be packetized at a constant time interval T as much as possible. However, when packetization is performed at the time interval T, payload data necessary for packetization may not be available.

  For example, even if an attempt is made to packetize the next packet 2020 at the time interval T following the TSP 2018, at the timing when the TSP 2020 is generated, data at the beginning of the PES packet 2014 that is data to be stored in the payload of the TSP 2020, that is, Since the media information 2008 is not encoded, the TSP 2020 cannot be configured.

The same applies to the case of TSP2022. Even if the TSP 2022 is output at a time interval T away from the timing at which the TSP 2020 is to be output, at the timing when the TSP 2022 is generated, the data of the PES packet necessary for configuring the payload of the TSP 2022, that is, the media information 2008 and 2010 Is not encoded, TSP 2022 cannot be configured.
The problem described above can occur not only when the TSP is transmitted at such a predetermined time interval T but also when the TSP is transmitted at the PCR insertion interval.

FIG. 16 is a conceptual diagram when a TSP is generated by configuring a PES packet at the PCR insertion interval.
In the method of Patent Document 2, a PCR is inserted into a TSP 3016 that is transmitted at the timing of time Tn, a PCR is inserted into the TSP 3020 that is transmitted at the timing of time T (n + 1), and the size of the PES packet is changed to each PCR. It is characterized by being an integral multiple of the size of the payload in the TSP included in the insertion interval.

  Here, since PCR is necessary for synchronous playback of video and audio, it must be ensured that it is inserted into the TSP within a predetermined period. However, as shown in FIG. 16, time T (n + 1) Even if the TSP 3020 is output at the timing of), the encoded media information 3008 constituting the PES packet 3014 necessary for generating the TSP 3020 is not encoded at the timing of generating the TSP 3020, so the TSP 3020 is configured. I can't.

  As described above, in order not to perform stuffing, encoded media information for filling the packet to be transmitted at the time of packetization must be prepared, but an attempt is made to comply with a predetermined transmission timing. If this is the case, the TSP cannot be sent out because the encoded media information is not aligned, which increases the transmission delay of the TSP, and conversely degrades the transmission efficiency.

  Therefore, in constructing the TSP, it is desired to establish a technique that minimizes stuffing and further reduces transmission delay when PCR is also transmitted.

Therefore, the present invention has been made in view of such problems and circumstances, and an object thereof is to provide a media information processing method capable of reducing the transmission delay of TSP.
It is another object of the present invention to provide a media information processing method that minimizes the occurrence of stuffing in TSP and is excellent in transmission efficiency.

  In order to achieve the above object, a media information processing method according to the present invention is a media information processing method that encodes media information and packetizes the encoded media information into a predetermined data size. An encoding step for encoding the encoded media information, an encoded media information holding step for temporarily holding the encoded media information, and the encoded media information held in the encoded media information holding step within a predetermined time. In the packet size monitoring step for monitoring whether or not media information is aligned for the predetermined data size, and after detecting that the encoded media information is aligned for the predetermined data size in the packet size monitoring step, A packetizing step for packetizing the encoded media information. And butterflies. In this way, the encoded media information is held until the encoded media information of the size necessary for packetization is obtained, and packetization is performed after the encoded media information is obtained. Since it is no longer possible to send a packet without information being aligned, it is possible to prevent an increase in packet transmission delay.

  In the media information processing method, when the packet size monitoring step detects that the encoded media information has not been prepared for the predetermined data size, the packet has the predetermined data size. It includes a stuffing step for adjusting as follows. As a result, stuffing is performed only when necessary to comply with the packet transmission interval, so that the generation of stuffing can be minimized and the packet transmission interval can be kept constant. A media information processing method is realized.

  Note that the present invention can be realized not only as such a media information processing method but also as a media information processing apparatus including the characteristic steps included in such a media information processing method as a means. It can also be realized as a program for causing a computer to execute steps. Needless to say, such a program can be distributed via a recording medium such as a CD-ROM or a transmission medium such as the Internet.

As described above, according to the media information processing method according to the present invention, it is not possible to send out TS packets because the encoded media information necessary for constructing the TS packets is not aligned. Can be shortened.
Further, since the occurrence of stuffing in the TS packet can be minimized while observing the transmission timing of the TS packet, packetization with excellent transmission efficiency can be realized.

Hereinafter, the best mode for carrying out the present invention will be described in detail.
(Embodiment 1)
First, the media information processing method according to Embodiment 1 of the present invention will be described.
FIG. 1 is a block diagram showing the configuration of a media information processing apparatus that implements the media information processing method according to the first embodiment.

  This media information processing device 1 is a device that takes media information such as video and audio, encodes them, divides them into TSPs, and multiplexes the TSPs of each media information into one stream. , An ES storage unit 11, a TS packet size monitoring unit 12, a packet dividing unit 13, and a multiplexing unit 16. Here, the media information processing apparatus 1 includes the number of types of media information for the encoding unit 10, ES storage unit 11, TS packet size monitoring unit 12, and packet dividing unit 13.

The encoding unit 10 is a processing unit that takes in media information and encodes it using a predetermined encoding method, and stores the encoded media information (ES) in the ES storage unit 11.
The ES storage unit 11 is a storage device such as a memory or a hard disk for temporarily storing an ES.
The TS packet size monitoring unit 12 checks the size of the ES stored in the ES storage unit 11 as needed, and outputs a TS packet generation instruction to the TS packet dividing unit 15 when the size for creating a TS packet is complete. Is a processing unit.

The packet division unit 13 is a processing unit that divides the ES accumulated in the ES accumulation unit 11 into PES packets and then divides the ES into TSPs. The PES packet division unit 14 divides the ES into PES packets, and the PES packet. And a TS packet dividing unit 15 for dividing the data into TSP.
The PES packet dividing unit 14 is a processing unit that generates PES packets by reading out ESs that are sequentially stored in the ES storage unit 11 and dividing the ESs into appropriate units, and is realized by a CPU or a memory.

The TS packet dividing unit 15 reads the PES packet generated by the PES packet dividing unit 14 based on the TS packet generation instruction output from the TS packet size monitoring unit 12, and generates a TSP by dividing the PES packet in a predetermined unit. , A processing unit that outputs to the multiplexing unit 16.
The multiplexing unit 16 outputs TSP output from each TS packet dividing unit 15 for each media information, NULL packet necessary for packet control, program information, etc. (CH-1, CH-2,..., CH -N) is a processing unit that accumulates and multiplexes them into one stream.

  The media information processing apparatus 1 configured as described above operates as follows to realize a media information processing method.

FIG. 2 is a flowchart showing the processing operation of the media information processing apparatus 1.
First, the media information processing apparatus 1 starts input of media information in the encoding unit 10 (S10), encodes the media information (S12), and sequentially stores ES, that is, encoded media information, into the ES storage unit 11. Store (S14).

  Next, the media information processing apparatus 1 detects the code amount of the encoded media information held in the ES storage unit 11 in the TS packet size monitoring unit 12 (S16), and calculates the TSP based on the detected code amount. It is determined whether there is data corresponding to the size of the payload necessary for configuration (S18). In parallel with this process, the media information processing apparatus 1 reads the ESs sequentially stored in the ES storage unit 11 in the PES packet dividing unit 14 and generates PES packets by dividing the ESs in appropriate units. Yes.

  Here, when there is no data corresponding to the size of the payload (No in S18), the media information processing apparatus 1 holds the encoded media information in the ES storage unit 11 or the PES packet division unit 14 performs PES. While the packet is held in the memory, sequentially inputted media information is encoded (S12), and the same operation is repeated until data corresponding to the size of the payload is obtained.

When there is data for the size of the payload (Yes in S18), the media information processing apparatus 1 outputs a TS packet generation instruction from the TS packet size monitoring unit 12 to the TS packet dividing unit 15, and the TS packet dividing unit 15 2, the PES packetized encoded media information is converted into TS packets (S 20) and output to the multiplexing unit 16.
The media information processing apparatus 1 repeats such operations from steps S12 to S20 until the input of media information is completed (loop A), and generates a TSP.

FIG. 3 is a conceptual diagram of the media information processing apparatus 1 until the encoded media information is converted to TSP at a predetermined time interval.
As shown in FIG. 3, the media information processing apparatus 1 generates a PES packet 112 from the ESs 104 and 106, and generates TSPs 116 and 118 by dividing the PES packet 112 by a fixed length. Similarly, a PES packet 114 is generated from the ESs 108 and 110, and the PES packet 114 is divided by a fixed length to generate TSPs 120 and 122.

  Here, the media information processing apparatus 1 sends the TSP 120 after waiting for a time (α) until the ES data necessary for configuring the TSP 120, that is, the ES 108, not the time interval T from the sending of the TSP 118. Similarly, the media information processing apparatus 1 transmits the TSP 122 after waiting for a time (β) until the ES 110 is prepared, that is, the ES data necessary for configuring the TSP 122, not the time interval T from the transmission of the TSP 120.

Note that the media information processing apparatus 1 has a time interval T + α and a time interval T + β of 0.7 seconds or less, which is a value defined in the MPEG-2 system standard, and an operation rule for digital television broadcasting or the like. The transmission delay is set to a value that can be minimized within the time interval separately determined in (1).
On the other hand, when TSP is sent out at the PCR insertion interval, it is as follows.

FIG. 4 is a conceptual diagram of the media information processing apparatus 1 until the encoded media information is converted to TSP at the PCR insertion interval.
As shown in FIG. 4, the media information processing apparatus 1 generates a PES packet 212 from the ESs 204 and 206, divides the PES packet 212 by a fixed length, and generates TSPs 216 and 218. Similarly, a PES packet 214 is generated from the ESs 208 and 210, and the PES packet 214 is divided by a fixed length to generate a TSP 220.

Here, the media information processing apparatus 1 determines that the TSP 220 is not the PCR insertion interval PCR (Tn + 1−Tn) from the transmission of the TSP 218 but the ES data necessary to configure the TSP 220, that is, the time until the ES 208 and 210 are prepared. (Γ) Wait for transmission.
In this case, the media information processing apparatus 1 has a PCR insertion interval PCR (Tn + 1−Tn + γ) of 0.1 second or less, which is a value defined by the MPEG-2 system standard, and digital television broadcasting or the like. The transmission delay is set to a value that can be minimized within the PCR insertion interval that is separately determined in the operation regulations of the above.

  It should be noted that the PCR insertion interval separately defined in the operation regulations for digital television broadcasting and the like is an MPEG-2 system in order to enable fine editing by adjusting clock synchronization at short intervals. It may be shorter than the value defined in the standard. Specifically, T / N = 40 ms (where N is the number of PCRs required until synchronization and T is a desirable time until synchronization), that is, 0.04 seconds. is there.

  As described above, according to the media information processing method according to the first embodiment, the encoded media information is held until the encoded media information for the size necessary for configuring the TSP is completed, and the completed stages Since TS packets cannot be sent because the encoded media information is not aligned, the TSP can be generated without stuffing while minimizing transmission delay. .

(Embodiment 2)
Next, a media information processing method according to Embodiment 2 of the present invention will be described.
The media information processing method according to the second embodiment is substantially the same as the media information processing method according to the first embodiment, but the stuffing is performed when the TSP transmission timing is monitored and a predetermined transmission timing elapses. This differs from the media information processing method according to the first embodiment in that it is performed.

  FIG. 5 is a block diagram showing the configuration of a media information processing apparatus that implements the media information processing method according to the second embodiment.

Similar to the media information processing apparatus 1 in the first embodiment, the media information processing apparatus 2 is an apparatus that takes in media information, encodes it, divides it into TSPs, and multiplexes the TSP of each media into one stream. The same number of encoding units 20 and packetizing units 30 as the number of types of media information, and one multiplexing unit 40 are provided.
The encoding unit 20 is a processing unit that encodes media information and outputs the encoded media information to the packetizing unit 30, and includes a media information encoding unit 21, a code amount detection unit 22, and a code amount control unit 23.

  The media information encoding unit 21 takes in the media information and encodes it by a predetermined encoding method, and stores the encoded media information (ES) in the ES accumulation unit 31 of the packetizing unit 30. Here, the media information encoding unit 21 receives a code amount instruction from the code amount control unit 23, and executes encoding of the media information by changing the quantization parameter so that the code amount is generated.

The code amount detection unit 22 detects the code amount for each unit of the predetermined encoding process for the media information encoded by the media information encoding unit 21, and outputs the detected code amount to the code amount control unit 23.
The code amount control unit 23 acquires a detected code amount output from the code amount detection unit 22 and a control signal instructing adjustment of the code amount output from the TS packet size monitoring unit 32 of the packetizing unit 30. The code amount in the media information encoding unit 21 is determined and output to the media information encoding unit 21. At this time, the code amount control unit 23 determines the code amount so that the reception buffer of the decoding device that decodes the encoded media information transmitted at a predetermined bit rate does not overflow or underflow.

The packetizing unit 30 is a processing unit that generates a TSP from the ES and outputs the TSP to the multiplexing unit 40. The ES accumulating unit 31, a TS packet size monitoring unit 32, a packet transmission timing monitoring unit 33, and a packet dividing unit 34 and a stuffing unit 35.
The ES storage unit 31 is a storage device such as a memory or a hard disk for temporarily storing an ES.

  The TS packet size monitoring unit 32 checks the size of the ES stored in the ES storage unit 31 as needed, and outputs a TS packet generation instruction to the TS packet division unit 37 when the sizes for creating the TSP are prepared. . Further, when the TS packet size monitoring unit 32 acquires a timing signal indicating that the TSP transmission timing is reached from the packet transmission timing monitoring unit 33, the TS packet size monitoring unit 32 creates the TSP, and thus stores the ES stored in the ES storage unit 31. The stuffing unit 35 is instructed to insert a stuffing bit that is less than the stuffing bit, and a TS packet generation instruction is output to the TS packet dividing unit 37. Further, the TS packet size monitoring unit 32 outputs a control signal instructing the code amount control unit 23 of the encoding unit 20 to adjust the code amount based on the checked ES size, and performs bit rate control by feedback. It is possible.

The packet transmission timing monitoring unit 33 has a timer for monitoring the TSP transmission timing, and outputs a timing signal to the TS packet size monitoring unit 32 when the TSP transmission timing comes.
The packet dividing unit 34 includes a PES packet dividing unit 36 that converts the ES stored in the ES storage unit 31 into a PES packet, and a TS packet dividing unit 37 that converts the PES packet into a TS packet.

The PES packet dividing unit 36 sequentially reads out the ES accumulated in the ES accumulating unit 31, and generates a PES packet by dividing the ES into appropriate units.
The TS packet dividing unit 37 reads the PES packet generated by the PES packet dividing unit 36 based on the TS packet generation instruction output from the TS packet size monitoring unit 32, and generates a TSP by dividing the PES packet in a predetermined unit. And output to the multiplexing unit 40. Here, when the TS packet dividing unit 37 acquires the stuffing bit output from the stuffing unit 35, the TS packet dividing unit 37 generates a TSP by inserting the stuffing bit into the PES packet and outputs the TSP to the multiplexing unit 40.

When the stuffing unit 35 obtains a stuffing bit insertion instruction from the TS packet size monitoring unit 32, the stuffing unit 35 generates stuffing bits for the instructed size and outputs the stuffing bit to the TS packet division unit 37.
Similar to the multiplexing unit 16 of the first embodiment, the multiplexing unit 40 outputs TSP output from each TS packet dividing unit 37 for each piece of media information, a NULL packet necessary for packet control, program information, and the like. (CH-1, CH-2,..., CH-n) are accumulated and multiplexed into one stream.

The media information processing apparatus 2 configured as described above operates as follows to realize a media information processing method.
FIG. 6 is a flowchart showing the processing operation of the media information processing apparatus 2.
First, when the media information encoding unit 21 starts to input media information (S30), the media information processing apparatus 2 encodes the media information at a predetermined encoding rate (S32), and converts the encoded media information into the encoded media information. The data are sequentially stored in the ES accumulation unit 11 (S34). Here, in the media information processing apparatus 2, the code amount detection unit 22 detects the code amount in the media information encoding unit 21 and outputs the detected code amount to the code amount control unit 23.

  Next, the media information processing apparatus 2 detects the code amount of the encoded media information held in the ES storage unit 31 in the TS packet size monitoring unit 32 (S36), and calculates the TSP based on the detected code amount. It is determined whether or not there is data for the size of the payload necessary for configuration (S38). At this stage, the media information processing apparatus 2 stores data corresponding to the size of the payload necessary to construct the next TS packet in the ES accumulation unit 31 based on the detected code amount. The TS packet size monitoring unit 32 outputs a control signal that instructs the code amount control unit 23 to adjust the code amount. In the same manner as in the first embodiment, the media information processing apparatus 2 reads the ESs sequentially stored in the ES storage unit 31 in the PES packet dividing unit 36 in parallel with this process, To generate a PES packet.

  Here, if there is data corresponding to the size of the payload (Yes in S38), the media information processing apparatus 2 sends the TS packet size from the TS packet size monitoring unit 32 to the TS packet dividing unit, as in the first embodiment. The TS packet generation instruction is output to 37, and the TS packet dividing unit 37 converts the encoded media information that has been converted into PES packets into TS packets (S40), and outputs the TS packet to the multiplexing unit 40.

On the other hand, if there is no data for the size of the payload (No in S38), the media information processing apparatus 2 determines whether the TS packet size monitoring unit 32 has acquired a timing signal from the packet transmission timing monitoring unit 33 or not. Thus, it is determined whether or not the TS packet transmission timing has been exceeded (S42).
Here, when the timing signal has been acquired by the TS packet size monitoring unit 32 and the transmission timing of the TS packet has been exceeded (Yes in S42), the TS packet size monitoring unit 32 is insufficient to configure a TSP. The stuffing bit insertion instruction is output to the stuffing unit 35, and the TS packet generation instruction is output to the TS packet dividing unit 37. Then, the stuffing unit 35 performs stuffing processing for generating stuffing bits for the instructed size (S44), outputs the stuffing bit to the TS packet dividing unit 37, and the TS packet dividing unit 37 encodes the PES packetized encoded media. The information and the stuffing bit are combined into a TS packet (S40) and output to the multiplexing unit 40.

  On the other hand, when the TS packet size monitoring unit 32 has not acquired the timing signal and the transmission timing of the TS packet has not been exceeded (No in S42), the media information processing apparatus 2 uses the encoded media in the ES storage unit 31. The same operation is performed until the data corresponding to the size of the payload is obtained or the transmission timing of the TS packet is exceeded while the information is held or the PES packet is held in the memory by the PES packet dividing unit 36. repeat.

  Then, the media information processing apparatus 2 repeats such operations from steps S32 to S44 until the input of media information is completed (loop B), and generates a TSP. Here, the operations from step S32 to S44 are performed, and when the next loop is entered, the media information processing apparatus 2 generates the code amount specified by the code amount control unit 23 so as to generate the code amount. The media information is encoded by adjusting the encoding rate (S32).

FIG. 7 is a conceptual diagram until the encoded media information is converted to TSP through stuffing processing to TSP in the media information processing apparatus 2.
As shown in FIG. 7, the media information processing apparatus 2 generates a PES packet 312 from ESs 304 and 306, and divides the PES packet 312 by a fixed length to generate TSPs 316, 318, and 320. Similarly, a PES packet 314 is generated from the ESs 308 and 310, and the PES packet 314 is divided by a fixed length to generate TSPs 326 and 328.

At this time, the media information processing apparatus 2 is sufficient to monitor the TSP transmission timing and to configure the TSP 320 to comply with this transmission timing, although the payload 324 constituting the TSP 320 is less than the fixed length size. Since the stuffing bits 322 corresponding to the number are generated and the TSP 320 is generated by performing the stuffing process, the transmission interval of each TSP can always be kept at the predetermined time interval T. Here, the stuffing process is performed by inserting a stuffing bit 322 in the adaptation field of the TSP 320 as defined in the MPEG-2 system standard.
This stuffing process is not limited to being performed on the header portion of the TSP, but may be performed on the encoded media information itself.

FIG. 8 is a conceptual diagram of the media information processing apparatus 2 until the encoded media information is converted to TSP through stuffing processing on the encoded media information.
As shown in FIG. 8, the media information processing apparatus 2 generates a PES packet 416 from the ESs 404 and 406, divides the PES packet 416 by a fixed length, and generates TSPs 422, 424, and 426. Similarly, the PES packet 420 is generated from the ESs 412 and 414, and the TSPs 428 and 430 are generated by dividing the PES packet 420 by a fixed length.

  At this time, the media information processing apparatus 2 monitors the TSP transmission timing, although the ES408 portion, which is the actual data in the ES 406, is converted into a PES packet as it is, which is less than the fixed length size of the TSP 426. In order to comply with the TSP 426, the stuffing bits 410 that are insufficient to configure the TSP 426 are generated in advance, and the stuffing process is performed to generate the ES 406. Therefore, the stuffing bits 418 are included in the PES packet 416, and the TSP 426 is generated. As in the case described above, the transmission interval of each TSP can always be kept at the predetermined time interval T. As described above, when stuffing is performed on the encoded media information itself, stuffing processing is performed using stuffing macroblocks, USER_DATA, STUFING_START_CODE, and the like, taking MPEG-4 video encoding as an example. Also for speech coding, stuffing processing can be performed using the stuffing syntax defined by each coding method.

When stuffing is performed on the encoded media information itself, the media information processing apparatus 2 is provided with the stuffing unit 35 in the encoding unit 20, and the stuffing bits are output from the stuffing unit 35 to the media information encoding unit 21. It is necessary to change the configuration.
As described above, according to the media information processing method according to the second embodiment, when TS packet transmission timing is monitored and stuffing is necessary to comply with the transmission timing (for example, encoding that has actually occurred). Stuffing is performed only when the media information is smaller than the expected amount of data), so that TS packetization that maintains the transmission interval stipulated in the MPEG standard and broadcast operation regulations is realized while minimizing the occurrence of stuffing be able to.

(Embodiment 3)
Further, by recording the program for realizing the media information processing method shown in each embodiment on a storage medium such as a flexible disk, the processing shown in each embodiment can be performed independently. It can be easily implemented in a computer system.

FIG. 9 is an explanatory diagram in the case of implementing by a computer system using the flexible disk storing the media information processing method of the first and second embodiments.
FIG. 9B shows the appearance, cross-sectional structure, and flexible disk as seen from the front of the flexible disk, and FIG. 9A shows an example of the physical format of the flexible disk that is the recording medium body. The flexible disk FD is built in the case F, and on the surface of the disk, a plurality of tracks Tr are formed concentrically from the outer periphery toward the inner periphery, and each track is divided into 16 sectors Se in the angular direction. ing. Therefore, in the flexible disk storing the program, the medium information processing method as the program is recorded in an area allocated on the flexible disk FD.

  FIG. 9C shows a configuration for recording and reproducing the program on the flexible disk FD. When the program is recorded on the flexible disk FD, the media information processing method as the program is written from the computer system Cs via the flexible disk drive. When the media information processing method is built in a computer system by a program in a flexible disk, the program is read from the flexible disk by a flexible disk drive and transferred to the computer system.

  In the above description, a flexible disk is used as the recording medium, but the same can be done using an optical disk. Further, the recording medium is not limited to this, and any recording medium such as an IC card or a ROM cassette capable of recording a program can be similarly implemented.

(Embodiment 4)
FIG. 10 to FIG. 13 are diagrams for explaining a device that performs the media information processing described in the above embodiment, and a system that uses this device.

FIG. 10 is a block diagram showing an overall configuration of a content supply system ex100 that implements a content distribution service. The communication service providing area is divided into desired sizes, and base stations ex107 to ex110, which are fixed radio stations, are installed in each cell.
The content supply system ex100 includes, for example, a computer ex111, a PDA (personal digital assistant) ex112, a camera ex113, a mobile phone ex114, a camera via the Internet ex101, the Internet service provider ex102, the telephone network ex104, and the base stations ex107 to ex110. Each device such as the attached mobile phone ex115 is connected.

However, the content supply system ex100 is not limited to the combination as shown in FIG. 10, and any of the combinations may be connected. Further, each device may be directly connected to the telephone network ex104 without going through the base stations ex107 to ex110 which are fixed wireless stations.
The camera ex113 is a device capable of shooting a moving image such as a digital video camera. The mobile phone is a PDC (Personal Digital Communications) system, a CDMA (Code Division Multiple Access) system, a W-CDMA (Wideband-Code Division Multiple Access) system, or a GSM (Global System for Mobile Communications) system mobile phone, Alternatively, PHS (Personal Handyphone System) or the like may be used.

  In addition, the streaming server ex103 is connected from the camera ex113 through the base station ex109 and the telephone network ex104, and live distribution or the like based on the encoded data transmitted by the user using the camera ex113 becomes possible. The encoded processing of the captured data may be performed by the camera ex113 or may be performed by a server or the like that performs data transmission processing. Further, the moving image data shot by the camera 116 may be transmitted to the streaming server ex103 via the computer ex111. The camera ex116 is a device that can shoot still images and moving images, such as a digital camera. In this case, the encoding of the moving image data may be performed by the camera ex116 or the computer ex111. The encoding process is performed in the LSI ex117 included in the computer ex111 and the camera ex116. Note that image encoding / decoding software may be incorporated into any storage medium (CD-ROM, flexible disk, hard disk, etc.) that is a recording medium readable by the computer ex111 or the like. Furthermore, you may transmit moving image data with the mobile telephone ex115 with a camera. The moving image data at this time is data encoded by the LSI included in the mobile phone ex115.

  In this content supply system ex100, the content (for example, video shot of live music) taken by the user with the camera ex113, camera ex116, etc. is subjected to media information processing and transmitted to the streaming server ex103 as in the above embodiment. On the other hand, the streaming server ex103 distributes the content data to the requested client. Examples of the client include a computer ex111, a PDA ex112, a camera ex113, a mobile phone ex114, and the like that can decrypt the data subjected to the media information processing. In this way, the content supply system ex100 can receive and reproduce the encoded data at the client, and also realize personal broadcasting by receiving, decoding, and reproducing in real time at the client. It is a system that becomes possible.

The media information processing apparatus described in each of the above embodiments may be used for media information processing of each device constituting this system.
A mobile phone will be described as an example.
FIG. 11 is a diagram showing the mobile phone ex115 using the media information processing method described in the above embodiment. The cellular phone ex115 includes an antenna ex201 for transmitting and receiving radio waves to and from the base station ex110, a camera such as a CCD camera, a camera unit ex203 capable of taking a still image, a video shot by the camera unit ex203, and an antenna ex201. A display unit ex202 such as a liquid crystal display that displays data obtained by decoding received video and the like, a main body unit composed of a group of operation keys ex204, an audio output unit ex208 such as a speaker for audio output, and audio input To store encoded data or decoded data such as a voice input unit ex205 such as a microphone, captured video or still image data, received mail data, video data or still image data, etc. Recording medium ex207, and slot portion ex20 for enabling recording medium ex207 to be attached to mobile phone ex115 The it has. The recording medium ex207 stores a flash memory element which is a kind of EEPROM (Electrically Erasable and Programmable Read Only Memory) which is a nonvolatile memory that can be electrically rewritten and erased in a plastic case such as an SD card.

  Further, the cellular phone ex115 will be described with reference to FIG. The cellular phone ex115 controls the power supply circuit ex310, the operation input control unit ex304, and the image coding for the main control unit ex311 which is configured to control the respective units of the main body unit including the display unit ex202 and the operation key ex204. Unit ex312, camera interface unit ex303, LCD (Liquid Crystal Display) control unit ex302, image decoding unit ex309, demultiplexing unit ex308, recording / reproducing unit ex307, modulation / demodulation circuit unit ex306, and audio processing unit ex305 via a synchronization bus ex313 Are connected to each other.

When the end call and power key are turned on by a user operation, the power supply circuit ex310 activates the camera-equipped digital mobile phone ex115 by supplying power from the battery pack to each unit. .
The mobile phone ex115 converts the voice signal collected by the voice input unit ex205 in the voice call mode into digital voice data by the voice processing unit ex305 based on the control of the main control unit ex311 including a CPU, a ROM, a RAM, and the like. The modulation / demodulation circuit unit ex306 performs spread spectrum processing, and the transmission / reception circuit unit ex301 performs digital analog conversion processing and frequency conversion processing, and then transmits the result via the antenna ex201. In addition, the cellular phone ex115 amplifies the received signal received by the antenna ex201 in the voice call mode, performs frequency conversion processing and analog-digital conversion processing, performs spectrum despreading processing by the modulation / demodulation circuit unit ex306, and analog audio by the voice processing unit ex305. After conversion into a signal, this is output via the audio output unit ex208.

  Further, when an e-mail is transmitted in the data communication mode, text data of the e-mail input by operating the operation key ex204 of the main body is sent to the main control unit ex311 via the operation input control unit ex304. The main control unit ex311 performs spread spectrum processing on the text data in the modulation / demodulation circuit unit ex306, performs digital analog conversion processing and frequency conversion processing in the transmission / reception circuit unit ex301, and then transmits the text data to the base station ex110 via the antenna ex201.

  When transmitting image data in the data communication mode, the image data captured by the camera unit ex203 is supplied to the image encoding unit ex312 via the camera interface unit ex303. When image data is not transmitted, the image data captured by the camera unit ex203 can be directly displayed on the display unit ex202 via the camera interface unit ex303 and the LCD control unit ex302.

  The image encoding unit ex312 has a configuration including the media information processing apparatus described in the present invention, and media information processing using the image data supplied from the camera unit ex203 in the media information processing apparatus described in the above embodiment. It is converted into encoded image data by compression encoding according to the method, and this is sent to the demultiplexing unit ex308. At the same time, the cellular phone ex115 sends the sound collected by the audio input unit ex205 during imaging by the camera unit ex203 to the demultiplexing unit ex308 as digital audio data via the audio processing unit ex305.

  The demultiplexing unit ex308 multiplexes the encoded image data supplied from the image encoding unit ex312 and the audio data supplied from the audio processing unit ex305 by a predetermined method, and the multiplexed data obtained as a result is a modulation / demodulation circuit unit A spectrum spread process is performed in ex306, a digital analog conversion process and a frequency conversion process are performed in the transmission / reception circuit unit ex301, and then the signal is transmitted through the antenna ex201.

When receiving data of a moving image file linked to a home page or the like in the data communication mode, the received signal received from the base station ex110 via the antenna ex201 is subjected to spectrum despreading processing by the modulation / demodulation circuit unit ex306, and the resulting multiplexing is obtained. Is sent to the demultiplexing unit ex308.
In addition, in order to decode the multiplexed data received via the antenna ex201, the demultiplexing unit ex308 separates the multiplexed data to generate an encoded bitstream of image data and an encoded bitstream of audio data. The encoded image data is supplied to the image decoding unit ex309 via the synchronization bus ex313, and the audio data is supplied to the audio processing unit ex305.

  Next, the image decoding unit ex309 is configured to include the image decoding device described in the present invention, and a decoding method corresponding to the encoding method described in the above embodiment for an encoded bit stream of image data. To generate playback moving image data, which is supplied to the display unit ex202 via the LCD control unit ex302, thereby displaying, for example, moving image data included in the moving image file linked to the homepage . At the same time, the audio processing unit ex305 converts the audio data into an analog audio signal, and then supplies the analog audio signal to the audio output unit ex208. Thus, for example, the audio data included in the moving image file linked to the home page is reproduced. The

  It should be noted that the present invention is not limited to the above-described system, and recently, digital broadcasting using satellites and terrestrial waves has become a hot topic. As shown in FIG. be able to. Specifically, in the broadcasting station ex409, the encoded bit stream of the video information is transmitted to the communication or broadcasting satellite ex410 via radio waves. Receiving this, the broadcasting satellite ex410 transmits a radio wave for broadcasting, and receives the radio wave with a home antenna ex406 having a satellite broadcasting receiving facility, such as a television (receiver) ex401 or a set top box (STB) ex407. The device decodes the encoded bit stream and reproduces it. In addition, the image decoding apparatus described in the above embodiment can also be implemented in a playback apparatus ex403 that reads and decodes an encoded bitstream recorded on a storage medium ex402 such as a CD or DVD that is a recording medium. is there. In this case, the reproduced video signal is displayed on the monitor ex404. Further, a configuration in which an image decoding device is mounted in a set-top box ex407 connected to a cable ex405 for cable television or an antenna ex406 for satellite / terrestrial broadcasting, and this is reproduced on the monitor ex408 of the television is also conceivable. At this time, the image decoding apparatus may be incorporated in the television instead of the set top box. It is also possible to receive a signal from the satellite ex410 or the base station ex107 by the car ex412 having the antenna ex411 and reproduce a moving image on a display device such as the car navigation ex413 that the car ex412 has.

  Further, the image signal can be encoded by the image encoding device shown in the above embodiment and recorded on a recording medium. As a specific example, there is a recorder ex420 such as a DVD recorder that records an image signal on a DVD disk ex421 or a disk recorder that records on a hard disk. Further, it can be recorded on the SD card ex422. If the recorder ex420 includes the image decoding device described in the above embodiment, the image signal recorded on the DVD disc ex421 or the SD card ex422 can be reproduced and displayed on the monitor ex408.

For example, the configuration of the car navigation ex413 may be a configuration excluding the camera unit ex203, the camera interface unit ex303, and the image encoding unit ex312 in the configuration illustrated in FIG. 12, and the same applies to the computer ex111 and the television (receiver). ) Ex401 can also be considered.
In addition to the transmission / reception type terminal having both the encoder and the decoder, the terminal such as the mobile phone ex114 has three mounting formats: a transmitting terminal having only an encoder and a receiving terminal having only a decoder. Can be considered.

As described above, the media information processing method described in the above embodiment can be used in any of the devices and systems described above, and by doing so, the effects described in the above embodiment can be obtained.
Although the media information processing method according to the present invention has been described based on the respective embodiments, the present invention is not limited to these embodiments.

  For example, in each of the above embodiments, in the media information processing apparatus, the ES storage unit is shown as an independent configuration, but may be configured to be included in the packet dividing unit or the TS packet size monitoring unit.

  In addition, as described in the second embodiment, the size required to configure the TSP is increased even if the code amount is adjusted by changing the quantization parameter and the actual data size of the encoded media information is increased. If they are not aligned, stuffing processing may be performed, or the stuffing processing may be performed without adjusting the code amount if the necessary size for configuring the TSP is not prepared.

  The media information processing method according to the present invention can be applied to packetization and multiplexing processing of media information in digital television broadcasting or the like, and is particularly suitable for packet multiplexing systems such as MPEG2-TS.

It is a block diagram which shows the structure of the media information processing apparatus in Embodiment 1 of this invention. It is a flowchart which shows the processing operation of a media information processing apparatus same as the above. It is a conceptual diagram until encoding media information is converted into TSP at a predetermined time interval. It is a conceptual diagram until encoding media information is converted into TSP by the insertion interval of PCR. It is a block diagram which shows the structure of the media information processing apparatus in Embodiment 2 of this invention. It is a flowchart which shows the processing operation of a media information processing apparatus same as the above. It is a conceptual diagram until encoding media information is TSP through the stuffing process to TSP. It is a conceptual diagram until encoding media information is TSP through the stuffing process to encoding media information. It is explanatory drawing about the recording medium for storing the program for implement | achieving the media information processing method of each embodiment by a computer system. It is a block diagram which shows the whole structure of a content supply system. It is an external view of a mobile phone. It is a block diagram which shows the structure of a mobile telephone. It is a figure which shows the example of the system for digital broadcasting. It is a figure for demonstrating the outline | summary of MPEG-2 TS. It is a conceptual diagram until it converts into ES after packetizing ES which encoded media information with the conventional media information processing method. It is a conceptual diagram in the case of constructing a PES packet at a PCR insertion interval by a conventional media information processing method and generating a TSP.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1, 2 Media information processing apparatus 10, 20 Encoding part 11, 31 ES storage part 12, 32 TS packet size monitoring part 13, 34 Packet division part 14, 36 PES packet division part 15, 37 TS packet division part 16, 40 Multiplexing unit 21 Media information encoding unit 22 Code amount detection unit 23 Code amount control unit 30 Packetization unit 33 Packet transmission timing monitoring unit 35 Stuffing unit

Claims (10)

A media information processing method for encoding media information and packetizing the encoded media information into a predetermined data size,
An encoding step for capturing and encoding the media information;
An encoded media information holding step for temporarily holding the encoded media information;
A packet size monitoring step for monitoring whether or not the encoded media information held in the encoded media information holding step is equal to the predetermined data size within a predetermined time; and
The packet size monitoring step includes a packetizing step of packetizing the encoded media information after detecting that the encoded media information has been prepared for the predetermined data size. Information processing method. The media information processing method according to claim 1, wherein the predetermined time is a transmission interval of packets constituting each frame of media information determined in advance. The media information processing method according to claim 1, wherein the predetermined time is a predetermined clock cycle for transmitting a packet defined by a standard related to packetization of the encoded media information. The media information processing method according to claim 1, wherein the predetermined time is a predetermined clock cycle for transmitting a packet defined in a broadcast operation rule relating to packetization of the encoded media information. The media information processing method further includes:
The packet size monitoring step includes a stuffing step of adjusting a packet to have the predetermined data size when it is detected that the encoded media information is not equal to the predetermined data size. The media information processing method according to any one of claims 1 to 4. In the stuffing step, the encoding rate in the encoding step is adjusted so that the encoded media information is the predetermined data size,
The media information processing method according to claim 5, wherein, in the encoding step, the media information is encoded at a rate adjusted in the stuffing step. In the stuffing step, a stuffing bit that compensates for the amount less than the predetermined data size is generated,
6. The media information processing method according to claim 5, wherein in the packetizing step, the encoded media information and the stuffing bits are combined into a packet. A media information processing apparatus that encodes media information and packetizes the encoded media information into a predetermined data size,
Encoding means for capturing and encoding media information;
Encoded media information holding means having a storage area for temporarily storing encoded media information;
Packet size monitoring means for monitoring whether or not the encoded media information held in the encoded media information holding means is equal to the predetermined data size within a predetermined time;
The packet size monitoring means comprises packetizing means for packetizing the encoded media information after detecting that the encoded media information has been prepared for the predetermined data size. Information processing device. The media information processing apparatus further includes:
When the packet size monitoring unit detects that the encoded media information has not been prepared for the predetermined data size, the packet size monitoring unit includes a stuffing unit that adjusts the packet to have the predetermined data size. The media information processing apparatus according to claim 8. A program for a media information processing apparatus that encodes media information and packetizes the encoded media information into a predetermined data size,
An encoding step for capturing and encoding the media information;
An encoded media information holding step for temporarily holding the encoded media information;
A packet size monitoring step for monitoring whether the encoded media information held in the encoded media information holding step is equal to the predetermined data size within a predetermined time; and
In the packet size monitoring step, after detecting that the encoded media information has been prepared for the predetermined data size, a packetizing step for packetizing the encoded media information is executed by a computer. Program.

Images