JPH10276407A - Video information providing management method and system therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information providing management method and its system which read each information such as video, audio and character graphics from a transmitting packet at a high speed and easily reproduce and composite it. SOLUTION: A server controlling part 11 of a multimedia information providing server 1 generates a packet header to which the frame number of a video frame, the existence of a start code, a byte position and a bit position are added for video information in a packet that is encoded and divided, and adds the byte position of character graphic information which performs synchronous display of the video information to it. An information transmitting part 14 transfers the packet header together with a packet to a multimedia information receiving client 2. The client controlling part 22 of the client 2 fast reads video information and character graphic information in a frame unit by reading and referring to the packet header at the time of receiving the video information, synchronizes it with video sound in a frame interval which is defined based on the frame number that is referred to and further synchronizes with and displays the character graphic information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a real-time video information providing service such as video-on-demand which provides video information via a network. The present invention relates to a video information provision management technology that enables synchronous reproduction with character / graphic information.

[0002]

2. Description of the Related Art In recent years, with the spread of the Internet, the improvement of arithmetic processing performance of personal computers, and the improvement of image processing techniques, demands for image communication services using communication lines have increased. The encoding format of image information applicable to these image communication services is MPEG1 (international standard ISO / IEC1) used in video CDs.
1172, 1993, 8) and H.264 mainly used for TV telephones. 261 (ITU-T Rec. H. 261:
“Video Codec For AudioVis
ual Services at p × 64kbit /
s ", Mar. 1993).

FIG. 7 is a diagram showing the structure of an MPEG1 data format that defines the encoding format of video information. A PSC (Pic) is used as a code indicating the head position of video frame information (a video frame is called a picture in MPEG).
cure start code).
Further, as an upper layer, a sequence layer composed of video frames (pictures) corresponding to 0.4 to several seconds and an SHC (Sequence Head) indicating the head position thereof.
er Code) and TC (T
image Code) is defined. Also, in order to ensure that the sequence header and the start code of the picture layer are arranged at byte boundaries, an arbitrary number of 0s are inserted immediately before the start code.

FIG. 8 is a diagram showing a data format of an MPEG1 system which defines a multiplexing method of video information and audio information. In order to realize output synchronization between video information and audio information, a video packet and an audio packet are merged. Data format. Control information is arranged at the head of the video packet and the audio packet in order to maintain the output synchronization of the video information and the audio information, and PTS (Pres
and the decoding time management information DTS (Decoding Time).
(Stamp) is added.

On the other hand, H. 261 is H.264. 221 (CCIT
T Rec. H. 221: "Frame Struct
ure for a 64 to 1920 kbit /
sChannel in Audio Visual
Teleservices ", Blue Book, F
assistle III. 6, 1988), a method for multiplexing video information and audio information is defined by a bit-based synchronous transmission method, but multiplexing in a packet format is not specified. H. In FIG. 26, FIG.
H. As shown in the configuration diagram of the H.261 data, a flag indicating the head position of a video frame is a PSC (Picture Start Code) having the same name as MPEG1.
e) and a video frame number TR (Te
Mental Reference) is defined. H. The PSC of H.261 uses the same name as MPEG1, but differs in length and allocation code. Also, M
In PEG1, it was located at a byte boundary,
H. In H.261, it is arranged at an arbitrary bit unit position.

[0006]

As described above, in the conventional video / audio information encoding format, since the frame information such as the head position of the video frame and the display interval of the frame is included in the video information. However, there were the following problems.

(1) MPEG1 employs a method of arranging a start code of video frame information at a byte boundary so that a start code can be searched relatively easily.
However, in this method, it is necessary to insert a dummy bit in order to arrange the start code on a byte boundary, and at the same time, there is a problem that the amount of data increases, and the information received because the start code does not always exist at the beginning of a transmission packet. Has to be searched.

(2) H. In H.261, the start code of a video frame does not always exist at a byte boundary, but in order to search for a start code from an arbitrary byte position, processing must be performed on a bit-by-bit basis, which requires a great deal of processing time.

(3) Character / graphic information synchronized with video information and audio information cannot be provided.

[0010] These processes cause problems such as an increase in transmission capacity and an increase in load on a server and a client device, particularly in a system in which information is retrieved via a communication line. Furthermore, there is a problem that it is impossible to synchronize the reproduction of multimedia information combining video and graphics.

The present invention relates to H. With regard to encoded video information generated in bit units such as H.261 and MPEG1, a start code of a video frame included in a transmission packet can be searched at high speed by simple processing, and video information and audio information or video information can be searched. It is an object of the present invention to provide a video information provision management method and system that facilitates reproduction synchronization of audio information and character / graphic information.

[0012]

SUMMARY OF THE INVENTION In order to solve the above problems, a method according to the present invention encodes video information and audio information, or multimedia information including them and character / graphic information, and divides them into packets to form a communication network. And receiving the video information and the audio information or the video information and the audio information and the text / graphic information by searching for the start position of the video information from the packet. In the transmitting step, the video information in the packet is searched for information on the presence or absence of a start code of a video frame and the position of the start code, and the information is added thereto, or the video is added to the information. A packet header to which information about the position of the character / graphic information to be synchronized with the information display is added. And having the steps of creating, a, and transmitting the created packet header with a packet of the multimedia information.

In the receiving step, a step of separating a packet header transmitted together with the multimedia information may be performed, and a start code of a video frame or a start code of a video frame may be calculated from information added to the separated packet header. Searching for character / graphic information to be displayed in synchronization with the video frame, and controlling reproduction synchronization with audio information at video frame intervals described in the packet header.

In the step of performing the reproduction synchronization control,
Starting the reproduction of one of the audio information and the video information and calculating the reproduction end time of the one, calculating the reproduction end time of the other, and calculating the other reproduction end time that is equal to or less than the one reproduction end time. The reproduction of the audio information or the video information is started.

[0015] Similarly, in order to solve the above problems, the system according to the present invention divides encoded video information and audio information, or encoded video information and audio information and character / graphic information, and alternately arranges them. A multimedia information creating unit for creating multimedia information in a format, a multimedia information storage unit for storing the created multimedia information, and packetizing the stored multimedia information, A packet header to which a start code byte position of a video frame included in the information or information including a start code byte position and a start code bit position, or information for managing display synchronization of character / graphic information in addition to these is created. A transmission control unit, and the packetized multimedia An information transmitting unit for transmitting through the communication network by adding the created packet header to restore information,
And a multimedia information providing server having the following.

An information receiving unit for receiving the packetized multimedia information via the communication network;
Video information and audio information, or video information and audio information, character / graphic information, and a packet header are separated from the received packet information and stored, and the information added to the packet header and the description in the packet header are stored. The video information and the audio information based on the video frame interval, and a reception control unit for controlling the reproduction synchronization of the character / graphic information; and the separated video information and audio information, or the video information and the audio information, and the character / graphic information. A multimedia information storage unit for storing, a frame management information storage unit for storing the separated packet header, an input instruction unit for giving an instruction including start and stop of reproduction to the multimedia information providing server, and the video information Or video information and text / graphic information Wherein the information display unit that displays, an audio output unit which outputs the audio information, in that it has a multimedia information receiving client provided with the.

According to the present invention, (1) when transmitting coded video information and audio information, or character / graphic information in addition to these, for example, by dividing them into fixed-length packets, Searching for the start code of the video frame, the video frame number, the byte position or byte position and bit position of the start code, and creating a packet header to which the information is added; (2) Synchronizing with the display of the video information If there is character / graphic information to be made, the byte position and bit position are embedded in the packet header, and (3) the packet header created above is transferred together with the packet-divided multimedia information to obtain (a) video information. By reading and referencing only the packet header at the time of reception, the video frame is Cry video information or video information as well as the graphic character data can be read out at high speed,
(B) The video information can be displayed in synchronism with the voice or character / graphic information at the frame interval specified based on the referenced video frame number.

[0018]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 is a block diagram showing an embodiment of the present invention. In FIG. 1, 1 is a multimedia information providing server, 2 is a multimedia information receiving client, and 3 is a communication line connecting them. The multimedia information providing server 1 includes a server control unit 11,
It has a multimedia information creation unit 12, a multimedia information storage unit 13, and an information transmission unit 14. On the other hand, the multimedia information receiving client 2
1, client control unit 22, information receiving unit 23, frame management information storage unit 24, multimedia information storage unit 2
5, an information display unit 26, and an audio output unit 27.

A process of transmitting and receiving multimedia information and displaying multimedia information at a specified frame interval with reference to a video frame number will be described below with reference to FIG. FIG. 2 shows a message sequence example at this time.

In this embodiment, H.264 is used as the video encoding format. 261 as an audio encoding format.
711 (CCITT Rec. G. 711: “64bi”
t / s Pulse Code Modulatio
n ", Blue Book, Fascicle II
I. 4, 1988) will be described.
Obviously, other encoding formats based on T and other audio encoding formats can be easily realized.

First, in the multimedia information creating section 12 of the multimedia information providing server 1, the video and audio signals input from the video camera or the video deck and the character / graphic information input from the mouse and keyboard are sent to the server control section. 11 by the control of H.11. 26
1 and G. The data is converted into bit stream data of 711 and code information and stored in the multimedia information storage unit 13.

Next, in accordance with the search instruction from the input instruction section 21 of the multimedia information receiving client 2, the server control section 11 deletes the video information, audio information, and character / graphic information stored in the multimedia information storage section 13. The information is divided by a predetermined length, merged into fixed-length packet data as shown in FIG.
Multimedia information receiving client 2 via
Send to Upon receiving the last multimedia packet data, the multimedia information receiving client 2 transmits a reception completion notification to the video information providing server 1, and the communication ends.

Here, the following processing is performed to write information for determining the presence / absence of the start code of the video frame and the start code byte position in the packet header. FIG. 3 shows a processing flow for creating and transmitting a packet header.

First, PSC (Picture) which is a start code of a video frame is obtained from video information in a packet.
Start Code); In 261, a 20-bit bit string “00010 (hexadecimal representation)” is searched,
When the start code is included, the packet number P, the number m of video frames included in the packet, the frame number v indicating a sequential number of the video frame, and the byte position including the start of the start code as shown in FIG. , And a packet header consisting of a start code bit position b indicating an offset of the start code from the byte boundary is generated. When a plurality of video frames are included in a packet, the number of start codes included in the packet is embedded, and a frame number, a byte position, and a bit position are calculated and added to all the included start codes. When the start code is not included, 0 is embedded in the number of start codes, and the frame number, the byte position, and the bit position are omitted. Here, the sequential frame number is TR (Tempora) immediately after the PSC.
It can be calculated by reading the skipped frame number from (l Reference). That is, assuming that the frame number before skipping is v and the number of skipped frames is s, the skipped frame number to be read is v + s, and can be calculated by adding 1 to this. If the start code is not included, a dummy packet header is added. Further, time control information (for example, a time stamp or an execution delay command) included in the character / graphic information is searched, and a byte position (character / graphic information start byte position) C corresponding to the display time of the video frame is calculated. Add to header.

The packet header created as described above is added to the beginning of the multimedia information shown in FIG.
The information transmitting unit 14 is transmitted to the multimedia information receiving client 2 via the communication line 3.

Next, the packet receiving process will be described. FIG. 6 shows a processing flow of packet reception and synchronous reproduction.

The transmitted packet information is depacketized by the information receiving unit 23 under the control of the client control unit 22, separated into a packet header, video information, audio information, and character / graphic information. 24, multimedia information storage unit 2
5 is stored. Thereafter, video information, audio information, and character / graphic information designated for retrieval by the same processing are packetized, a packet header is added thereto, and the packet information is transmitted to the multimedia information receiving client 2. The frame management information storage unit 24, the multimedia information It is stored in the storage unit 25.

Simultaneously with the reception of the packet information, the client control section 22 reads the audio information of one packet stored in the multimedia information storage section 25, transfers it to the audio information output section 27, and outputs it. The reproduction end time Ta from the point is calculated. 6
G. 4 kbps. In the case of 711, 10 KB is 1.25 seconds. Subsequently, the start code information of the second frame is read from the packet header stored and stored in the frame management information storage unit 24, the frame number v, the start byte position B, and the character / graphic byte position C are read, and the first video is read. The reproduction end time Tv of the frame (including the skip frame) is calculated by (v-1) / F. Where F is 1
The number of frames per second. When the frame number is 5, it takes about 0.17 seconds. The reproduction end time Ta of the audio information is compared with the reproduction end time Tv of the video information.
If v is smaller than or equal to Ta, the video information stored in the multimedia information storage unit 25 is read up to the second start code byte position and read from the information display unit 2.
6 to display the first video frame. At the same time, the character / graphic information byte position B is referred to, and the character / graphic information up to the byte position is read and displayed. Similarly, referring to the frame numbers of the third and subsequent video frame information, the reproduction end time of the video frame is calculated,
By comparing with the playback end time of the audio information,
Realizes synchronous playback of audio, video, and character / graphic information.

In the embodiment, the byte position and the bit position are recorded in the packet header. However, since the start code of the video frame is known to start from a narrow range of 0 to 7 bits, Obviously, it is easy to detect without recording the bit position. Further, in the above embodiment, the processing is performed for each start code, but the processing may be performed for a plurality of start codes in accordance with the buffer capacity of the information display unit 26 and the audio information output unit 27. . Furthermore, in the present embodiment, the sequence is such that audio information is output first, but it is apparent that the same can be realized by outputting video information and character / graphic information first.

[0031]

As described above, according to the present invention,
The frame management information can be easily separated from the received packet information, and the video frame start code position and the display interval can be searched at high speed by simple processing. Therefore, one frame of video information and character / graphic information can be extracted at high speed from the received multimedia information, and can be reproduced in real time in synchronization with audio for a specified display time. Also,
Dummy bits for setting the start code of a video frame on a byte boundary and frame management information for controlling the display time can be separated from the video information, so that the amount of transmitted information can be reduced.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of the present invention.

FIG. 2 is a diagram showing a message sequence.

FIG. 3 is a flowchart of a packet transmission process in the embodiment.

FIG. 4 is a diagram showing a packet format obtained by merging video information and audio information in the embodiment.

FIG. 5 is a diagram showing a packet header format for managing a video frame start code included in video information in a packet in the embodiment.

FIG. 6 is a flowchart of a packet reception process in the embodiment.

FIG. 7 is a diagram showing a data format of MPEG1 which is a conventional technique.

FIG. 8 is a diagram showing a data format of an MPEG1 system in which video information and audio information are merged according to a conventional technique.

FIG. 9 shows a conventional technique, H.264. FIG. 261 is a diagram illustrating a data format of H.261.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Multimedia information provision server 11 ... Server control part 12 ... Multimedia information creation part 13 ... Multimedia information storage part 14 ... Information transmission part 2: Multimedia information reception client 21 ... Input instruction part 22 ... Client control part 23 ... Information receiving unit 24 ... Frame management information storage unit 25 ... Multimedia information storage unit 26 ... Information display unit 27 ... Audio output unit 3 ... Communication line

Claims (5)

[Claims] 1. Encoding multimedia information including video information and audio information, or multimedia information including them and character / graphic information, dividing the packet into packets, and transmitting the packets through a communication network, and searching a start position of the video information from the packets. A video information providing management method, comprising: a receiving step of synchronizing the video information with the audio information or the video information with the audio information and the character / graphic information. Information about the presence or absence of a start code and the position of the start code were searched for and added to this information, or in addition to this information, information about the position of the character / graphic information synchronized with the display of the video information was added. Creating a packet header; and Transmitting the multimedia information together with the packet of the multimedia information. 2. The method according to claim 2, wherein in the receiving step, a packet header transmitted together with the multimedia information is separated, and a start code of a video frame or a start code of a video frame is obtained from information added to the separated packet header. A step of retrieving character / graphic information to be displayed in synchronization with the video frame, and performing reproduction synchronization control with audio information at a video frame interval described in the packet header. Video information provision management method described in. 3. In the step of performing the reproduction synchronization control, the reproduction of one of the audio information and the video information is started, the reproduction end time of one of the audio information and the video information is calculated, and the reproduction end time of the other is calculated. 3. The video information provision management method according to claim 2, wherein the reproduction of audio information or video information having the other reproduction end time that is equal to or shorter than the one reproduction end time is started. 4. A multimedia information creation section for creating multimedia information in a format in which encoded video information and audio information, or encoded video information and audio information and character / graphic information are divided and alternately arranged. A multimedia information storage unit for storing the created multimedia information; and packetizing the stored multimedia information, at the same time as a start code byte position or start of a video frame included in the video information in the packet. A transmission control unit that creates a packet header to which information including a code byte position and a start code bit position, or information for managing display synchronization of character / graphic information in addition to the information, and the packetized multimedia information To the communication network by adding the created packet header to A video information provision management system, comprising: a multimedia information provision server comprising: an information transmission unit configured to transmit the information via a multimedia information provision server. 5. An information receiving unit for receiving the packetized multimedia information via the communication network; and video information and audio information from the received packet information; Information and a packet header are separately stored, and the reproduction synchronization control of the video information, audio information, and character / graphic information is performed based on the information added to the packet header and the video frame interval described in the packet header. A multimedia information storage unit for storing the separated video information and audio information, or the video information and audio information, character / graphic information, and a frame management information storage for storing the separated packet header. And a multimedia information providing server Multimedia information, comprising: an input instructing unit that issues instructions including start and stop of reproduction, an information display unit that displays the video information or the video information and character / graphic information, and an audio output unit that outputs the audio information. The video information provision management system according to claim 4, further comprising a receiving client.