CN1941889B - Method and apparatus for transmitting and receiving dynamic image data - Google Patents

Method and apparatus for transmitting and receiving dynamic image data Download PDF

Info

Publication number
CN1941889B
CN1941889B CN 200610099755 CN200610099755A CN1941889B CN 1941889 B CN1941889 B CN 1941889B CN 200610099755 CN200610099755 CN 200610099755 CN 200610099755 A CN200610099755 A CN 200610099755A CN 1941889 B CN1941889 B CN 1941889B
Authority
CN
China
Prior art keywords
data
information
priority
time
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
CN 200610099755
Other languages
Chinese (zh)
Other versions
CN1941889A (en
Inventor
山口孝雄
栄藤稔
荒川博
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Panasonic Holdings Corp
Original Assignee
Matsushita Electric Industrial Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Matsushita Electric Industrial Co Ltd filed Critical Matsushita Electric Industrial Co Ltd
Publication of CN1941889A publication Critical patent/CN1941889A/en
Application granted granted Critical
Publication of CN1941889B publication Critical patent/CN1941889B/en
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Landscapes

  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

The invention comprises a reception management unit (11) which receives data and information including transmission format information from a storage unit or through a communication line, a separator unit (12) for analyzing and separating the received information, a transmission unit (13) for transmitting information to the storage unit or through a communication line, an image-expanding unit (14) for expanding the image, an image expansion mangement unit (15) which manages the processing state of the image-expanding unit (14) that expands one or more images, an image synthesizing unit (16) for synthesizing an image based on the expanded data, an output unit (17) for outputting the results of synthesization, and a terminal control unit (18) for controlling all the above-mentioned means, wherein a plurality of images are formed simultaneously and which can cope with a dynamic change of the transmission format information.

Description

The method and the equipment thereof of transmission and receiving dynamic image data
The present invention is that international filing date is March 13, application number in the 1998 (international application no: PCT/JP98/01084), denomination of invention divides an application for the method and the equipment thereof of receiving dynamic image data " send and " that is 200410032361.0.In addition, application number is that 200410032361.0 application itself is that international filing date is that March 13, application number in 1998 are 98800657.X (international application no: PCT/JP98/01084), denomination of invention divides an application for the method and the equipment thereof of receiving dynamic image data " send and ".
Technical field
The present invention relates to a kind of audio-video transmitter and audio-video receiver, data processing equipment and method, Wave data sending method and equipment and Wave data method of reseptance and equipment and video transmission method and equipment and video receiving method and equipment.
Background technology
Existing a kind of equipment, it has satisfied the in esse requirement of the other side's image of feeling that you occur in front, its purpose is to carry out real world images communication, this is by extract someone image from the landscape image that you are for example arranged, then, with this person's the image (being someone image that the other side sends) and the virtual space image mutual superposition that will show, and shown (the Japanese patent application No.4-24914) that realizes with the other side's image of storage in advance.
Specifically, under the situation of prior art, had that to relate to the accelerogram picture synthetic and reduce some inventions (Official Journal of the open No.5-46592 of Japan Patent for example: image synthesizer) of the method for memory.
Though prior art has proposed to utilize the synthetic communication system that bidimensional rest image or three-dimensional CG data are synthesized of image, but from following viewpoint, synthetic simultaneously multiple image and multiple sound and the system that shows, its implementation is not also specifically discussed.
That is, the method for enumerating from below, existing does not have the concrete problem of discussing.
(A1) a kind of method, its utilizes the environment that transmits data and control information (by the information of the data packet transmission different with control terminal side deal with data) independently at many logical transport circuits that are made of software on the one or more actual transmissions circuit to transmit (communicate by letter and broadcast) and control chart picture and sound down.
(A2) a kind of method, it dynamically changes will be added to image to be sent or the header message in the voice data (corresponding to control information of the present invention);
(A3) a kind of method, it dynamically changes the header message (corresponding to control information transmission of the present invention) that will add, so that transmit;
(A4) a kind of method, it by dynamic moving multiple connection with separate many logical transport circuits and come transmission information;
(A5) a kind of method, it has considered reading of program or data and growth cycle, with images and sound;
(A6) a kind ofly quick switching images and sound method have been considered.
Yet, with regard to the data volume that dynamic adjustment will be transferred to network, method (the distribution self-adaptation type VOD systematic research that H.Jinzenji and T.Tajiri write that has proposed to change the method for coded system and frame data are discussed according to the type of picture frame, D-8 publishes in the D-8 (1995) of electronics, information and society of system of communication engineering association (IEICE)).
The dynamic throughput conversion algorithm that high quality graphic can be provided under the restricted processing time has proposed as a kind of throughput of adjusting coder side that (the dynamic throughput conversion of the utilization algorithm that T.Osako, Yajima, H.Kodera, H.Watanabe, K.shimamura etc. write is encoded to software video, publish paper periodical D-2 in IEICE, Vol.80-D-2, the second phase, the 444-458 page or leaf, 1997).
And, exist the MPEG1/MPEG2 system as the example of realizing reproduced in synchronization image and sound.
(B1) conventional method of image being discussed corresponding to the type of picture frame has such problem, promptly by handling a plurality of video data streams or a plurality of audio data stream, be difficult to sound synchronously emphasis reproduce important scene, reflection editor's intention, this is because the grade of manageable information is arranged in single information flow.
(B2) and, because its necessary condition is to realize MPEG1/MPEG2 with hardware, so decoder must be decoded to each bit stream that provides.Therefore, its problem is exactly how to adapt to the situation of decoder by output that exceed.
In addition, for images, more existing systems, for example H.261 (the professional video coder-decoder of ITU-T suggestion px.64 acoustic image H.261), and they use hardware composing.Therefore, owing to considered the upper limit of necessary performance during hardware, so there not be generation to finish the situation of decoding in can not be at the appointed time in design.
The top fixed time represents to send the needed time of bit stream that image of coding obtains.If decoding can not be finished in this time, its extra time just becomes delay.If this delay accumulation, then transmitter side to receiver side just increase of delay, system just can not be used as video telephone.This state must be avoided.
And, when finishing decoding in can not be at the appointed time because communication counterpart has produced the bit stream beyond the regulation, then with regard to produce can not images problem.
The problems referred to above not only are created in the video data, and are created in the voice data.
Yet, in recent years, because the expansion that the layout of personal computer (PC) network environment that forms is brought the Internet and ISDN so transmission rate improves, can utilize PC and Network Transmission image.And the requirement of user's images also grows at top speed.Again since cpu performance improve, so can come decoding with software fully to vision signal.
Yet, because different personal computer on the structure, for example the different personal computers in aspect such as CPU, highway width or accelerator can be carried out identical software, are difficult so consider the upper limit of necessary performance in advance, therefore, produced the at the appointed time problem of interior decoding of image.
And, when sending the video data encoder that length surpasses the receiver throughput, finish coding in just can not be at the appointed time.
Problem (C1): the delay of decoded picture in reducing at the appointed time.
When the Wave data that vision signal is entered as claim C1 of the present invention, perhaps the Wave data that vision signal is output into claim C7 of the present invention is when dealing with problems 1 means, remaining issues is not utilize the real work efficient reduction of transmission line because part is transmitted bit stream.And, exist some according to last coded image (for example P image), produce the coded system of current decoded video.Yet,, change in time and make the serious problem of deterioration in image quality so exist because last decoded picture recovers fully by 1 the means of dealing with problems.
Problem (C2): under the situation of 1 means of dealing with problems, the real work efficient of transmission line reduces.And, enlarged deterioration of image quality.
Under the situation of installing with software, the frame rate of image is by the needed time appointment of once encoding.Therefore, when the frame rate of user's appointment surpasses the throughput of computer, can not meet this appointment.
Problem (C3): when the frame rate of user's appointment surpasses the throughput of computer, can not meet this appointment.
The problem (A1) of considering first kind of prior art the object of the present invention is to provide a kind of audio-video transmitter and audio-video receiver and data processing equipment and method, one of to address the above problem at least to (A6).
And the problem (B1) of considering second kind of prior art another object of the present invention is to provide a kind of data processing equipment and method, one of to address the above problem at least to (B2).
Again, the problem (C1) of considering the third prior art is to (C3), a further object of the present invention is to provide a kind of Wave data method of reseptance and equipment, Wave data sending method and equipment and video transmission method and equipment and video receiving method and equipment, one of to address the above problem at least.
Summary of the invention
The 1st aspect of the present invention is a kind of audio-video transmitting apparatus, it comprises dispensing device, sends the structure of the data that relate to the content of transmission method and/or will send or the identifier that content is represented as the transmission format information by transmission line identical with the transmission line of the data that will send or the transmission line different with this data transmission link; Wherein
The described data that will send are video data and/or voice data.
In audio-video transmitting apparatus of the present invention, described transmission format information be included at least be added on the described data with the data control information of controlling described data, be added on the described data with send described data send control information and the information thrin of the processing of control terminal side in.
In audio-video transmitting apparatus of the present invention, described at least data control information, sending control information and controlling the information thrin that described end side handles dynamically changes.
In audio-video transmitting apparatus of the present invention, described data are divided into a plurality of packets, and described data control information or described sending control information not only are added to the first packet of described partition data bag, also are added in the intermediate data bag.
In audio-video transmitting apparatus of the present invention, will whether use the timing information relevant to be included in the described transmission format information as the identifier of the information representation of the recovery time of the described data of expression with described data.
In audio-video transmitting apparatus of the present invention, described transmission format information is the structural information of described data, after the signal of receiving equipment output that receives the transmission structural information of described data confirmed that this structural information can receive, described dispensing device sent corresponding data to described receiving equipment.
In audio-video transmitting apparatus of the present invention, described transmission format information comprises that the identifier of program that (1) sign receiving equipment uses later on or data and (2) are as knowing the special time that described program or data are used or using the mark, counting of information of the term of validity of described program or data and regularly at least a among the three.
In audio-video transmitting apparatus of the present invention, utilization transmits Sequence Number the described special time of described program or data use is sent as sending control information, with the identification transfer sequence, perhaps the information that will send as the packet different with the data of control terminal side processing is come.
In audio-video transmitting apparatus of the present invention, also comprise storage device, be used to store multinomial structure and a plurality of identifier thereof that relates to the content and/or the described data that will send of described transmission method, described identifier is included in described data control information at least, send control information and the information thrin of the described transmission format information of conduct that the control terminal side is handled in.
In audio-video transmitting apparatus of the present invention, also comprise storage device, be used to store the multinomial content of described transmission method and/or the structure of the described data that will send of relating to, described content is included in described data control information at least, send control information and the information thrin of the described transport format information of conduct that the control terminal side is handled in.
In audio-video transmitting apparatus of the present invention, increased the default designation symbol, whether change the structure of content that relates to described transmission method and the data that maybe will send with expression.
In audio-video transmitting apparatus of the present invention, described identifier or described default designation symbol be added to information to be sent the fixing presumptive area of length in or on the described precalculated position.
Audio-video receiving equipment of the present invention comprises: receiving system receives the described transmission format information that above-mentioned any audio-video transmitting apparatus sends; Send the information interpretation device, explain the transmission format information of described reception.
In audio-video receiving equipment of the present invention, also comprise storage device, be used to store multinomial structure and a plurality of identifier thereof that relates to the content and/or the described data that will transmit of transmission method; The content that is stored in the described storage device is used to explain described transmission format information.
Audio-video transmitting apparatus of the present invention, it comprises: the information multiplex machine is used to control multiplexed transmission data and/or control information; The beginning of the information of many logical transport lines and end; Wherein, not only described information multiplex machine multiplexed described data and/or control information are sent as multiplexed control information, also described multiplexed beginning and end control contents with described information multiplex machine are sent as multiplexed control information, described data comprise video data and/or voice data.
It is multiplexed to select to arrange described multiplexed control information not carry out in audio-video transmitting apparatus of the present invention, thereby before described data and/or control information, send described multiplexed control information, perhaps send described multiplexed control information and do not carry out multiplexed by the transmission line different with the transmission line of described data of transmission and/or control information.
Audio-video receiving equipment of the present invention comprises: receiving system receives described multiplexed control information and the described multiplexed data and/or the control information of the audio-video transmitting device transmission of claim 15; Separator separates described multiplexed data and/or control information according to described multiplexed control information.
Audio-video receiving equipment of the present invention comprises: main audio-visual apparatus is used to watch and the broadcast listening program; The audio-visual support device is used for periodically detecting the state with the different broadcast programs of broadcast program of watching by described main audio-visual apparatus and listening to; Wherein carry out described detection so that when the described broadcast program of watching by described main audio-visual apparatus and listen to is switched to other broadcast program, can handle required program and/or data smoothly, described data comprise video data and/or voice data.
Preferred value can be according to circumstances in audio-video transmitting apparatus of the present invention, and the deviant of the information of the priority by sending the expression processing said data changes.
Audio-video receiving equipment of the present invention comprises: receiving system receives the coded message that is added to the information relevant with the priority of handling in advance under overload; Priority is determined device, with determining threshold value, as the standard of selecting whether to handle the object in the described information that described receiving system receives; Wherein
The time of the described reception information of output with after beginning to handle through time ratio, perhaps the decoding described reception information time with begin to handle after through time ratio, changing described threshold value according to comparative result, and comprise video data and/or voice data, as coded object.
Comprise that in audio-video receiving equipment of the present invention re-send request may priority determines device, when being used for needs and retransmitting described information, determine threshold value, some are lost in transmission and the standard of the described information that do not receive as selecting request repeat whether, and described definite threshold value be at least according to described priority determine device control priority, retransmit between number of times, information dropout factor, the intracoded frame insertion at interval and priority one of divide and to determine.
Audio-video transmitting apparatus of the present invention, comprise: retransmit priority and determine device, when being used for that request repeat is described not to receive information, determine threshold value, some lose the standard of the described information that does not receive when sending as selecting request repeat whether, wherein, described definite threshold value be at least according to the priority of the described audio-video receiving equipment of claim 20 determine device control priority, retransmit between number of times, information dropout factor, the intracoded frame insertion at interval and priority one of divide and to determine.
Audio-video transmitting apparatus of the present invention, the priority that utilization is added in the described coded message is transmitted described coded message, thereby compare the result with decoding that is added to described coded message or output cycle and judge that described coded message is written to transmit buffer to be postponed, and then makes the minimizing of transmission information beginning to transmit the back elapsed time if (1) actual transfer rate surpasses video or audio-frequency information target transmission speed or (2).
Data processing equipment of the present invention, it comprises: receiving system, receiving data sequence, this data sequence comprises (1) audio or video time series data, (2) priority between the time series data of the priority of handling between the described time series data value of expression, and priority in (3) a plurality of time series datas, the latter is used to divide described time series data value, with the processing priority between the expression institute dividing data value; Data processing equipment when occurring a plurality of described time series data value simultaneously, utilizes between described mutual time series data in the priority and described time series data priority to handle simultaneously.
Data processing equipment of the present invention, comprise: receiving system, receive DS, this DS comprises: (1) audio or video time series data, (2) priority between the time series data of the priority of handling between the described time series data value of expression, and priority in (3) a plurality of interior time series datas, the latter is used to divide described time series data value, with the processing priority between the expression institute dividing data value; Data processing equipment, according to priority between described time series data is that each described time series data value is distributed throughput, and, reduce the processing quality of institute's dividing data in the described time series data adaptively according to priority in the described time series data, so that each described time series data is remained in the throughput of described distribution.
Data processing equipment of the present invention, it is characterized in that, priority in the time series data of the described video data lifting of every frame video, and when the described video data of every frame is divided into a plurality of packet, only priority in the described time series data is added to the stem of packet, with the stem of the described video data of an addressable frame as message transmission independently.
According to above-mentioned arbitrary data processing equipment, priority is described in the stem of packet in the wherein said time series data, to carry out processed.
According to above-mentioned arbitrary data processing equipment, wherein, can represent that the scope of the value of priority in the described time series data is variable, to carry out processed.
Data processing method of the present invention, it comprises the following step: input data sequence, this data sequence comprises: priority between the time series data of the priority that audio or video time series data and representing is handled between the described time series data value; And utilize between described time series data priority that processed is become relatively or the absolute priority value.
Data processing method of the present invention, it comprises the following step: the audio or video time series data is classified; Input DS, this DS comprise described time series data and represent the interior priority of a plurality of time series datas of processing priority between the described grouped data value; Utilize the interior priority of described time series data that processed is become relatively or the absolute priority value.
And, in order to solve problem (C1), the invention is characterized in:, for example import vision signal as Wave data according to above-mentioned Wave data sending method; Or above-mentioned Wave data method of reseptance, for example export vision signal as Wave data.
And, in order to solve problem (C2), the invention is characterized in:
(d) according to above-mentioned Wave data method of reseptance, the every group time of implementation of output by estimating to obtain; Or
(d) import the serial data that constitutes by every group of time of implementation;
(e), according to each time of implementation of receiving system, carry out time counting number to every group, in the needed time of the determined code length of sign that sends by rate controller etc., to finish decoding according to above-mentioned Wave data sending method.
And, in order to solve problem (C3), the invention is characterized in:
(d) according to the output of needed time of video signal coding and counting device respectively carry out number of times, estimate every group of time of implementation;
(e) according to above-mentioned Wave data sending method, utilize the above-mentioned time of implementation to estimate, and calculating treatmenting time is no more than every group of execution number of times being handled the time of an image by definite can be used for of frame rate of user's appointment to the needed processing time of video signal coding.
The present invention has said structure, behind the execution number of times with the execution number of times that obtains necessary processing and inessential processing, sends to receiver side and to carry out number of times, and estimate that according to carrying out number of times and decode time each handles the needed time.
According to estimated time to each processing, reduce the execution number of times of inessential processing, make the needed time ratio fixed time of decoding short, can control to the fixed time or shorter to decode time like this, to keep delay little.
And, by sending the time of implementation that necessity that receiver side estimates handles and the time of implementation of inessential processing to transmitter side, and determine that according to each time of implementation each carries out the value that number of times can be arranged to the decoding time of implementation to be equal to or less than the fixed time at transmitter side.
And, the time of implementation by estimating necessary processing and the time of implementation of inessential processing, and determine that according to each time of implementation with by the user of the frame rate of user's appointment decision each carries out number of times the fixed time, can be arranged to coding estimated time to be equal to or less than the value of user's fixed time.
A kind of method of reseptance of the present invention has:
Receive the step of the title image relevant with program from the broadcasting channel of the title image of the image of broadcasting by the multifrequency channel radio only;
By switching program by the user, when required program that in memory, has the longer time for reading of needs or data, in case select the user think audiovisual program title image and to user prompt, perhaps point out and currently read, after reading required program or data by memory, get back to the step that the user thinks the program of audiovisual once more.
In method of reseptance of the present invention, described title image can comprise the broadcast image that obtains by the program of periodically sampling by a plurality of channel broadcastings.
Summary of drawings
Fig. 1 is the schematic block diagram of the audio-video R-T unit of one embodiment of the invention;
Fig. 2 is the schematic diagram that receives control section and separating part;
Fig. 3 is the schematic diagram that utilizes the method for the transmission of a plurality of logical transport circuit and control of video and audio frequency;
Fig. 4 is the schematic diagram that dynamically changes the method that is added to the video wanted or the header message in the voice data;
Fig. 5 (a) and 5 (b) are the schematic diagrames that increases the method for AL information;
Fig. 6 (a) is the schematic diagram of example that increases the method for AL information to 6 (d);
Fig. 7 is by dynamically moving multiplexed and the schematic diagram that separates a plurality of logical transport circuit method of transmitting information;
Fig. 8 is the schematic diagram of transmission broadcast program process;
Fig. 9 (a) shows when program or data occurring at receiving terminal, has considered the image or the sound sending method with start-up time of reading of program or data;
Fig. 9 (b) shows when transmission program or data, considered program or data read or start-up time transmission of video or the method for audio frequency;
Figure 10 (a) is the schematic diagram corresponding to fast switching method;
Figure 10 (b) is the schematic diagram corresponding to fast switching method;
Figure 11 (a) shows the object lesson of the agreement of actual transmissions between terminal;
Figure 11 (b) shows the object lesson of the agreement of actual transmissions between terminal;
Figure 12 shows the object lesson of the agreement of actual transmissions between terminal;
Figure 13 (a) shows the object lesson of the agreement of actual transmissions between terminal;
Figure 13 (b) shows the object lesson of the agreement of actual transmissions between terminal;
Figure 13 (c) shows the object lesson of the agreement of actual transmissions between terminal;
Figure 14 shows the object lesson of the agreement of actual transmissions between terminal;
Figure 15 shows the object lesson of the agreement of actual transmissions between terminal;
Figure 16 (a) shows the object lesson of the agreement of actual transmissions between terminal;
Figure 16 (b) shows the object lesson of the agreement of actual transmissions between terminal;
Figure 17 shows the object lesson of the agreement of actual transmissions between terminal;
Figure 18 shows the object lesson of the agreement of actual transmissions between terminal;
Figure 19 (a) shows the object lesson of the agreement of actual transmissions between terminal;
Figure 19 (b) shows the object lesson of the agreement of actual transmissions between terminal;
Figure 20 (a) to 20 (c) be the block diagram of the demonstration system of CGD of the present invention;
The method of additional priority when Figure 21 shows the encoder overload;
When Figure 22 shows overload, in the method for receiving terminal decision priority;
Figure 23 shows Iterim Change priority;
Figure 24 shows data flow priority and object priority;
Figure 25 is the schematic block diagram of the video encoder and the Video Decoder of one embodiment of the invention;
Figure 26 is the schematic block diagram of the audio coder and the audio decoder of one embodiment of the invention;
The priority extention and the priority deciding section of control processed when Figure 27 (a) and 27 (b) show overload;
Figure 28 (a) and 28 (b) show additional priority level;
Figure 29 shows the method for distributing priority to the multiresolution video data;
Figure 30 shows the method that constitutes the communication pay(useful) load;
Figure 31 illustrates the method that produces data corresponding to the communication pay(useful) load;
Figure 32 shows the relation between object priority, data flow priority and the communication data packet priority;
Figure 33 is the block diagram of the transmitter of first embodiment of the invention;
Figure 34 is the schematic diagram of first embodiment;
Figure 35 is the block diagram of the receiver of third embodiment of the invention;
Figure 36 is the block diagram of the receiver of fifth embodiment of the invention;
Figure 37 is the schematic diagram of the 5th embodiment;
Figure 38 is the block diagram of the transmitter of sixth embodiment of the invention;
Figure 39 is the block diagram of the transmitter of eighth embodiment of the invention;
Figure 40 is the flow chart of the sending method of second embodiment of the invention;
Figure 41 is the flow chart of the method for reseptance of fourth embodiment of the invention;
Figure 42 is the flow chart of the sending method of seventh embodiment of the invention;
Figure 43 is the flow chart of the sending method of ninth embodiment of the invention;
Figure 44 is the block diagram of audio-video dispensing device of the present invention;
Figure 45 is the block diagram of audio-video receiving system of the present invention;
Figure 46 shows the key-drawing to the priority attachment device of the video of audio-video dispensing device of the present invention and voice data increase priority;
Figure 47 shows by explanation and is added to the key-drawing that priority that whether video of audio-video receiving system of the present invention and the decision of the priority in the voice data decode is determined device.
Symbol description
11 receive control section
12 separating parts
13 send part
14 video expansions (image spreading part)
15 videos expansion control section (image spreading control section)
16 video composite parts (image synthesizing section)
17 outputs
18 terminal control sections
4011 send control section
4012 video codings parts (image encoding part)
4013 receive control section
4014 video decodes parts (picture decoding part)
4015 video composite parts (image synthesizing section)
4016 outputs
4101 video encoders (image encoder)
4102 Video Decoders (image decoder)
301 receiving systems
302 estimation units
303 Video Decoders (being moving image decoder)
304 number cutting devices
306 outputs
307 inputs
3031 variable-length decoders
3032 anti-quadrature converting means
3033 switchs
3034 motion compensation units
3035 time of implementation measurement mechanisms
Embodiments of the present invention
Embodiments of the invention are described with reference to the accompanying drawings.
The embodiment that describes below mainly solves each in the above-mentioned problem (A1) to (A6).
Used " image " of the present invention comprises rest image and moving image.And, can be to resemble the two dimensional image of computer graphical (CG) or the 3 d image data that constitutes by wire-frame model as the image of object.
Fig. 1 is the schematic block diagram of the audio-video transceiver of one embodiment of the invention.
In Fig. 1, the reception control section 11 of reception information and the transmission part 13 of the information of transmission are the information carrying means such as coaxial cable, CATV, LAN and modulator-demodulator etc.Communication environment can be the environment that can need not consider multiplex machine with many logical transport circuits, and for example the Internet perhaps can be the environment that must consider multiplex machine, for example analog telephone or satellite broadcasting.
And, as the terminal connected system, can be set forth in, or the system of broadcast-type video or voice data is broadcast in broadcasting via satellite, CATV or the Internet such as the system that carries out two-way transmission video or voice data between the terminals such as visual telephone or TeleConference Bridge.The present invention has considered these terminal connected systems.
Separating part 12 shown in Figure 1 is to be used to analyze information that receives and the device that data are separated with control information.Specifically, separating part 12 will send the transmission header message and the data that append in the data of usefulness and decompose, and the Data Control stem and the data content that perhaps append in the data self decompose.Image spreading part 14 is to expand the device of the image that receives.For example, image expanding can be the compressed image of standardized motion (dynamically) or rest image, and for example H.261, H.263, MPEG 1/2 or JPEG etc. also can not be above-mentioned compressed images.
Image spreading control section 15 shown in Figure 1 is the devices that are used for the monitoring picture extended mode.For example,, when reception buffer almost will overflow, can read reception buffer by sky, and expanded images not, and, can restart expanded images being in to carry out the extended mode of image the time by the extended mode of monitoring picture.
In Fig. 1, image synthesizing section 16 is the devices that are used for synthetic expanded images.Image combining method can be by using such as scripts such as JAVA, VRML or MHEG, describe image and structural information thereof and come (display position and demonstration time (and, can also comprise the display cycle)), the relation between the method to image packets, image display layer (degree of depth), object ID (SSRC that describes later on) and their attribute defines.The script of describing synthetic method inputs or outputs by network or local storage.
Output 17 is display or printer, is used for the synthetic result of output image.Terminal control section 18 is the devices that are used to control each part.In addition, can adopt the structure of expanded sound rather than image (can be by image spreading is partly made into the sound expansion, the image spreading control section is made into sound expansion control section, image synthesizing section is made into speech synthesiser assign to constitute this structure), perhaps adopt the both sides of expanded images and sound, in synchronous on the retention time, the structure of synthesizing and showing.
Also can utilize the image compression part of compressed image.Control chart partly sends image and sound as the contract sound compression control of part of the sound compression section of the image compression control section of compression section, compression sound and guide sound sound press.
Fig. 2 shows and receives control section and separating part.
The reception control section 11 of Fig. 1 is made of the control information receiving unit 102 of the control information of Data Receiving part 101 that receives data and reception control data, separating part 12 is explained the transformat storage area 103 of the transfer structure (back detailed description) that transmits content and is explained that according to the transfer structure that is stored in the form storage area 103 the transmission information interpretation part 104 that transmits content constitutes, and can receive data and control information independently by this structure by storage.Therefore, for example, can make in receiver, video or voice data, to delete or move these data to become easy.
As mentioned above, communication environment as the object that receives control section 11 can utilize the environment (internet-class environment) that can use many logical transport circuits and need not consider multiplex machine, as the Internet, also can utilize the environment (former class environment) that to consider multiplex machine, as analog telephone or satellite broadcasting etc.Yet,, be prerequisite with the communication environment that has many logical transport circuits (logical channel) (for example, under the situation of the communication environment that can use TCP/IP, using the statement that is called " communication port " usually) from user's angle.
As shown in Figure 2, suppose to receive control section 11 and receive the data transmission link of one or more types and the control logic transmission line of one or more types, to control the data that to transmit.Also can prepare plurality of transmission lines transmission data, also can only prepare a transmission lines control data.Can also prepare transmission lines control data when each transfer of data, as at the RTP/RTCP that H.323 also uses.When considering to utilize the broadcasting of UDP, can use the communication system of utilizing single communication port (multicast address).
The schematic diagram of Fig. 3 is used to explain the method for utilizing the transmission of many logical transport circuits and control of video and voice data.The data that will send itself are called ES (elementary stream), if image, it can be the image information of a frame or less than the image information of the GOB unit or the macro block unit of a frame.
If voice data, the regular length that can use the user to determine.And, the Data Control header message that append in the data that will send is called AL (adaptation layer information).As AL information, the information, expression data reproduction that can enumerate expression and whether be starting position that can the deal with data information and the information of the priority of expression data processing etc. constantly.Data control information of the present invention is corresponding to AL information.And the used ES of the present invention not necessarily must be consistent with MPEG 1/2 defined content with AL.
Whether expression is that the information of starting position that can deal with data specifically comprises two types information.First kind is the random access mark, that is, if image is to represent can read individually and the independent information of reproducing the front and back data as (I image) in the frame.Second kind is the mark that the expression conduct can be read individually, can be defined as access flag, that is, if image represents that it is the starting position of the image of GOB unit, perhaps be the information of image macro starting position.Therefore, that just represents that it is a part in the way of data if there is not access flag.These two kinds of random access marks and random labelling are not always to need as the information of the starting position of expression energy deal with data.
Sometimes under the situation such as the real time communication of video conference etc., additional these two kinds of marks can not have problems yet.Yet in order to simplify editing, the random access mark is essential.Also can before transmitting data, mark whether perhaps need which kind of mark by needs by the communication port decision.
Designation data is the information of the time synchronized when the information representation reproduced image of now and sound again, and under the situation of MPEG1/2, it is called PTS (representing time tag).Owing under the situation such as real time communication such as video conferences, generally do not consider time synchronized, the information of expression recovery time might not be essential.The time interval between the coded frame can be essential information.
By adjusting the time interval, can prevent that interframe is separated with than great fluctuation process at receiver side.Yet receiver side postpones owing to adjusting to reproduce at interval.Therefore, can judge that sometimes the temporal information of representing frame period between the coded frame is unnecessary.
For the information of determining the expression data reproduction time is expression PTS or expression frame period, also can determine before the transmission data, transmitting the data reproduction time by communication port does not append in the data itself, and, also can transmit data with the data control information that is determined this decision notice receiving terminal.
When because the load of the load of receiving terminal or network former thereby can not handle or send the information of priority of expression deal with data the time can stop to handle or sending data, also can reduce the load of receiving terminal or network.
Receiving terminal can be used image spreading control section 15 deal with data, can handle with link terminal or router under the situation of network.Can represent priority with numerical value or mark.And, when the load of receiving terminal or network has unexpected fluctuation, the deviant of the information of expression data processing priority is sent with data as control information or data control information (AL information), and with this deviant and the priority value addition of allocating in advance to video or voice data, thus, can dynamic priority be set corresponding to the mode of operation of system.
And, by whether identification is had scrambled, whether copyright is arranged and be original paper or duplicate etc. information as control information with and data identifier (SSRC) transmission that separates of data, can easily remove scrambling at via node.
The stream unit that the information of expression data processing priority can be made of the frame set of a plurality of images or sound is additional, also can video or the frame unit of voice data additional.
Provide the priority attachment device for sending terminal unit, this device is for utilizing such as H.263 or G.723 waiting coding method, according to the processing priority of benchmark decision under the coded message overload situations of prior regulation, and make the device (see Figure 46) of coded message corresponding to the priority of determining.
Figure 46 is used for explaining the schematic diagram that priority is added to the priority attachment device 5201 of image or sound.
That is, as shown in figure 46, according to prior predetermined rule, to coding video frequency data (handling) and coding audio data (handling) additional priority by audio coding apparatus 5203 by video coding apparatus 5202.The rale store of additional priority is in priority ancillary rules 5204.These rules comprise the priority higher than the priority of P frame (inter coded images frame) additional to the rule of I frame (in-frame encoding picture frame) or the additional rule to image of the priority lower than the priority of voice data.And, can change these rules according to user's appointment.
The priority extra objects under the situation of image, for the picture frame of scene variation, editor or user's appointment or stream, under the situation of sound, then is ensonified zone and no sound area.
Be defined in the addition method of the priority of the picture frame of priority of processing in when overload or audio frame unit: priority is appended to the method for communication stem and when encoding, priority is embedded into the method in the stem of the bit stream of video or audio data coding, last method can obtain precedence information under the situation of not decoding, then a method can be handled single bit stream independently with the ground of system independence.
Priority is appended to the situation of communication stem, when a picture frame (for example P of the I frame of intraframe coding or interframe encode or B frame) when being divided into a plurality of transmits data packets, under the situation of image, as information independently only at the stem additional priority that transmits addressable picture frame (when priority is the same in the phase diagram picture frame, can think that priority does not change before the addressable picture frame of the next one occurs).
And the scope that enables to represent the value of priority is variable (for example, can according to purposes, with 16 or 32 bit representation temporal informations), also can make it configurable with control information.
Decoder has, the receiver terminal device (seeing Figure 47) of the priority determination device of the priority decision processing method when transshipping according to the various coded messages of receive.
Figure 47 is used for explaining the priority that appends to image or sound and determines the priority that could decode to determine device 5301 schematic diagrames.
That is, as shown in figure 47, priority comprises the priority and the priority that appends in every two field picture or the voice data of each data flow that is added to each image or sound.Can utilize these priority independently of one another, perhaps make frame priority corresponding to data flow priority.Priority determines that device 5301 is according to definite data flow or the frame that will decode of these priority.
Decoding is that two types priority of the priority of the processing when utilizing decision terminal overload is carried out.
That is, defined: be used to stipulate data flow priority (Stream Priority, priority between time series) such as the relative priority level between the bit stream of image and sound; Be used for stipulating (Figure 24) such as the relative priority level between the decoding units such as picture frame of same data stream (Frame Priority, priority in the time series).
Preceding a kind of data flow priority can be handled a plurality of videos or voice data.A kind of frame priority in back can be according to the variation of the scene of image or editor's intention, even same in-frame encoding picture frame (I frame) also can add different priority.
By making data flow priority, and manage corresponding to the time of distributing to the operating system (OS) when image or sound encoded or decode or the priority of handling, can be on the 0S layer management processing time.For example, under the situation of the Windows of Microsoft 95/NT, can on five 0S layers, define priority.When realizing coding or decoding function with thread unit with software, can be in the data flow priority decision priority of distributing on the OS layer of each thread according to the data flow of process object.
Above-mentioned frame priority and data flow priority can be applied on transmission medium or the data medium.For example, is the packet priority definition that will send addressed location priority, can be according to the frame priority of Access UnitPriority=Stream Priority-Frame Priorit and the relation between the data flow priority, decision sends relevant priority with packet and the processing priority of terminal when transshipping.
And, can be used as data medium to floppy disk or CD.Moreover recording medium is not limited only to these, but also can be such as IC-card or ROM box etc., as long as the object of energy logging program can be implemented equally.And, can be object with the relay that carries out the data relaying such as images such as router or gateway or sound.
As the concrete grammar that utilizes priority, when receiving terminal transships, the priority of the priority threshold value of decision coded message to be processed is determined that device is arranged at image spreading control section or sound expansion control section, elapsed time compares with beginning to handle afterwards the time that will show (PTS), perhaps the time that will decode (DTS) with begin to handle after through time ratio, change the priority threshold value (insertion that also can reference frame at interval or priority etc. as the information that changes threshold value) of coded message to be processed according to comparative result.
In the example shown in Figure 20 (a), when coding, encode to having the QCIF that catches or the image of CIF size with encoder (H.263), precedence information (CGD, calculating moderate reduction), frame type and the sequence number (SN) of the processing sequence when exporting time tag (PTS), the expression overload of the time of expression decode time (DTS) or display image with coded message.
And, in the example shown in Figure 20 (b), also record, encode, reproduce time tag (PTS), precedence information (CGD) and the sequence number (SN) of the time of sound with coded message output decoder time (DTS) or expression by encoder (G.721) by microphone.
Shown in Figure 20 (c), in when decoding, image and sound are offered buffer separately respectively, their DTS (decode time) separately with begin to handle the back elapsed time and compare.If DTS does not lag behind, then this image harmony cent is supplied with their corresponding decoders (H.263 and G.721) indescribably.
The example of Figure 21 described a kind of when encoder transships the method for additional priority.For image, " 0 " of high priority and " 1 " is distributed to I frame (in-frame encoding picture frame) (numeral is more little, and priority is low more).The priority of P frame " 2 " is lower than the I frame.Owing to give the I frame two kinds of other priority assignment of level, so when the load of decoding terminal is big, can only reproduce the I frame that priority is " 0 ".And, must be according to the insertion interval of priority addition method adjustment I frame.
The example of Figure 22 shows when overload, determines the method for priority at receiving terminal.The priority of the frame that will discard is arranged to the value bigger than cutOffPriority.That is, all picture frames all as the object of handling.By being notified to receiver side (step 101) from transmitter side, the priority maximum that appends to picture frame can learn this value in advance.
When DTS with begin to handle after elapsed time compare, and consequently elapsed time is longer than DTS (when decoding processing has little time), then reduces the priority threshold value of image to be processed or sound, to reduce processing (step 102).Yet,, increase the threshold value of priority, to increase the quantity (step 103) of manageable image or sound if elapsed time is shorter than DTS (when decoding processing is able to do in time) after beginning to handle.
If the image before this frame has been skipped the P frame, then do not handle.Otherwise the priority deviant is appended on the priority of picture frame (or audio data frame), this priority deviant and priority threshold value are compared.When this deviant is no more than this threshold value, the data that decode are offered decoder (step 104).
The priority skew can be used for the performance of precheck machine, and this deviant sent to receiving terminal (user also can indicate at receiving terminal), also can be used for changing a plurality of videos in the data flow and the priority (for example, the deviant of background reduces processing by increasing backmost) of audio data stream.
When being object with the multiple data stream, also can be to each data flow additional priority, and judge the decoding of skipping to image or sound.And, under the situation of real time communication,, can judge that decoding is in advance or lags behind, and realize as described above skipping by handling TR (interim benchmark) and the same processing of DTS H.263.
Figure 23 shows the variation of the time of utilizing the priority that above-mentioned algorithm carries out.
Figure 23 shows the variation that will append to the priority in the picture frame.This priority is the priority that decision could be decoded when terminal is transshipped, and it is additional in each frame.Priority value is more little, and priority is high more.In the example of Figure 23,0 priority is the highest.When the threshold value of priority is 3, has added value and do not decode greater than the frame of 3 priority and discard, to have added value be 3 or decode less than the frame of 3 priority.According to discarded selectively these frames of priority, load that can control terminal.Also can and append to relation between the decode time (DTS) of each frame, dynamically determine priority threshold value according to the current processing time.According to same process, this technology not only can be applied to picture frame, also can be applied to sound.
Consider that at transmission line be under the situation of the Internet etc., the coded message of losing in transmitting the way need be retransmitted and show, receiving the re-send request may priority determining section of threshold value that control section is provided with the priority of the coded message that decision will retransmit, precedence information according to re-send request may priority determination portion, retransmit number of times, the information dropout rate, the insertion of intracoded frame at interval, be subjected to the rank (for example Pyatyi is preferential) of the priority of priority determining section control, to receiving the priority threshold value of the definite coded message that will retransmit of control section, and the additional priority threshold value of giving the coded message that needs request repeat of decision, like this, in the time must retransmitting the coded message of losing when transmitting, can only retransmit desired image of receiver side or voice data.If the repeating transmission number of times is a lot of or the information dropout rate is very high, then be necessary to improve the priority of wanting retransmission of information, reduce and retransmit number of times or reduce Loss Rate.And, by understanding the priority that the priority determining section is used, can prevent to send the outer information of process object.
Sending terminal, when actual transfer rate surpasses the target transmission rate of information of transmitting terminal, perhaps when beginning to transmit the time ratio handling the back elapsed time and append to the coded message that to decode or to show, when coded message is written to the process lag of transmit buffer, available append in the coded message, the priority when the overload that the priority determination section branch of receiving terminal utilizes, by reducing the transmission of information, make it as one man send image or voice data with targeted rate.Also can be incorporated into the transmission terminal in the transmission terminal, and can control and send the fault that the terminal overload causes by will when receiving the overload of being carried out, handling skip functions.
By only sending information necessary in the above-mentioned AL information as required, can adjust and to send to such as the amount of information on the narrow band communication passages such as analog of telephone line.Decision will append to the data control information in the data self on sending terminal before sending data, the data control information that will use is sent to receiving terminal as control information (for example only using the random access mark), and according to the control information that obtains at receiving terminal, by rewriteeing and being stored in the relevant information of transmission structure (seeing Figure 16) that sends in the form storage area 103, can be binned in the employed AL information of terminal (data control information) that sends.
Fig. 4 is a schematic diagram, is used for explaining that dynamic change will append to the method for the header message of video to be sent or voice data.In the example of Fig. 4, the data that will send (ES) resolve into data slice and with the form of communication stem the identifying information (sequence number) of these data slice additional representation data ordinal relations, expression can the deal with data sheet the information (marker bit) and the temporal information (time mark) relevant of starting position with the transmission of data slice, suppose that wherein above-mentioned each information is corresponding to control information transmission of the present invention.
Specifically, (real time transport protocol RFC1889) utilizes information, marker bit, time tag, object ID (being called SSRC) and the version number of above-mentioned sequence number as the stem of communicating by letter to RTP.Though can expand the header message item, above-mentioned every always additional as fixterm.Yet, to transmit a plurality of different coded images or the following time of environment of sound simultaneously when appearing at together such as real time communication such as visual telephone and transmission such as accumulation medium such as VODs, because the implication of communication stem differs from one another, so means of identification must be arranged.
For example, under MPEG 1/2 situation, as mentioned above, time tag information represents that PTS is the recovery time.Yet, H.261 or H.263, the blanking time when time tag information is represented information encoded.But, in the time will making H.263, must be labeled as PTS information by express time with the synchronous processing of voice data.This is that in the time interval between the RTP definition time label information presentation code frame, the time mark of first frame is at random because under situation H.263.
Therefore, the mark that must add (a) express time sign and whether be PTS as communication header message (must expand the communication stem) or (b) H.263 or the header message of Payload H.261 (promptly, AL information) (in this case, must expand Payload information).
The marker bit of information that as expression whether is starting position that can the deal with data sheet is additional as the RTP header message.And, as AL information, as mentioned above, often wish to make its have access flag that expression is starting position that can visit data and expression can the random access data to obtain the random access mark of AL information.Owing to making its dual mark reduce efficient, so also can consider to substitute the method for AL mark by the mark that the communication stem is prepared with communication stem.
(c) newly establishing the not additional AL mark of expression in the communication stem will substitute at the AL mark that the communication stem is added, or the marker bit of definition communication stem is identical with AL, (compare with the situation that the AL mark is provided, intended interpretation can carry out quickly), thus, address the above problem.That is, adopt a kind of mark, whether the expressive notation position has the implication identical with the AL mark.In this case, think and to be the improvement of communication stem or in the scope that expands, to describe stem.
Yet, (d) also can be with the implication of marker bit of communication stem, interpret table is shown and has one of random access mark and access flag at least in AL.In this case, know its implication, can come corresponding by the version number of communication stem with original different.And needing only by access flag or the processing of random access mark are set in communicate by letter stem or AL stem is very simple (for the former, consider that two stems all are provided with the situation of mark, but must expand the communication stem again).
The information of additional representation data processing priority has been described, as AL information.By data processing priority is appended to the communication stem, can make the judgement of the processing of data processing priority, obtain even on network, also can not carry out the explanation of data content handling.And, under the situation of IPv6, can on the layer lower, add this priority than RTP level.
By timer or counter, can judge how the state of the packet that is transmitted changes to valid period of RTP communication stem additional representation data processing.For example, when the decoding software of needs is stored in the memory of low access rate, can judges the needed information of decoder and when need information by counter or timer.In this case, according to purposes, for AL information, the precedence information of timer or counter or the precedence information of data processing are optional.
Fig. 5 (a) and 5 (b) and Fig. 6 (a) are the schematic diagram of explaining the method for additional AL information to 6 (d).
By sending control information, can select to handle the grade of transmission information to receiving terminal.Described control information, still shown in Fig. 5 (b), adds AL in each data slice after the data (ES) that will send are resolved into more than one data slice the stem that AL only is added to the data that will send for being shown in Fig. 5 (a).When visit lags behind when being a problem, be effective by AL being appended to segmentation cuts in the data.
As mentioned above, for in advance sending receiving terminal in the recombination of receiver side data control information or the variation that data control information is configured to the method in the data, receiving terminal can utilize the expression of mark, counter or timer, statement is prepared to AL information or communication stem notice receiving terminal, makes receiving terminal can successfully carry out correspondence.
In above-mentioned example, described and avoided with the method for AL information reproduction RTP stem (or communication stem) and the method for expansion RTP or AL information communication stem.Yet, for the present invention, always must not use RTP.For example, also can utilize UDP or TCP to redefine independently communicate by letter stem or AL information.Though the internet-class environment has utilized RTP sometimes, in former class environment, do not define such as multi-functional stems such as RTP.About thinking methods following four types (referring to Fig. 6 (a) to 6 (d)) are arranged to the AL information and the stem of communicating by letter.
(1) header message of RTP or AL information is proofreaied and correct and expand, so that distributed to the header message of RTP and distributed to the header message of AL not overlapping (particularly, the information overlap of time mark, the precedence information of timer, counter or data processing becomes extend information).Perhaps, do not expand RTP stem and AL information and RTP repeats also irrespective method even can utilize.They are corresponding to the content that has so far illustrated.Because the part of RTP, in fact in being used H.323, to have compatible RTP be effective so expand.(referring to Fig. 6 (a).)
(2) be not limited to RTP, simplify communication stem (for example, only using sequence number etc.),, offer AL information as multi-functional control information remaining content.And, by making the project that AL information is used is set changeably before communication, can specify the transmission form neatly like this.(referring to Fig. 6 (b).)
(3), and control information is all offered the communication stem no matter RTP how, simplifies AL information (extreme example is to AL additional information not).Sequence number, time mark, marker bit, payload types and the object ID of the communication stem that can often be referenced continually are as fix information, and will be to data processing precedence information and timer information as extend information, the identifier whether the expression extend information exists is set, as long as extend information is defined, then with reference to this information (referring to Fig. 6 (c).)
(4) irrelevant with RTP, simplify communication stem and AL information, communication stem or AL information as the also transmission of its form of the packet definitions of separating.For example, to AL information only defined label position, time mark and object ID, to communication stem also defined nucleotide sequence number only, as the packet different with these information (second packet), definition pay(useful) load information, data processing precedence information and timer information etc. also send (referring to Fig. 6 (d).)
As mentioned above, when considering purposes and appended to header message in image or the voice data, preferably according to this purposes, can freely define (customization) is to separate the packet (second packet) of transmission with the stem of communicating by letter, AL information and data.
The schematic diagram of Fig. 7 be used for explaining by dynamic multipath multiplexing with separate many logical transport circuits and come method of transmitting information.It is by assigning to realize at hop configuration information multiplexing section with in reception control section configuration information separated part.Described information multiplexing section is for reducing the quantity of logical transport circuit, according to user's the appointment or the quantity of used logical transport circuit, can make the multiplexed beginning of the logical transport line information that transmits a plurality of data or control information or the information multiplexing section of end, this separating part is from the multiplexed information that is received.
In Fig. 7, the information multiplexing section is called " Group MUX ", and specifically, it can use the multiplex system such as H.223.Can be arranged at transmission terminal Group MUX.By it being arranged at relay router or terminal, can carry out the correspondence of narrow band communication passage.And, if Group MUX in H.223, realize, then can make H.223 and H.324 the interconnection.
For the control information (multiplexed control information) that obtains the information multiplexing section apace, by the control information in another logical transport circuit transmission information multiplexing section, and the control information in the information multiplexing section not with the information multiplexing section in data carry out multiplexedly, can reduce the delay that causes because of multiplexed.Thereby, by notice is to transmit behind control information relevant with the information multiplexing section and the data multiplex, still control information is not carried out multiplexed with data, do not carry out multiplexed, but transmit by another logical transport circuit, and transmit, it is the consistency that keeps multiplexed with tradition that the user can select, and still reduces because the multiplexed delay that causes.At this, be for example to represent that the information multiplexing section carries out the information of the content which type of is multiplexed, expression is multiplexed to every data about the multiplexed control information of information multiplexing section.
Equally, as mentioned above, according to expression method, for example according to mark, counter or timer, can transmit the notice of sending method, the information of the combination that this method sends the information of passing on multiplexed beginning and end at least, pass on the multiplexing logic transmission line with the transfer approach of multiplexed relevant control information (multiplexed control information), transmit as control information, perhaps send to receiving terminal with data, thereby reduce the time that is provided with at receiver side as data control information.And, as mentioned above, the project of expressive notation, counter or timer can be set in transmission RTP stem.
When having a plurality of information multiplexing sections or a plurality of information separated part, only need transmit with identifier and be used for identifying information multiplexing section or information separated part control information (multiplexed control information), just can identification control information belong to which information multiplexing section.Control information (multiplexed control information) comprises multiplexing modes etc.And the identifier for information multiplexing section or information separated part can determine between terminal by utilizing the table of random numbers, to produce the identifier of information multiplexing section.For example, can send and receiving terminal between produce random number in the scope that stipulate, and to adopt its maximum be the identifier (identifier) of information multiplexing section.
Because the multiplexed data of information multiplexing section are former different with the multimedia type that defines among the RTP, so need be illustrated in the multiplexed information of information multiplexing section (defining new media type H.223) to RTP pay(useful) load type definition.
As the method that improves the access speed of multiplexed data, by the information multiplexing section is transmitted or the information of record with the arranged in order of control information and data message, seek to analyze apace multiplexed information.And, will fix according to the project that the data control information that appends in the control information is described, and add and the multiplexed identifier different (unique pattern) with data, can rapid analysis go out header message.
The schematic diagram of Fig. 8 is used to explain the transmission course of broadcast program.By the information of the corresponding relation between the identifier of the identifier of logical transport circuit and broadcast program as broadcast program is transmitted, perhaps send control information or make increase that the identifier of broadcast program is appended to data as data control information (AL information) and transmit, can pick out in the data of plurality of transmission lines transmission and broadcast for which programming.And, with the relation between the identifier (for example LAN port numbers) of data identifier (being SSRC under the situation of RTP) and logical transport circuit, transmit to receiving terminal as control information, and confirmed receiving terminal can receiving control information after (Ack/Reject), by sending corresponding data, even control information and data all respectively by independently transmission line transmission, also can form the corresponding relation between the data.
With expression to the identifier of the transmission sequence of broadcast program or data with have the expression broadcast program or data can be as the counter of valid expiration date of information or the information combination of timer.And, can realize not having the broadcasting (when effective time limit almost will lose efficacy,, also beginning to reproduce the information or the data of broadcast program) of Return Channel even information is not enough by it being appended to broadcast program or data are sent.And, can consider a kind of like this method, promptly use single communication port (multicast address), do not separate the method that control information and data are broadcasted.
In the communication that does not have Return Channel, must before sending data, send the control information more abundant fully, so that receiving terminal can be learnt data structure than data.And control information generally should send by not producing the high transmission channel of data-bag lost, reliability.Yet, when utilizing the low transmission channel of reliability, must periodically send control information with identical traffic sequence number.Yet this is not limited to send the situation of the control information relevant with the time that is provided with.
And, by selecting to can be used as the additional project of data message (for example access flag, random access mark, data reproduction time (PTS), or the data processing precedence information etc.), before sending data and at transmitter side, determine it is to transmit by transmission line Different Logic transmission line with data as the identifier (SSRC) of control information with data, still data control information is transmitted as data control information (AL information), as control information notice and transmission, can control and transmit data at receiver side neatly.
Thereby, can be not to the AL additional information and data information.Therefore, when using RTP transmitted image or voice data, just needn't enlarge definition to former defined pay(useful) load.
Fig. 9 (a) and 9 (b) are for having considered the schematic diagram with the sending method of the image of start-up time or sound of reading of program or data.Specifically, when as there not being the unidirectional as satellite broadcasting or portable terminal device of Return Channel, terminal resource is for having in limited time, or program or data are present in the receiver side terminal and use, and required program (for example H.263, MPEG 1/2, or audio decoder software etc.) or data (for example video data or voice data) be stored in need longer time for reading memory (DVD for example, file server on hard disk or the network) time, can be according to such as being the recognizer of prior needs or the identifier of data, the identifier (for example SSRC or logical channel number (logical Channel Number)) of the data flow that transmits or estimation are at the needed specific time point mark of receiving terminal, expression methods such as counter (lifting/lowering counting) or timer, by receiving as control information, or receive with data as data control information, reduce the time that is provided with (Figure 18) of required program or data.
When router or data, by from transmitter side with the information on the memory ground (for example hard disk or memory) of the receiving terminal of representation program or data, begin or read the type of required time, terminal or storage ground with beginning or read the corresponding relation (for example relation between cpu power, memory device and the average response time) between the required time and send with the program or the data of the order of expression utilization, like this when in fact receiving terminal needs program or data, then can the plan program or the storage ground and the time for reading of data.
The schematic diagram of Figure 10 (a) and 10 (b) is used for explaining the method corresponding to switching (switching TV channel) fast.
The situation that only receives image with traditional satellite broadcasting is different, when must be at the receiving terminal executive program, the fetch program and to time that is provided with of startup be a big problem.Also there is same problem in this situation for the portable terminal device of available resource-constrained.
As a solution, can reduce the time that is provided with in the following manner: (a) possess the periodically auxilliary audiovisual part of audiovisual of the main audiovisual part utilizing the user to watch and listen to and receiving terminal in the receiver side terminal, described auxilliary audiovisual is partly for being used for the program beyond the program of audiovisual just the user, and required program and data are present in reading of data and need in the storage device of time, and receive the user just the receiving terminal of the program beyond the program of audiovisual carry out periodically audiovisual, and to the recognizer of prior needs or the identifier of data, the mark of the special time that is used to estimate that receiving terminal is required, the information of counter or timer and receive as control information (sending to the information that control terminal is handled) by the packet different with the packet of data with corresponding relation between the program, or receive with data, and prepare fetch program or data in advance as data control information (AL information).
As second kind of solution, only broadcasting is set by the broadcasting channel of the title image of the image of multifrequency channel radio, and switches program by the user, can prevent that by the following method screen from stopping to show when being provided with.Promptly, when in the memory of the longer time for reading of needs, having required program or data, in case select the user to think the title image of the program of audiovisual, and it to user prompt, perhaps point out and currently read, after meanwhile reading required program or data, get back to the program that the user thinks audiovisual once more by memory.Title image described herein comprises the broadcast image that the program by multichannel broadcasting is carried out the periodic sample acquisition.
And timer body is between now, and the required program of the data flow that sends from transmitter side of for example indicating to decode is from now on to when needing.Counter be send and receiving terminal between the basic time unit that determines, it is so long as represent the information of which time and get final product.Mark and the data that sent before time necessary is set or control information (information that the different packet of packet by the data handled with control terminal sends) send and notify.Can be embedded into timer, counter in the data and send, also can send them as control information.
And, as determining to be provided with time method, for example when utilizing when being operated in the transmission lines such as ISDN on the base, the transmission sequence number of identification transfer sequence is used as control information, need the moment of program or data with notice to receiving terminal from sending terminal, be notified to receiving terminal as data control information with data or as control information, thus, can predict the time that is provided with.And, when the transmission time owing to as the Internet beat or when postponing to fluctuate, then consider the propagation delay of transmission, only need the means that realized at RTCP (media transmission protocol that the Internet uses) according to, by beating or time delay is added to the transmission time in the time of setting.
Figure 11 (a) shows the object lesson of the actual agreement that transmits between the terminal to 19 (b).
Transmission form and transmission course have been described in ASN.1.And, send the H.245 expansion of form according to ITU.Shown in Figure 11 (a), the object of image and sound can have hierarchy.Under the situation of this example, each Target id has the attribute of broadcast program identifier (Program ID) and object ID (S SRC) and the structural information between the image, synthetic method and uses such as scripts such as Java or VRML and describe.
Figure 11 (a) shows the example that concerns between the object.
In Figure 11 (a), to liking such as medium such as sound, image, CG and texts.In the example of Figure 11 (a), object constitutes hierarchy.Each object has program number (" Program ID " is equivalent to the TV channel), is used for " Object ID " object identifier of identifying object.When transmitting each object, making object identifier by making itself and the corresponding identifying object easily of SSRC (synchronous source identifier) according to RTP (media transmission protocol of the transmission medium that the Internet uses, RTP).And, can utilize such as the structure between the descriptive language description objects such as JAVA or VRML.
The method of connection object can be by two types.A kind of is broadcast-type, and object is unilaterally sent from the transmitter side terminal.Another kind is the type (communication type) of transmission object between transmission and receiving terminal (terminal A and B).
For example, under the situation of the Internet, can use RTP as transmission method.Under the situation of visual telephone standard, utilize the transmission channel that is called LCNO, control information transmission.In the example of Figure 11 (a), transmit with a plurality of transmission channels.(Program ID) distributes to these passages identical program passage.
The schematic diagram of Figure 11 (b) is used to explain how to realize this agreement that has realized the function that the present invention describes.The used host-host protocol (H.245) of visual telephone standard (H.324 and H.323) is described below.H.245, the function that the present invention describes realizes by expansion.
Describing method shown in the example among Figure 11 (b) is the protocol description method that is called ASN.1." the performance of terminal capabilities collection (" Terminal Capabilityset ") statement terminal.In the example of Figure 11 (b), the function that is described as " mpeg 4 Capability " is to tradition expansion H.245.
In Figure 12, " mpeg 4 performances " (" MPEG4 Capability ") described and can also describe the maximum multiplexed function number (" Max Number of Mux ") that terminal can realize simultaneously in the maximum image data (" Max Namber of Video ") and the maximum acoustic number of signals (" Max Numbeof Sounds ") of terminal processes.
In Figure 12, be expressed as manageable largest object amount (" Number of ProcessObject ").And, the mark of representing whether can change communication stem (being expressed as AL in Figure 12) has been described.When the value of mark is a true time, can change the communication stem.For utilize " MPEG 4 confirming performances (" MEG 4Capbility ACK ") pass on manageable number of objects between terminal each other; can accept (processing) these objects if be communicated side; be communicated side " MPEG 4 Capbility ACK " are sent back to the terminal of transmission " MPEG 4Capbility "; if can not, then to this terminal loopback " MEPG 4 performances refusal " (" MPEG4Capbility Reject ").
Figure 13 (a) shows how to describe and utilizes above-mentioned Group MUX that a plurality of logics are believed that passages are multiplexed into the agreement of a transmission channel (being the LAN transmission channel in this example), so that logical channel common transmitted passage.In the example of Figure 13 (a), make multiplex machine (Group MUX) corresponding to LAN (local area network (LAN)) transmission channel (LAN port numbers, " LAN Poxt Number ")." Group Mux ID " (" Group Max ID ") is the identifier that is used to differentiate multiplex machine.For " establishment group Max " (" Create group Mux "), make the shared multiplex machine of terminal, and between terminal, transmit mutually, can accept (use) this multiplex machine if be transmitted side, just to the terminal loopback " Greate Group Mux Ack " that sends " Greate Group Mux ", if can not accept, just to this terminal loopback " Greate Group Mux Reject ".Can realize separator with identical method, as the means of carrying out the operation opposite with multiplexed means.
At Figure 13 (b), the situation (" Destory GroupMux ": " destruction group Mux ") of deleting the multiplexed means that produced has been described.
In Figure 13 (c), the relation between LAN transmission channel and a plurality of logical channel has been described.
According to the transmission channel of " LAN Port Numbef " description LAN, (" Logical Port Number ") described logical channel according to " logical port number ".
Under the situation of Figure 13 (c), can make corresponding nearly 15 logical channels of transmission channel of a LAN.
In Figure 13, when the quantity of operable MUX only was one, Group Mux ID did not just need.And, in order to use a plurality of Mux, Group Mux ID all must be arranged for H.223 each order.Can usage flag, to concern between the port that uses between notice multiplex machine and the separator.Can also utility command it can be selected to be control information is carried out multiplexed, still by another logical transport line transmission information.
Figure 13 (a) is to the explanation example shown in Figure 13 (c), and transmission channel is utilized LAN.Yet, also can use utilization for example H.223 or the system of the Internet protocol of MPEG 2 etc.
In Figure 14, the protocol description of " opening logical channel " (" Open Logical Channel ") display definition transmission channel attribute.In the example of Figure 14, " MPEG 4 logical channel parameters " (" MPEG4 Logical Channel Parameters ") is according to H.245 protocol extension and definition.
Figure 15 shows the transmission channel for LAN, expression program number (being equivalent to the TV channel), corresponding with the title of program (" MPEG 4 Logical Cannel Parameters ").
And " broadcast channel program " (" Broadcesting Channel Program ") expression sends the describing method of corresponding relation between LAN transmission channel and the program number according to broadcast-type in Figure 15.The example of Figure 15 makes to send and reaches corresponding relation between 1023 transmission channels and the program number.Because under the situation of broadcasting, transmitter side unilaterally sends to receiver side, so consider the phenomenon of losing during the transmission, is necessary periodically to send these information.
In Figure 16 (a), the attribute (" definition of MPEG4 object type ") (" MPEG4 Object Calssdefinition ") of the object (for example image or sound) as program transmission has been described.Object information (" object structure unit ") (" Object Structure Element ") corresponding to program identifier (" ProgramID ").Can make and reach 1023 objects corresponding to program identifier.For object information, mark (" scrambling mark ") that whether LAN transmission channel (" LAN port Number "), expression use scrambling has been described (ScrambleFlag), definition changes the field (" CGD skew ") (" CGDOffset ") of deviant of processing priority and the identifier (" medium type ") (" Mediatype ") of discerning the medium type (image or sound) that will transmit during the terminal overload.
In the example shown in Figure 16 (b), increased AL (in this example, being defined as the required additional information of a frame decoding image) with the decoding of control to ES (in this example, be defined as corresponding to the image of a frame serial data).For AL information, be defined as follows.
(1) random access mark (mark whether expression can independently reproduce, the picture frame of true respresentation intraframe coding)
(2) represent time tag " Presentation Time Stamp " (time that frame shows)
(3) CGD Priority (value of the priority of the processing priority of decision when terminal is transshipped)
This example shows the situation of utilizing RTP (by the agreement of the continuous medium of the Internet transmission, i.e. real time transport protocol) to transmit the serial data of a frame." AL Reconfiguration " changes the peaked transmission statement that can represent with above-mentioned AL.
The example of Figure 16 (b) makes and can be shown as " random access mark maximum number of digits " (" Random Access Flag Max Bit ") reaching 2 bit tables.For example, when " 0 ", do not use random access mark (" Random Access Flag ").In the time of two, maximum equals 3.
And, can represent (for example 3 ∧ 6) with real part and mantissa part.When data not being set, also can under default definite state, work.
In Figure 17, " request is set " (" Setup Request ") expression transmission is provided with the transmission statement of time." Setup Request " sent before the transmission program, make the transmission channel number (" LogicalChannel Number ") that to transmit, the program ID that will carry out (" executive program number ") (" excutive the ProgramNumber ") data ID (" data Number ") that will use and order ID that will carry out (" fill order number ") (" excutive Command Number ") become to correspond to each other, and be transferred to receiving terminal.And, as other expression method, can adopt the timer value (" timer ") that begins to carry out after carrying out authorization mark (" flag "), the counter (" counter ") that begins to carry out when being described in reception " SetupRequest " how many times and being illustrated in, and make them corresponding to transmission channel number through how long.
As the example of the request that will propose, can enumerate the start-up time that rewrites AL information and guarantee Group Mux.
The schematic diagram of Figure 18 is used for explaining the transmission statement, and the AL that whether utilizes Figure 16 (b) to describe transmits AL (" control AL definition ") (" Control AL Definstion ") from sending terminal to receiving terminal.
In Figure 18,, then use the random access mark if " use of random access mark " (" Random Access Flag Use ") is true.Otherwise just do not use.Can transmit as control information the AL change notification by the passage that separates with data transmission channel, or, transmit with data by the transmission channel identical with data.
As the program of carrying out usefulness, can list decoding program.Can be used for broadcasting and communicate by letter request is set.According to above-mentioned request, which of AL information receiving terminal indication as control information, used as.And, can similarly which be used to receiving terminal indication do communication stem, which is used as AL information, which is used as control information.
Figure 19 (a) shows the example of transmission statement, it uses information frame identifier (" stem ID ") (" headerID "), between transmission and receiving terminal, change the structure (data control information, control information transmission and control information) of header message according to purposes.
In Figure 19 (a), " classification ES stem " (" class ES_header ") be according to the information frame identifier, send and receiving terminal between distinguish the data control information by the channel transfer identical with data transmission channel structure or with the structure of control information transmission institute information transmitted.
For example,, only use " ES of buffer scale " (" bufferSize ES "), and work as " header ID " when value is 1, increase (" reserved ") also use as " header ID " when value is 0.
And, by using default designation symbol (" use header extension ") (" use HeaderExtension "), judge the information frame that whether uses default form.When " use Header Extension " is true time, use the item in the if statement.Suppose between transmission and receiving terminal, to have determined these structural informations in advance.And, can use any one party of information frame identifier and default designation symbol.
In the example of Figure 19 (b), " AL configuration " (" AL Configuration ") represents such example, between transmission and receiving terminal, changes the structure of the control information that sends by the transmission channel different with the data of transmission according to purposes.The information frame identifier is identical with the situation of Figure 19 (a) with the using method of default designation symbol.
Under situation of the present invention, specifically describe realization from following viewpoint and synthesize and show a plurality of moving images and a plurality of sound method simultaneously.
(1) by many logical transport circuit images and sound (communicate by letter and broadcast) and control their method.Especially, described by making the method for the independent and transmission of the logical transport circuit that control information and data are transmitted respectively.
(2) dynamically change the method that appends to the header message (AL information) in the data of wanting image transmitted or sound.
(3) dynamically change into transmission and the method for additional communication header message.
Specifically, for (2) and (3), described comprehensively in the information of the AL information and the stem repetition of communicate by letter and the method for carrying out control method and AL information being transmitted as control information.
(4) to many logical transport circuits carry out dynamically multiplexed with separate and method of transmitting information.
Describe the method for the port number of saving transmission line and realized effectively multiplexed method.
(5) the reading of program or data, the image of start-up time and the transmission method of sound have been considered.And, described from the aspect of various functions and purposes and shortened the lip-deep time method that is provided with.
(6) at the quick image that switches of channel or the transmission method of sound.
The present invention is not limited in synthetic two dimensional image.Also can use the expression method of two dimensional image and 3-D view combination, perhaps can comprise the image combining method that synthesizes multiple image adjacent to each other as wide field-of-view image (panoramic picture).
And, be not limited in such as wired two-way catv and B-ISDN as the communication form of object of the present invention.For example, can use from the radio wave (for example VHF frequency band or UHF frequency band) of central terminal, perhaps satellite broadcasting and the analog phone line from home terminal to central terminal or the N-ISDN (image, sound or data are also not necessarily by multiplexed) that transmit information to home terminal images and sound.
And, can use radio communications system, such as IrDA, PHS (personal hand-held phone), the perhaps communication form of WLAN.And the object terminal can be the portable terminal of for example portable data assistance or desktop terminals such as box or personal computer for example are set.And, as application, can also comprise visual telephone, multiple spot surveillance, multimedia system, multimedia database retrieval system and game machine.The present invention not only comprises receiving terminal, but also comprises server and the transponder that is connected to receiving terminal.
In above-mentioned example, the communicate by letter method of stem or AL information of method that RTP (communication) stem and AL information repeats and expansion RTP of avoiding has been described.Yet the present invention might not use RTP.For example, also can utilize UDP or TCP to redefine independently communicate by letter stem or AL information.Though the internet-class environment uses RTP sometimes, for former class environment and undefined such as multi-functional stems such as RTP.As mentioned above, the notion about the AL information and the stem of communicating by letter has four kinds.
Therefore, by dynamically determine send and the frame of the various information of information, control information transmission or the control information of the data control information of receiving terminal use (for example: comprise at first the random access mark is distributed as 1 label information, thereafter information sequence as 16 label informations orders that distribute, additional information and follow the frame of the information of figure place), can carry out change, also can change according to purposes or transmission line corresponding to the information frame of situation.
The frame of each information can be as Fig. 6 (a) to 6 (d) shown any, under the situation of RTP, data control information (AL) can be every kind of medium header message (for example, under situation H.263, can be the H.263 intrinsic video header message or the header message of pay(useful) load), control information transmission can be the RTP header message, and control information can be the information of control such as RTP such as RTCP.
And, between transmission and receiving terminal, set in advance under the situation of known information frame, by representing that the default identifier that whether sends or receive is arranged at data control information, control information transmission and control information (by the information data packet transmission different with data, that control terminal is handled), can understand information frame and whether change.By only when having carried out change, the default designation symbol is set and according to method shown in Figure 16, notice has changed content (for example time mark information changes over 16 from 32), the unnecessary configuration information of transmission when this can prevent that the frame information of information from not changing.
For example, the information frame that change data control information can have two kinds of following methods.At first, describing when data itself are carried out the change method of information frame of data control information, default designation symbol (will be written to fixed area or position), the changing content of descriptor frame then of the information that exists in the data that the information frame of data control information describes are set.
As another kind of method is when describe becoming control information (information frame control information) into the method for the information frame of data, when changing the information frame in the data control information, the default designation that control information provides is set, the content of the information frame of the data control information that description will change, and according to ACK/Reject after receiving terminal notice and the information frame of confirming data control information have changed, the data that the transmission information frame has changed.When changing the information frame of control information transmission and control information itself, also can realize (Figure 19) by above-mentioned two kinds of methods.
More particularly, though the header message of MPEG2 is fixed, the program map table that is associated with audio data stream by the video data stream (transmission data flow) to MPEG2-TS is provided with default designation symbol (by the PSI definition), and the configuration data stream of change method of the frame of video data stream and voice data stream information has been described in definition, as long as the default designation symbol can at first be explained configuration data stream once being provided with, then according to the content interpret video of configuration data stream and the stem of audio data stream.Configuration data stream can be a content shown in Figure 19.
The of the present invention content (transport format information) relevant with the structure of transmission method and/or the data that will transmit in the above-described embodiments, for example corresponding to information frame.
The foregoing description has mainly been described the structure content relevant, that will change with transmission method and/or the data that will transmit.Yet, be not limited to this, also can be to use the structure of only transmitting content designator.In this case, as dispensing device, for example, as shown in figure 44, it can be that the audio-video dispensing device comprises: (1) dispensing device 5001, the relevant interior identifier of perhaps representing content of structure with transmission and transmission method and/or the data that will send as sending format content, uses transmission line identical with the transmission data or the transmission line different with described transmission line to transmit; (2) storage device 5002, the relevant content and the identifier thereof of content of the data that are used to store polytype and transmission method and/or will send, described identifier is at least one information of handling in data control information, control information transmission and control terminal side.And, as receiving system, for example, as shown in figure 45, can be the audio-video receiving system, comprise: be used to receive receiving system 5101 and the transmission format information interpreting means 5102 that is used to explain the transmission format information that receives from the transport format information of audio-video dispensing device transmission.And, the audio-video receiving system can also have storage device 5103, the content that it is used to store polytype and transmission method and/or the structure of the data that will send is relevant and the identifier of these contents, when when sending format information and receive identifier, can be with the content that is stored in the content interpret identifier in the storage device.
More particularly, be predetermined and prepare the various types of information frame in transmission and receiving terminal, by transmit identification and the data control information that is used to carry out above-mentioned various types of information frame with data, the information frame identifier of control information transmission and control information (transmission frame control information), perhaps transmit as control information, like this, can discern polytype data control information, the information frame of various types of information can be freely selected in polytype control information transmission and polytype control information according to the capacity of medium type that will transmit or transmission line.Identifier of the present invention is corresponding to above-mentioned information frame identifier.
By identifier being appended in the predetermined regular length zone of wanting information transmitted or on the precalculated position,, also can read and explain these information identifiers and default designation symbol even even be changed at receiver side terminal information frame.
And, except the structure that the foregoing description is described, also can use a kind of like this structure, only broadcasting is set by broadcast channel by the title image of the image of multichannel broadcasting in a plurality of passages, the user is by switching the program of wanting audiovisual, when being provided with of program or data needs to spend some times, the title image of the program that selection in a single day will be watched and listen to, and to user prompt.
As mentioned above, the present invention can dynamically determine sending and frame, control information transmission or the control information of the data control information that receiving terminal uses, can be corresponding to the frame of state change message, and also according to purposes or transmission line change.
And, can be by between transmission and receiving terminal, using the known information that sets in advance, data control information, control information transmission and control information are provided with expression separately whether send or receive the also default designation symbol of process information, can learn whether information frame changes, and and if only if when having carried out change by default designation symbol and notifying changed content are set, can when not changing information frame, prevent to transmit unnecessary configuration information.
Moreover, by being predetermined and preparing multiple information frame in transmission and receiving terminal, by transmit the information frame identifier that is used to discern polytype data control information, control information transmission and control information with data, perhaps transmit and to discern polytype data control information, control information transmission and control information respectively, and can freely select various types of information frames according to the type of the medium that will transmit or the capacity of transmission line as control information.
By identifier being appended in the predetermined regular length zone of wanting information transmitted or preposition,, also can read and explain these information identifiers and default designation symbol even even be changed at receiver side terminal information frame.
Embodiments of the invention are described with reference to the accompanying drawings.
In this example, explained a problem in the above-mentioned problem (B1) to (B3).
Used " image " of the present invention comprises rest image or moving image.And the image of proposition can be a two dimensional image, and for example computer graphical (CG) image perhaps can be the 3 d image data that constitutes with the wire frame type.
Figure 25 is the schematic block diagram of the image encoder or the image decoder of embodiments of the invention.
Sending or write down various coded messages, to send control sections 4011 are the devices that are used to transmit the information of coaxial cable, CATV, LAN or modulator-demodulator etc.Image encoder 4101 has image encoding part 4012, be used for to such as H.263, MPEG1/2, jpeg image information is encoded or carry out Huffman coding, also has the control section 4011 of transmission.And the output 4016 of image encoder 4102 is made of following part: receive the image synthesizing section 4015 of the reception control section 4013 of various coded messages, the picture decoding part 4014 of the various image informations that receive of decoding, synthetic one or more decoded picture and the output that constitutes by the display and the printer of output image 4016.
Figure 26 is the schematic block diagram of the audio coder and the audio decoder of one embodiment of the invention.
G.721 or the audio-frequency information of the MPEG1 audio signal audio coding part 4022 of encoding audio coder (vocoder 4021) comprising: send or write down the transmission control section 4021 of various coded messages and to such as.Audio decoder (voice decoder) 4202 comprises: receive the reception control section 4023 of various coded messages, the audio decoder part 4024 of the above-mentioned audio-frequency information of decoding, audio frequency composite part (the speech synthesiser branch) 4025 of synthetic one or more decoded audio signals and the output device 4026 of output audio signal.
Above-mentioned encoder or decoder carry out concrete coding or decoding to the time series data of sound or moving image separately.
Figure 25 and 26 communication environment can be can use many logical transport circuits and the environment that need not consider multiplex machine, as the situation of the Internet, also can be the essential environment of considering multiplex machine, as the situation of analog telephone or satellite broadcasting.As the terminal connected system, can enumerate system such as twocouese images or sound between terminals such as visual telephone or video conference call system, or in the broadcast-type image of satellite broadcasting, CATV or interconnected online broadcast or the system of sound.
Equally, composograph or sound method, can use such as scripts such as JAVA, VRML or MHEG and speak, describe image and sound, image and sound structural information (display position and demonstration time), audio-video group technology, by describe layer (degree of depth) that image shows and object ID (discerning ID) such as various objects such as image or sound and and their attribute between relation, can define composograph or sound method.The script of describing synthetic method can obtain from network or local storage.
And image encoder, image decoder, audio coder and audio decoder that can the combination in any any number constitute and send or receiving terminal.
The schematic diagram of Figure 27 (a) is used to explain the priority extention and the priority deciding section of the priority when the control overload is handled.H.263 or coding method G.723 utilization is such as, the priority of the coded message of the processing during according to predetermined standard decision overload, and be provided with and make priority extention 31 that coded message is associated with definite priority in image encoder 4101 and audio coder 4201.
The additional benchmark of priority is to be the scene variation under the situation of image, and picture frame, the data flow of editor or user's appointment are ensonified zone and no sound area under the situation of sound.
The priority addition method of the priority during the definition overload can have two kinds.During to the addition method of communication stem additional priority with at coding priority is embedded into the method in the stem of bit stream of the video that will encode or audio frequency.Preceding a kind of method can obtain the information relevant with priority without decoded information, and a kind of method in back can be handled a bit stream independently, and and system independence.
Shown in Figure 27 (b), when precedence information being appended to the communication stem, a picture frame (for example P of the I frame of intraframe coding or interframe encode or B frame) when being divided into a plurality of transmits data packets, under the situation of image, as independent information only the transmission addressable picture frame the stem additional priority (when when same image medium priority is identical, can suppose until till the next addressable picture frame appearance, priority does not change).
In decoder, be arranged at image decoder 4102 and audio decoder 4202 according to the processed part 32 of the decision of the priority when the overload of the various coded messages that receive processing method again.
Figure 28 (a) is used to explain the grade of additional priority to the schematic diagram of 28 (c).Two types priority of the priority of the processing when decoding utilizes decision terminal overload is carried out.
That is, defined: data flow priority (the data flow priority " Stream Priority " of the priority of the processing when being used for regulation and transshipping with bit stream (such as the bit stream of image and sound) unit; Priority between time series data); Be used for stipulating frame priority (the frame priority " Fram Priority " of the priority of the processing when same data flow is transshipped with frame (for example picture frame) unit; Priority in the time series data) (referring to Figure 28 (a)).
Preceding a kind of data flow priority can be handled a plurality of videos or voice data.A kind of frame priority in back can be according to editor's purpose, to the additional different priority of the picture frame (I frame) of image scene variation or same intraframe coding.
The meaning of the value that data flow priority is represented can be considered situation about handling as relative value and the situation of handling as absolute value (referring to Figure 28 (b) and 28 (c)).
The processing of data flow priority and frame priority can be by transmitting terminal such as router or gateway etc. on network the time; If terminal then has to send and receiving terminal.
Consider with two kinds of method representation absolute values or relative value.A kind of is the method shown in Figure 28 (b), and another kind is the method shown in Figure 28 (c).
In Figure 28 (b), the value (but not being the value of having considered the load variations of real network or terminal) of the order of the processing (maybe will handle) when absolute value priority is expression by additional image data stream (video flowing) of editor or mechanicalness or audio stream overload.Relative value priority is the value according to the changing load absolute priority value of terminal or network.
By priority being divided into relative value and absolute value is controlled, according to load fluctuation of network etc., by only changing relative value at transmitter side or by transponder, can keep the absolute priority that is attached in video or the audio data stream, record it on hard disk simultaneously or on the VTR.Therefore, as long as write down the value of absolute priority, that load fluctuation that just can not be subjected to network etc. influences and reproduced image or sound.And, can be independent of data and transmit relatively or absolute priority by control channel.
In Figure 28 (b), can make its grade thinner than data flow priority, the frame priority of the priority of the processing of the frame during the definition overload is perhaps handled the value as absolute priority as the value of relative priority level.For example, by in coded image, describing absolute frame priority and being the load fluctuation of reflection network or terminal, to be described in the stem of the communication data packet of transfer encoding information for the relative frame priority of the absolute priority in the picture frame that appends to the front, like this, even under the situation of frame one-level, also can be when keeping original priority, and according to the load additional priority of network or terminal.
Can also not be the communication stem, but be independent of data, the relation of description and frame transmits the relative priority level in control channel.Like this, can keep the absolute priority that appended to image or audio data stream originally, simultaneously data record on hard disk or VTR.
And, in Figure 28 (b), not at the receiving terminal identifying recording layer, lead to when crossing transmitted data on network and reproduce data at receiving terminal, owing to needn't separate absolute value and relative value is controlled at receiving terminal, can be in the level of transmitter side to frame and data flow both sides, the value of calculated in advance absolute priority and relative priority level only transmit absolute value before transmission.
In Figure 28 (c), absolute value priority is from well-determined value between the frame of the acquisition of the relation between Stream Priority and the Fram Priority.Relative value priority is the value of expression editor or the image data stream that mechanically adds or audio data stream order of processing (maybe will handle) when overload.In the example of Figure 28 (c), the frame priority (relative of each data flow of image or sound; Relative value) or to each data flow additional data flow priority.
Absolute frame priority (absolute; Absolute value) tries to achieve (that is absolute frame priority=relative frame priority+data flow priority) by relative frame priority and data flow priority sum.Also can use the method for subtraction or multiplication by constants to obtain absolute priority.
Absolute frame priority is mainly used in network.This is because utilize the statement of absolute value not need to determine to determine the priority of every frame by considering Stream Priority and Fram Priority such as transponders such as router or gateways.By using absolute frame priority, can make in the discarded processing of the frame of transponder to obtain simplifying.
On the other hand, can expect relative frame priority is mainly used in and write down or edit accumulation system.Under the situation of edit operation, can handle a plurality of images and audio data stream simultaneously.In this case, because of the load condition of terminal or network, the quantity of energy reproduced image data flow and the quantity of frame might reach capacity.
In these cases, with only by Stream Priority being separated with Fram Priority and controlling, for example, only the editor thinks preferentially to show or the Stream Priority of the data flow that the user will see explains the situation difference of absolute value by changing, and needn't recomputate each Fram Priority.Therefore, must use absolute statement or statement relatively respectively according to purposes.
By description is data flow priority to be used as relative value or as absolute value, can be illustrated in the situation and the priority under the situation of accumulation of transmission effectively.
Under the situation of Figure 28 (b), subsidiary data flow priority is that absolute value or the mark of relative value or identifier are distinguished by the value that the statement of data flow priority is set.Under the situation of frame priority, do not need mark or identifier, this is because described relative value in the communication stem, has described absolute value in coded frame.In the example of Figure 28 (c), being provided with identification frame priority is absolute value or the mark of relative value or identifier.Under the situation of absolute value, frame priority is the priority that calculates according to data flow priority and relative frame priority, and therefore, transponder or terminal are not carried out this calculating.And, when terminal known calculations formula, can be according to the relative frame priority of absolute frame priority with data flow priority backwards calculation.For example, can also obtain to want the absolute priority (Access Unit Priority) of data packets for transmission from following relative statement:
" Access Unit Priority=data flow priority-frame priority "
In this case, because the acquisition after from data flow priority, having deducted of frame priority, so it also can be expressed as bad back priority.
One or more data flow priority are associated with the priority of data processing by TCP/IP logical channel (port numbers of LAN), and control data is handled.
Moreover, give image or sound by handle than character or low data flow priority or the frame priority assignment of control information, expection can reduce repeating transmission.Even this is because parts of images or sound are lost, in most of the cases can not have problems yet.
The schematic diagram of Figure 29 is used to explain the method for priority assignment being given the multiresolution video data.
When a data flow is made of a plurality of sub data flows, can be by data flow priority be appended in the sub data flow, and the logic when describing accumulation or transmission and or logic product, definition sub data flow processing method.
Under the situation of wavelet, a picture frame can be resolved into the picture frame of a plurality of different resolutions.And, even in the mode of DCT based encode, by picture frame being divided into high fdrequency component and low frequency component, and they are encoded, also can resolve into a picture frame picture frame of a plurality of different resolutions.
The data flow priority in a plurality of image data streams that the picture frame that appends to by a series of decomposition constitutes, describe between the image data stream relation with " with " (logic product) and " or " (logic with) define.Specifically, when the data flow priority of data flow A is 5, the data flow priority of data stream B is 10 o'clock (numerical value is more little, and priority is high more), and the relation between the image data stream can be defined as according to priority and discard under the situation of data flow data, discarded data stream B, but by the relation between the data of description stream, " with " situation under, even the priority of data stream B is lower than threshold value priority, also can transmit and data streams B, and not discarded.
Thereby, can make related data flow and do not go out of use and to handling." or " situation under, then be defined as and can discard related data flow.Can be with the front same, waste treatment is sending or receiving terminal all can carry out, even also can carry out transmitting terminal.
And as the operator of describing relation, when same video clipping was encoded into 24Kbps and 48Kbps respectively, existence was reproduction 24 or the situation of reproducing 48Kbps (describing relation with different logic and distance).
When the priority that is configured to 10 latter when the former priority was configured to 5, the user can reproduce the latter according to priority, does not perhaps follow priority, selects the latter.
The schematic diagram of Figure 30 is used to explain the constructive method of communication pay(useful) load.
When constituting by a plurality of sub data flows, for example according to the data flow priority that appends in the sub data flow,, constitute the transmission packet successively by from having the sub data flow of limit priority, can make on the transmits data packets one-level, to discard to become easy.And, improving precision, the information that will have an object of the high frame priority formation communication data packet that gathers can make the discarded of communication data packet one-level and become easy.
Be associated with communication data packet by the piece cutting structure that makes image, the packet that loopback is omitted becomes easy.That is, be associated with packet structure, need be used for again synchronous sync mark again by the piece cutting structure that makes moving image.If the piece cutting structure consistent with the communication data pack arrangement is inconsistent, then when generation missing data bag makes information impaired, must add sync mark again (informing the mark of loopback location), so that carry out again synchronous.
According to mentioned above, can think high mistake proofing measure is applied to have the communication data packet of high priority.And the piece cutting structure of image is represented the unit such as the integrated images information of GOB or MB etc.
The schematic diagram of Figure 31 is used to explain the method that data are associated with the communication pay(useful) load.By data flow or object are transmitted with control information or data with the method that communication data packet is associated, can produce data format arbitrarily according to communications status or purposes.For example under the situation of RTP (real time transport protocol), to the pay(useful) load of each coding definition RTP to be processed.Existing RTP form is fixing.Under situation H.263, as shown in figure 31, three kinds of data formats of Mode A to Mode C have been defined.Under situation H.263, not having definition is the communication pay(useful) load of object with the multi-resolution image form.
In the example of Figure 31, level number " Layer No. " and above-mentioned relation are described (" with ", " or ") be added in the data format of Mode A, and defined.
The schematic diagram of Figure 32 is used for the relation between interpreted frame priority, data flow priority and the communication data packet priority.
And Figure 32 shows appending to priority in the communication data packet as the communication data packet priority on transmission line, and makes data flow priority and frame priority and the corresponding example of communication data packet priority.
Usually, under the situation of utilizing IP to communicate, must make the frame priority or the data flow priority and the corresponding data of transmitting of the next IP priority of data packets that append in image or the voice data.Because being divided into the IP packet, image or voice data transmit, so these priority are corresponded to each other.In the example of Figure 32, because the value of data flow priority is 0 to 3, the value of frame priority from 0 to 5 is so the priority of upper data can get from 0 to 15.
Under the situation of IPv6, kept 0 to 7 in the priority (4) and preengage for Congestion Control is professional.In the priority 8 is that the professional or non-Congestion Control business of real time communication is preengage to 15.Priority 15 is limit priorities, and priority 8 is lowest priorities.This is illustrated in the priority of IP packet one-level.
Under the situation of the transfer of data of utilizing IP, must make upper 0 to 15 priority corresponding with the next IP precedence 8 to 15.For these priority are corresponded to each other, can eliminate the method for a part of upper priority, evaluation function perhaps also can be set these priority are corresponded to each other.The corresponding relation of bit data and the next IP precedence carries out in via node (router or gateway) or transmission and receiving terminal in the foundation.
The transmission means are not limited in IP.Can with have the TS (transmission data flow) that resembles ATM or MPEG2, have expression whether the packet of discardable mark be object.
So far frame priority and the data flow priority described can be applied to transmission medium or data medium.Can use floppy disk or CD as data medium.
Not only can use floppy disk or CD, also can use such as medium such as IC-card or ROM boxes, if can logging program on this medium can.Moreover, can be object also with the relay of audio-video transponder (for example, router or gateway).
And, according to the information of Stream Priority (priority between time series data) or Fram Priority (priority in the time series data), the definite time series data that will retransmit, thus realize preferential the repeating transmission.For example, can prevent according to precedence information when receiving terminal is decoded, with the data flow or the frame re-transmission of non-process object.
Also can separate, determine to have the data flow or the frame of answering important priority according to the relation of retransmitting number of times and successfully send between the number of times with the present preferential utmost point to be processed.
In addition, under the situation of transmitter side terminal,, thereby realize preferential the transmission according to the definite time series data that will send of information of Stream Priority (priority between time series data) or Fram Priority (priority in the time series data).For example, according to average transfer rate or retransmit number of times, determine the frame that will send or the priority of data flow, even also can send adapting to image or sound during network over loading.
It is synthetic that the foregoing description is not limited to two dimensional image.That also can use makes up two dimensional image and 3-D view and the expression method of acquisition, perhaps comprises the synthetic image combining method of image as wide field-of-view image (panoramic picture), that each image is adjacent to each other.And, be not limited to two-way catv or B-ISDN as the object of communication system of the present invention.For example, therefrom entreat lateral terminal to be transferred to home terminal and can use radio wave (for example VHF or UHF wave band) image and sound, perhaps use satellite broadcasting, the information from family's lateral terminal is sent to the center side terminal can use analog phone line or N-ISDN (image, sound or data are not necessarily multiplexed).Can also use and utilize wireless communication system, for example IrDA, PHS (personal hand-held phone) or WLAN.
And the object terminal can be portable terminal device (for example portable information terminal) or desktop terminal (for example top setting box or personal computer).
As mentioned above, the present invention can easily handle a plurality of video data streams and a plurality of audio data stream, and emphasis ground and the main scene fragment of sound reproduced in synchronization are arranged, with reflection editor's intention.
Embodiments of the invention are described with reference to the accompanying drawings.
Following embodiment has solved above-mentioned problem (C1) to one of (C3).
Figure 33 shows the structure of the transmitter of first embodiment.Symbol 2101 presentation video input terminals, the size of an image is vertical 144 pixels * horizontal 176 pixels.Symbol 2102 expression moving image encoders constitute (referring to Recommendation H.261) by four parts 1021,1022,1023 and 1024.
Symbol 1021 expression switchs, input picture is divided into macro block (square area of 16 pixels * 16 pixels), and decision is carried out intraframe coding or is carried out interframe encode these pieces, 1022 expression motion compensation units, according to can be according to this machine decoded picture that last time, coding result calculated, produce motion compensated image, the difference between compute motion compensated image and the input picture, and with the output result of macro block unit.Motion compensation comprises the half-pixel motion compensation with long process time and has the weakness both full-pixel motion compensation of reason time.Symbol 1023 expression orthogonal converters carry out dct transform to each macro block, and 1024 expression Variable Length Code devices carry out entropy coding to dct transform result and other coded message.
Symbol 2103 expression counting devices are counted the execution number of times of four parts of video encoder 2102, and are exported the count results of each input picture to converting means.In this case, count from the execution number of times of 1022 pairs of half-pixel motion compensation of motion compensation unit and the execution number of times of both full-pixel motion compensation.
Symbol 2104 expression converting means, output serial data as shown in figure 34.Symbol 2105 expression dispensing devices are multiplexed into a serial data to the serial data of the variable length code of video encoder 2102 and converting means 2104, and to data output end 2109 these data of output.
According to said structure, can send the execution number of times that to handle (switch 1021, orthogonal converter 1023 and variable-length code device 1024) and nonessential processing (motion compensation unit 1022) to receiving system.
The dispensing device of first embodiment is corresponding to claim 68.
Figure 40 is the flow chart of the sending method of second embodiment.
Because the operation of present embodiment is similar to first embodiment, thus note units corresponding.At step 801 (image input 2101) input picture, and segment the image into macro block in step 802.After this, repeating step 803 is to the processing of step 806, up to the processing of having finished according to the conditional branching in the step 807 for each macro block.And, for the number of times that makes the processing from step 803 to step 806 can be recorded as specific variable, when having carried out processing separately, just relevant variable is increased 1.
At first, in step 803 (switch 1021) judgement macro block to be processed is carried out coding or inter codes in the piece.When being when macro block is carried out inter codes, carries out motion compensation in step 804 (motion compensation unit 1022).After this, carry out dct transform and Variable Length Code in step 805 and step 806 (orthogonal converter 1023 and Variable Length Code device 1024).When the processing finished whole macro blocks (in step 807 under the situation of "Yes"), read the variable of expression in step 808 corresponding to each execution number of times of handling, produce serial data as shown in Figure 2, and multiplexed and this serial data of output and code.As long as continuing has input picture, with regard to the processing of repeated execution of steps 801 to step 808.Said structure can transmit the execution number of times of each processing.
The sending method of second embodiment is corresponding to claim 67.
Figure 35 shows the structure of the receiving system of the 3rd embodiment.
In Figure 35, symbol 307 expression inputs are imported the output of the transmitter of first embodiment, and 302 expression receiving systems are multiplexed by going according to the output of the dispensing device of first embodiment, obtain variable length code and serial data, and output.In this case, suppose to measure in advance and exported the needed time of a page data that receives.
Symbol 303 expressions are the Video Decoder of variable length code as input, and it is made of five parts.Symbol 3031 expression variable-length decoders, obtain DCT coefficient and other coded message from variable length code, 3032 expression anti-quadrature converting means, the DCT coefficient is carried out anti-dct transform, 3033 expression switchs are that the intraframe coding or the coded message of interframe encode are switched the output of each macro block up or down according to the expression macro block.Symbol 3034 expression motion compensation units utilize decoded picture and motion encoded information last time, produce motion compensated image, and with the output addition of anti-quadrature converting means 3032 after export.Symbol 3035 expression time of implementation measurement mechanisms are measured from variable length code being input to decoder 303 backs to time of implementation of finishing decoding and output image and output.
Symbol 302 expression estimation units, from serial data from receiving system 301, receive the execution number of times of each unit (variable-length decoder 3031, anti-quadrature converting means 3032, switch 3033 or motion compensation unit 3034) and the time of implementation of time of implementation measurement mechanism 305, to estimate the time of implementation of each unit.
Method of estimation for example can be used linear regression method, and the time of implementation of estimating is made as purpose variable y, and the execution number of times of each parts is made as explanatory variable x_i.In this case, can be considered as regression parameter a_i is the time of implementation of each unit.And, under the situation of linear regression, must accumulate abundant former data, the result has wasted many memories.Yet,, also can use Kalman filter to estimate the internal state variable for many memories of avoiding waste.Can think: observed value is made as the time of implementation, the time of implementation of each unit is made as the internal state variable, then observing matrix C then is when carrying out number of times by each step variation with each unit.Symbol 304 is represented the number of times cutting devices, changes the execution number of times of each unit, to cut down the execution number of times of both full-pixel motion compensation, only increases the execution number of times of the half-pixel motion compensation of quite counting.Show the method that calculating is quite counted below.
At first, receive the execution number of times of each unit and the time of implementation of estimation, to estimate the time of implementation from estimation unit 302.Surpass when receiving system 301 receives required time of data when the time of implementation, increase the execution number of times of both full-pixel motion compensation, reduce the execution number of times of half-pixel motion compensation, be no more than one time of back up to the last time.The output of symbol 306 expression decoded pictures.
Such a case is arranged, and promptly the designated movement compensation arrangement 3034, to carry out half-pixel motion compensation according to coded message.In this case, when surpassing the predetermined execution number of times of half-pixel motion compensation, the half-pix motion both full-pixel motion of rounding off, to carry out the both full-pixel motion compensation.
According to the above-mentioned first and the 3rd embodiment, estimate time of implementation of decoding according to the estimation time of implementation of each unit, may surpass reception when needed time of one page data (fixed time) when the decoding time of implementation, replace long half-pixel motion compensation of time of implementation with the both full-pixel motion compensation.Thereby, can prevent that the time of implementation from surpassing the fixed time, solution problem (C1) (corresponding to claim 68 and 74).
And, must and nonessential processing section be considered as two groups example corresponding to claim 66 and 72, the example that video section is considered as Wave data is corresponding to claim 64 and 70.
By making receiving system in IDCT calculates, not utilize high fdrequency component, can reduce the processing time that IDCT calculates.That is, the calculating of low frequency component is considered as necessary processing, it is nonessential processing during IDCT calculates that the calculating of high fdrequency component is considered as, and also can reduce the calculation times that IDCT calculates the medium-high frequency component.
Figure 41 is the flow chart of the method for reseptance of the 4th embodiment.
Because the operation and the 3rd embodiment of present embodiment are similar, so the note units corresponding.In step 901, the variable a_i (estimation unit 302) of the time of implementation of each unit is represented in initialization.In step 902, the input multiplexed data, and measure the required time of multiplexed these data (receiving system 301).In step 903, multiplexed data is divided into variable length code and serial data, and output (receiving system 301).In step 904, obtain from serial data (Fig. 2) and respectively to carry out number of times, and be arranged to x_i.In step 905, according to the time of implementation a_i and the actual number of times (number of times cutting device 304) of carrying out of each execution number of times x_i calculating of each unit.In step 906, begin to measure the decoding time of implementation.In step 907, begin to carry out following decoding subprogram.After this, in step 908, finish to measure the decoding time of implementation (moving image decoder 303 and time of implementation measurement mechanism 3035).In step 908, estimate time of implementation of each unit according to the actual execution time of each unit of decoding time of implementation of step 908 and step 906, to upgrade a_i (estimation unit 302).Multiplexed data to each input carries out above-mentioned processing.
And, in the decoding subprogram of step 907, carry out variable-length decoding (variable-length decoder 3031) in step 910, after step 911 is carried out anti-quadrature conversion (anti-quadrature converting means 3032), processing in the piece that step 912 obtains according to the processing by step 910/information that inter codes is handled, carry out bifurcated (switch 3033).Under the situation that the piece interframe encode is handled, carry out motion compensation (motion compensation unit 3034) in the step 913.In step 913, the time of implementation of half-pixel motion compensation is counted.When the time of implementation of counting surpasses the actual execution number of times that obtains in the step 905, replace half-pixel motion compensation to carry out with the both full-pixel motion compensation.Finished above-mentioned handle for each macro block (step 914) after, finish this subprogram.
According to the above-mentioned second and the 4th embodiment, estimate time of implementation of decoding according to the estimation time of implementation of each unit, may surpass reception when needed time of one page data (fixed time) when the time of implementation, replace long half-pixel motion compensation of time of implementation with the both full-pixel motion compensation.Thus, can prevent that the time of implementation from surpassing the fixed time, solution problem (C1) (corresponding to claim 67 and 73).
And nonessential and necessary processing section is considered as two groups example corresponding to claim 65 and 71, and the example that moving image portion is considered as Wave data is corresponding to claim 63 and 69.
Figure 36 shows the structure of the receiving system of the 5th embodiment.
Most of component parts of this embodiment are identical with the described component part of second embodiment.Yet, two component part and component parts that change is arranged that increased are described below.
Symbol 402 expression estimation units, it is time of implementation of each component part of obtaining of the estimated result by the estimation unit 302 that embodiment 2 is described to be revised as not to be to output to number of times cutting device 304 but to output to its independent device to obtain.Symbol 408 expression dispensing devices, according to the time of implementation generation serial data shown in Figure 37 of each unit, and output.When being microsecond that unit, can represent up to about 65 milliseconds during the time of implementation with 16 bit representations.Therefore, these about 65 milliseconds will be enough.Symbol 409 expressions send serial data to the output of dispensing device.
And can only be right after increases the steps that produce serial data shown in Figure 37 and just can obtain method of reseptance corresponding to the 5th embodiment in symbol 808 back of Figure 40.
Figure 38 shows the structure of the transmitter of the 6th embodiment.
Most of component part of this embodiment is described identical with first embodiment.Yet, the parts of two increases are described below.Symbol 606 is represented inputs, receives the serial data of the receiving system output of the 3rd embodiment 607, and 607 expression receiving systems receive serial data, and the time of implementation of exporting each unit.Symbol 608 is represented determination devices, obtains the time of implementation of each unit, and its process is described below.At first, switch 1021 is handled each macro block of image, and obtains the execution number of times of switch 1021 at this special time.And then, can determine the execution number of times of motion compensation unit 1022, orthogonal converter 1023 and Variable Length Code device 1024 uniquely according to result at this special time.Therefore, utilize these times of implementation of carrying out number of times and receiving system 607 transmissions, estimate in the required time of implementation of receiving system side decoding.The estimation decode time that obtains as the time of implementation and carry out product between the number of times, obtain by the summation of each component part.If the decode time of estimating is equal to or greater than the required time of the yardage (for example 16Kbit) of this image generation that transmits appointments such as through-rate controller (for example when transmission rate is 64Kbits/s, it is 250 milliseconds), then increase the execution number of times of both full-pixel motion compensation, reduce the execution number of times of half-pixel motion compensation, transmit the required time so that the decoding time of implementation of estimating is no more than.(, reduce the time of implementation that the number of times of both full-pixel motion compensation can reduce full prediction because the time of implementation of both full-pixel motion compensation is short.)
In addition, moving image encoder 2102 carries out various processing according to the execution number of times of determination device 608 appointments.For example, after motion compensation unit 1022 has been carried out half-pixel motion compensation with the predetermined execution number of times of half-pixel motion compensation, only carry out the both full-pixel motion compensation.
Can improve system of selection, half-pixel motion compensation is evenly distributed in image.For example, can use following method: at first try to achieve whole macro block that need carry out half-pixel motion compensation, try to achieve the merchant (3) that the number (for example 12) of above-mentioned macro block obtains divided by the execution number of times of half-pixel motion compensation (for example 4), to carry out serial number that the macro block of half-pixel motion compensation begins with the aliquot macro block (0 of above-mentioned merchant from needs then, 3,6 or 9) carry out half-pixel motion compensation.
According to the above-mentioned the 5th and the 6th embodiment, the unit time of implementation of each estimation is sent to transmitter side, estimate the time of implementation of decoding at transmitter side, and replace long half-pixel motion compensation of time of implementation, and make the decoding time of implementation of estimation be no more than the time (fixed time) that a page data may need that receives with the both full-pixel motion compensation.Thus, in the coded message of transmission, just can discard half-pixel motion compensation information, and can prevent that the time of implementation from surpassing the fixed time, solution problem (C2) (corresponding to claim 76 and 78).
In addition, under the situation of nonessential processing, can be divided into three kinds of motion compensation to coding between macro block: the proper motion compensation; 8 * 8 motion compensation and overlapped motion compensation.
Figure 42 is the flow chart of the sending method of the 7th embodiment.
Because the operation of present embodiment is similar to the 6th embodiment, so the note units corresponding.In step 1001, time of implementation of each processing is provided with initial value.At step 801 input picture (input 2101), and it is divided into macro block in step 802.In step 1002, be coding or the judgement of inter codes (switch 1021) in the piece to whole macro blocks.The result is because the execution number of times of each processing from step 1005 to step 806 is as can be known.Therefore, in step 1003, according to the execution number of times and the actual number of times (determining device 608) of carrying out of time of implementation calculating of above-mentioned each processing.
After this, repeating step 1005 is to the processing of step 806, up to the processing of having finished according to the conditional branching of step 807 whole macro blocks.
And, when execution is respectively handled, can be recorded in the specific variable for making step 1005 to the number of processes of step 806, variables corresponding is added 1.At first, carry out bifurcated (switch 1021) according to the result of determination of step 1002 in step 1005.Under the situation of inter codes, carry out motion compensation (motion compensation unit 1022) in step 804.In this case, the number of times of half-pixel motion compensation is counted.When institute's metering number surpasses the actual frequency that obtains in step 1003, carry out the both full-pixel motion compensation and replace carrying out half-pixel motion compensation.After this, in step 805 and 806, carry out dct transform and Variable Length Code (orthogonal converter 1023 and Variable Length Code device 1024).When the processing finished whole macro blocks, (in step 807 under the situation that is) reads the variable of expression corresponding to the execution number of times of each processing in step 808, produces serial data shown in Figure 2, and multiplexed and dateout string and code.In step 1004, receive serial data, and take out each time of implementation of handling and be provided with from this serial data.
As long as the processing of image input with regard to repeated execution of steps 801 to 1004 arranged.
According to the 5th embodiment and the 7th embodiment describe part last " and " paragraph of beginning, the time of implementation of the estimation of each unit is sent to transmitter side, estimate the decoding time of implementation at transmitter side, replace long half-pixel motion compensation of time of implementation with the both full-pixel motion compensation, make the decoding time of implementation of estimation be no more than the needed possibility time (fixed time) of a page data that receives.Thus, in the coded message of transmission, can not discard the information of half-pixel motion compensation, and can prevent that the time of implementation from surpassing the fixed time, solve problem (C2) (corresponding to claim 75 and 77).
Figure 39 shows the structure of the transmitting apparatus of eighth embodiment of the invention.
Most of component part of this embodiment is identical with first embodiment description.Therefore, four component parts of increase are described below.
Symbol 7010 expression time of implementation measurement mechanisms are used for measurement image and encode and time of implementation of output image to finishing to encoder 2102 input pictures after, and export the time of implementation that records.Symbol 706 expression estimation units, be used for receiving the execution number of times of each unit (switch 1021, motion compensation unit 1022, orthogonal converter 1023 and from variable-length decoder 1024) from the serial data of counting device 2103, receive the time of implementation from time of implementation measurement mechanism 7010, and estimate the time of implementation of each component part.Can utilize the identical method of estimation of describing with the estimation unit 302 of second embodiment.The input of the frame rate value that symbol 707 expression input users send, device is determined in 708 expressions, is used to obtain the time of implementation of each unit.The process prescription that obtains is as follows.
At first, whole macro blocks that switch 1021 is handled in the image are tried to achieve the execution number of times of switch 1021 at this special time.After this, can determine the execution number of times of motion compensation unit 1022, orthogonal converter 1023 and Variable Length Code device 1024 uniquely according to result to this special time.Then, each component part is tried to achieve the summation of the product between estimated time of each component part of carrying out number of times and sending by estimation unit 706.When the predictive coding time is equal to or greater than inverse from symbol 707 frame rate that transmits time that try to achieve, that can be used for one page image encoding, increase the execution number of times of both full-pixel motion compensation, reduce the execution number of times of half-pix.
Repeat the increase and decrease of above-mentioned execution number of times, and calculate the predictive coding time, equal or be shorter than the available time up to the predictive coding time, thereby determine respectively to carry out number of times.
And moving image encoder 2102 carries out various processing according to the execution number of times of determining device 608 appointments.For example, motion compensation unit 1022 has carried out only carrying out the both full-pixel motion compensation after the half-pixel motion compensation with the predetermined execution number of times of half-pixel motion compensation.
And, also can improve system of selection for making half-pixel motion compensation even distribution in image.For example, can use following method: at first try to achieve whole macro blocks that need carry out half-pixel motion compensation, this need be carried out the macroblock number (for example 12) of half-pixel motion compensation obtain merchant (3) divided by the execution number of times of half-pixel motion compensation (for example 4), to carry out serial number that the macro block of half-pixel motion compensation begins with the aliquot macro block (0 of above-mentioned merchant from needs then, 3,6 or 9) carry out half-pixel motion compensation.
The 8th top embodiment can solve problem (C3), it estimates the time of implementation of each processing, according to the required time of implementation of estimating of time of implementation predictive coding, and determine to carry out number of times, the predictive coding time is equaled or be shorter than the image that determines according to frame rate is encoded the available time (corresponding to claim 80).
And there is full searching motion vector detection method in motion compensation unit 1022 for detecting motion vector,, in the vector in 15 levels and vertical pixel scope, detects the vector that makes SAD (the absolute difference sum of each pixel) minimum that is.Also have a kind of three step motion vector detecting methods (in annex H.261, description being arranged).Three step motion vector detecting methods are selected equally distributed nine points in above-mentioned range of search, select to have the point of minimum SAD, select nine points again in the approaching narrow scope of above-mentioned point, with the point of selecting one to have minimum SAD once more.
Also can suitably reduce the time of implementation of full searching motion vector detection method, the execution number of times that suitably increases by three step motion vector detection methods in generation, these two kinds of methods are considered as nonessential processing method, and estimate time of implementation of each method in these two methods, according to the needed time of implementation of estimating of time of implementation predictive coding, equal or be shorter than the time of user's appointment so that predict the time of implementation.
And, except that three step motion vector detection methods, can also use the further fixedly motion vector detecting method of having simplified the retrieval number of times of handling,, perhaps can and use with return movement vector (0,0) consequent motion vector detecting method only.
Figure 43 is the flow chart of the sending method of the 9th embodiment.
Because the operation of present embodiment is similar to the 8th embodiment, so the note units corresponding.To the description of the detailed description of each flow process with reference to corresponding units.
And, because present embodiment is almost identical with second embodiment, so the following difference of only explaining.
In step 1101, the initial value of the time of implementation of each processing is set to variable a_i.In step 1102, input frame speed (input 707).In step 1103, according to the frame rate of each processing of step 1102 and time of implementation a_i and in 1002 piece/piece sound encoding process determines that each execution number of times of handling that the result obtains determines the actual number of times (determining device 708) of carrying out.In step 1105 and 1106, measure the time of implementation of coding.In step 1104, estimate time of implementation of each processing, more new variables a_i (estimation unit 706) according to the actual execution number of times of the time of implementation that obtains in the step 1106 and each processing.
According to above-mentioned the 9th embodiment, estimate the time of implementation of each processing earlier, can be according to the required time of implementation of estimated time of implementation motion compensation encoding.Therefore, determine to carry out number of times, the predictive coding time is equaled or be shorter than the image of determining according to frame rate is encoded the available time (corresponding to claim 79), thereby can solve problem (C3).
Under the situation of second embodiment, when in step 808, producing serial data, also can after opening code shown in Figure 2, increase the zone of two bytes, increase binary sign indicating number of code length to this zone.
And, in the 4th embodiment, when at step 902 input multiplexed data, also can be from two byte area code fetch length, and the execution number of times that will be used for step 905 from the sign indicating number transmission time of this code length and the acquisition of sign indicating number transmission rate calculates (reducing the execution number of times of half-pixel motion compensation, to be no more than a yard transmission time).This is corresponding to claim 81 and 83.
And, in first embodiment,, also can behind opening code shown in Figure 2, increase the zone of two bytes when when step 2104 produces serial data, in this zone, increase the binary code of code length.
And, in the 3rd embodiment, when at step 301 input multiplexed data, also can be from this two byte area code fetch length, and the execution number of times that will be used for step 304 from the sign indicating number transmission time of this code length and the acquisition of sign indicating number transmission speed calculates (reducing the execution number of times of half-pixel motion compensation, to be no more than a yard transmission time).This is corresponding to claim 82 and 84.
And, in the 4th embodiment, be right after actual execution number of times in step 909 back record half-pixel motion compensation, calculate maximum.When maximum is equal to or less than enough little value (for example 2 or 3), also can produce serial data (serial data that comprises concrete bit pattern), half-pixel motion compensation is not used in expression, and sends the serial data that produces.And, in a second embodiment, confirm whether to be right after and after step 808, receive this serial data, when receiving expression when not using the serial data of half-pixel motion compensation, also can make motion compensation process in step 808 always as the both full-pixel motion compensation.This is corresponding to claim 93 and 91.
And above-mentioned principle also can be applied to not be the situation of motion compensation.For example DCT calculates by without high fdrequency component, reduces the computing time of DCT.That is, under the situation of method of reseptance,, transmit the serial data of this ratio of expression above a certain value to transmitter side when IDCT calculates the time of implementation when ratio of whole computing time is surpassed a certain value.When transmitter side receives this serial data, also can calculate and only calculate low frequency component, and all high fdrequency components are reduced to zero by DCT.This is corresponding to claim 89.
And, though the present embodiment that utilized iamge description also can be applied to audio frequency to every kind of method rather than frequently looks.This is corresponding to claim 85 and 87.
And, in the 3rd embodiment, in step 3034, write down the actual execution number of times of half-pixel motion compensation, calculate the maximum number of times maximum of carrying out.Then, when this maximum is enough little value or more hour (for example 2 or 3), can produce and send expression and not use the serial data of half-pixel motion compensation (serial data that comprises concrete bit pattern).And in first embodiment, when receiving expression when not using the serial data of half-pixel motion compensation, the motion compensation process that can make step 1022 is always as the both full-pixel motion compensation.This is corresponding to claim 94 and 92.
And above-mentioned principle can be applied to not be the situation of motion compensation.For example DCT calculates by not using high fdrequency component can reduce the processing time that DCT calculates.That is, in method of reseptance,, transmit the serial data of this ratio of expression above a certain value to transmitter side when IDCT calculates the time of implementation when ratio of whole time of implementation is surpassed a certain value.
When transmitter side receives this serial data, can calculate by DCT and only calculate low frequency component, all high fdrequency components are reduced to zero.This is corresponding to claim 90.
And, though above the embodiment that utilized iamge description, also can be applied to sound rather than image to said method.This is corresponding to claim 86 and 88.
As mentioned above, according to claim 68 and 74 (as the first and the 3rd embodiment), estimate the time of implementation of decoding according to the estimation time of implementation of each unit, and, may surpass when receiving required time (fixed time) of a page data when the decoding time of implementation of estimating, replace long half-pixel motion compensation of time of implementation with the both full-pixel motion compensation.Thus, can prevent that the time of implementation from surpassing the fixed time, solution problem (C1).
And, according to claim 75 and 77 (for example the 5th and the 7th embodiment), the estimation time of implementation of each unit is sent to transmitter side, estimate the decoding time of implementation at transmitter side, and,, the decode time of estimating is suitable for receiving the needed time of a page data (fixed time) so that being no more than with long half-pixel motion compensation of both full-pixel motion compensation replacement time of implementation.Thus, in the coded message that sends, do not discard the information of half-pixel motion compensation, and can prevent that the time of implementation from surpassing the fixed time, solution problem (C2).
And, according to claim 79 (for example the 9th embodiment), can solve problem (C3), its method is to estimate the time of implementation of each processing, according to the estimated needed time of implementation of time of implementation predictive coding, and determine to carry out number of times, it is available to encoding the spendable time according to the image of frame rate decision that the predictive coding time is equal to or less than.
Therefore, even the present invention can realize also the slow down function (CGD: calculating moderate reduction) of quality deterioration of calculated load increase, by implementing to obtain very big interests.
And, computer utilization such as magnetic recording medium or optical recording media can be realized above-mentioned operation, wherein make the program of computer realize all or part of of each step (or each device) (the perhaps operation of each device) of describing among arbitrary embodiment in the foregoing description.
Industrial applicability
As mentioned above, the present invention can change information frame corresponding to state, purposes or transmission line, dynamically specified data control information, control information transmission and be used for the frame of the control information of sending and receiving terminal. And, be easy to process a plurality of video data streams or a plurality of audio data stream, and have emphasis ground and synchronous sound ground to reproduce main scene fragment, reflection editor's intention. In addition, according to the time of implementation of the element of each estimation, may surpass when receiving required time (fixed time) of a page data when the decoding time of implementation of estimating, replace long half-pixel motion compensation of time of implementation with the both full-pixel motion compensation, can prevent that the time of implementation from surpassing the time of appointment.

Claims (2)

1. method of reseptance is characterized in that having following steps:
Receive the step of the title image relevant with program from the broadcasting channel of the title image of the image of broadcasting by the multifrequency channel radio only;
By switching program by the user, when required program that in memory, has the longer time for reading of needs or data, in case select the user think audiovisual program title image and to user prompt, perhaps point out and currently read, after reading required program or data by memory, get back to the step that the user thinks the program of audiovisual once more.
2. method of reseptance as claimed in claim 1, described title image comprise the broadcast image that obtains by the program of periodically sampling by a plurality of channel broadcastings.
CN 200610099755 1997-03-17 1998-03-13 Method and apparatus for transmitting and receiving dynamic image data Expired - Lifetime CN1941889B (en)

Applications Claiming Priority (17)

Application Number Priority Date Filing Date Title
JP1997-062667 1997-03-17
JP6266797 1997-03-17
JP1997062667 1997-03-17
JP1997-090640 1997-04-09
JP1997090640 1997-04-09
JP9064097 1997-04-09
JP1997179342 1997-07-04
JP1997-179342 1997-07-04
JP17934297 1997-07-04
JP1997226027 1997-08-22
JP1997226045 1997-08-22
JP1997-226027 1997-08-22
JP22602797 1997-08-22
JP22604597 1997-08-22
JP1997-226045 1997-08-22
JP33210197 1997-12-02
JP332101/97 1997-12-02

Related Parent Applications (2)

Application Number Title Priority Date Filing Date
CNB98800657XA Division CN1190081C (en) 1997-03-17 1998-03-13 Method and apparatus for processing, transmitting and receiving dynamic image data
CNB2004100323610A Division CN100334880C (en) 1997-03-17 1998-03-13 Method and its device for transmitting and receiving dynamic image data

Publications (2)

Publication Number Publication Date
CN1941889A CN1941889A (en) 2007-04-04
CN1941889B true CN1941889B (en) 2011-06-08

Family

ID=36919307

Family Applications (6)

Application Number Title Priority Date Filing Date
CN 200910148801 Pending CN101616317A (en) 1997-03-17 1998-03-13 The method and the equipment thereof of transmission and receiving moving pictures data
CN 200610099755 Expired - Lifetime CN1941889B (en) 1997-03-17 1998-03-13 Method and apparatus for transmitting and receiving dynamic image data
CN 200610099756 Expired - Lifetime CN100589545C (en) 1997-03-17 1998-03-13 Method and apparatus for transmitting and receiving dynamic image data
CN 200610099753 Expired - Lifetime CN1941888B (en) 1997-03-17 1998-03-13 Method and apparatus for transmitting and receiving dynamic image data
CN 200610009301 Expired - Lifetime CN100473158C (en) 1997-03-17 1998-03-13 Method and apparatus for processing, transmitting and receiving dynamic image data
CN 200610009302 Expired - Lifetime CN100473159C (en) 1997-03-17 1998-03-13 Method and apparatus for processing, transmitting and receiving dynamic image data

Family Applications Before (1)

Application Number Title Priority Date Filing Date
CN 200910148801 Pending CN101616317A (en) 1997-03-17 1998-03-13 The method and the equipment thereof of transmission and receiving moving pictures data

Family Applications After (4)

Application Number Title Priority Date Filing Date
CN 200610099756 Expired - Lifetime CN100589545C (en) 1997-03-17 1998-03-13 Method and apparatus for transmitting and receiving dynamic image data
CN 200610099753 Expired - Lifetime CN1941888B (en) 1997-03-17 1998-03-13 Method and apparatus for transmitting and receiving dynamic image data
CN 200610009301 Expired - Lifetime CN100473158C (en) 1997-03-17 1998-03-13 Method and apparatus for processing, transmitting and receiving dynamic image data
CN 200610009302 Expired - Lifetime CN100473159C (en) 1997-03-17 1998-03-13 Method and apparatus for processing, transmitting and receiving dynamic image data

Country Status (1)

Country Link
CN (6) CN101616317A (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5764140B2 (en) 2009-12-16 2015-08-12 ダウ グローバル テクノロジーズ エルエルシー Production of epoxy resins using improved ion exchange resin catalysts.
US8416281B2 (en) * 2010-11-24 2013-04-09 International Business Machines Corporation Multipoint conference scalability for co-located participants
CN110990626A (en) * 2019-12-09 2020-04-10 深圳市迅雷网络技术有限公司 Picture processing method, device and system and storage medium

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH02152382A (en) * 1988-12-02 1990-06-12 Matsushita Electric Ind Co Ltd Teletext receiver
US5239540A (en) * 1990-11-27 1993-08-24 Scientific-Atlanta, Inc. Method and apparatus for transmitting, receiving and communicating digital data signals with corresponding program data signals which describe the digital data signals
KR0147210B1 (en) * 1994-07-15 1998-09-15 이헌조 A caption processing apparatus and method of the caption vcr with an alternative display device
KR100409187B1 (en) * 1994-08-16 2004-03-10 소니 가부시끼 가이샤 TV signal receiver and program switching device and method and remote controller

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
JP平5-137125A 1993.06.01
JP平5-327758A 1993.12.10
JP平6-70307A 1994.03.11
JP昭63-73786A 1988.04.04

Also Published As

Publication number Publication date
CN100473158C (en) 2009-03-25
CN1941888B (en) 2012-10-10
CN1941888A (en) 2007-04-04
CN1941890A (en) 2007-04-04
CN1941889A (en) 2007-04-04
CN1819655A (en) 2006-08-16
CN100473159C (en) 2009-03-25
CN1819654A (en) 2006-08-16
CN101616317A (en) 2009-12-30
CN100589545C (en) 2010-02-10

Similar Documents

Publication Publication Date Title
CN100334880C (en) Method and its device for transmitting and receiving dynamic image data
JP3516585B2 (en) Data processing device and data processing method
KR100587324B1 (en) Method for servicing digital multimedia broadcasting, transmitter/receiver, and data structure
CN101453639B (en) Encoding, decoding method and system for supporting multi-path video stream of ROI region
CN100505823C (en) Robust mode staggercasting storing content
CN100531298C (en) Multimedia signal matching system and method for performing picture-in-picture function
CN102067551B (en) Media stream processing
WO1999004566A1 (en) Method and system for multiplexing image signal, method and system for demultiplexing image signal, and transmission medium
CN1941889B (en) Method and apparatus for transmitting and receiving dynamic image data
KR20000022500A (en) Method of encoding information, its encoder, its decoding/ synthesizing method, its decoder/synthesizer and recording which those methods are recorded.
CN1774924B (en) Method and apparatus for preventing error propagation in a video sequence
JP4102223B2 (en) Data processing apparatus and data processing method
KR100530919B1 (en) Data processing method and data processing apparatus
JP2007221826A (en) Receiving terminal and reception method
JP3519722B2 (en) Data processing method and data processing device
JP3448047B2 (en) Transmitting device and receiving device
KR100530920B1 (en) Image and voice transmitting apparatus and receiving apparatus
JP2006304309A (en) Transmitter, receiver, and communication system
KR100713363B1 (en) Apparatus and method for transmitting mpeg
TW410298B (en) Video/audio transmission apparatus, video/audio receiving apparatus, data processing apparatus, data processing method, data processing apparatus and data processing method, transmission method and apparatus for waveform data
JP2004048657A (en) Image/audio receiving apparatus

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
CX01 Expiry of patent term

Granted publication date: 20110608

CX01 Expiry of patent term